性能问题，AWR High Event enq: US - contention
1.1问题现象
应用反馈业务执行SQL响应超时，需要数据库排除DB是否存在问题，创建AWR观察到top event 新增enq: US - contention ???
1.2问题分析
1） DB Time指标
2020-11-12 17:30 18:00 DB time 3310.19
2020-11-12 18:00 18:30 DB time 15883.77
2020-11-12 18:30 19:00 DB time 19463.52
2020-11-12 19:00 19:30 DB time 16001.57 2） AWR报告
Top 10 Foreground Events by Total Wait Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tota Wait % DB
Event Waits Time Avg(ms) time Wait Class
------------------------------ ------------ ---- ------- ------ ----------
latch: row cache objects 41,636,274 669. 16 57.4 Concurrenc
enq: US - contention 20,130,960 291. 14 25.0 Other
latch: ges resource hash list 11,265,725 55.8 5 4.8 Other
DB CPU 33.2 2.8
buffer busy waits 3,502,842 2301 1 .2 Concurrenc
log file sync 1,831,042 1774 1 .2 Commit
library cache: mutex X 301,887 1170 4 .1 Concurrenc
latch: cache buffers chains 1,222,834 924. 1 .1 Concurrenc
latch: enqueue hash chains 365,389 222. 1 .0 Other
latch: undo global data 505,480 116. 0 .0 Other Statistic Name Time (s) % of DB Time
------------------------------------------ ------------------ ------------
sql execute elapsed time 1,165,594.2 99.8
PL/SQL execution elapsed time 611,956.6 52.4
DB CPU 33,234.1 2.8
parse time elapsed 20.2 .0
connection management call elapsed time 17.0 .0
repeated bind elapsed time 3.2 .0
hard parse elapsed time 0.0 .0
sequence load elapsed time 0.0 .0
hard parse (sharing criteria) elapsed time 0.0 .0
failed parse elapsed time 0.0 .0
DB time 1,167,811.4
background elapsed time 1,515.8
background cpu time 513.7 -> Ordered by statistic name Statistic Total per Second per Trans
-------------------------------- ------------------ -------------- -------------
db block changes 97,449,293 54,125.4 47.7
execute count 14,890,901 8,270.7 7.3
gc cr block receive time 1,277 0.7 0.0
gc cr blocks received 28,544 15.9 0.0
gc current block receive time 24,427 13.6 0.0
gc current blocks received 94,906 52.7 0.1
logons cumulative 2,788 1.6 0.0
opened cursors cumulative 3,649,945 2,027.3 1.8
parse count (total) 2,649,940 1,471.8 1.3
parse time elapsed 1,781 1.0 0.0
physical reads 18,385 10.2 0.0
physical writes 2,808,767 1,560.1 1.4
redo size 23,166,050,528 12,866,904.3 11,345.2
session cursor cache hits 2,292,253 1,273.2 1.1
session logical reads 2,394,937,749 1,330,198.0 1,172.9
user calls 8,341,907 4,633.3 4.1
user commits 1,911,204 1,061.5 0.9
user rollbacks 130,726 72.6 0.1
workarea executions - optimal 206,244 114.6 0.1
------------------------------------------------------ 3）新出现的等待事件是什么？
enq: US - contention 参考如下文档
How to correct performance issues with enq: US - contention related to undo segments (Doc ID 1332738.1)
The wait event "enq: US Contention" is associated with contention on the latch in the row cache (dc_rollback_seg). Enqueue US - Contention can become a bottle-neck for performance if workload dictates that a lot of offlined undo segments must be onlined over a short period of time. The latch on the row cache can be unable to keep up with the workload.
This can happen for a number of reasons and some scenarios are legitimate workload demands.
Solution: ensure that peaks in onlined undo segments do not happen (see workaround #2). That is not always feasible.
从里面我们能掌握的信息有，UNDO的争用, 理想化的解决方法是避免undo段的使用出现峰值？？？ 呵呵。
4）UNDO表空间的使用情况查询
SQL> select status,sum(bytes)/1024/1024/1024,TABLESPACE_NAME from dba_undo_extents group by status,TABLESPACE_NAME;
STATUS SUM(BYTES)/1024/1024/1024 TABLESPACE_NAME
--------- ------------------------- ------------------------------
EXPIRED .194824219 UNDOTBS1
EXPIRED 2.67224121 UNDOTBS2
UNEXPIRED 29.1513672 UNDOTBS1
ACTIVE 7.76220703 UNDOTBS1
UNEXPIRED 10.0617676 UNDOTBS2
GROUP_NUMBER NAME TOTAL_GB FREE_GB FREE_PERCENT STATE TYPE CARE
------------ -------------------- ---------- ---------- ------------ --------------- ------ -----
1 ARCH 500.00 499.87 99.97 CONNECTED EXTERN
2 DATA 500.00 49.74 9.95 CONNECTED EXTERN *
3 OCR 30.00 29.10 96.99 MOUNTED NORMAL
4 REDO_LOG 100.00 85.87 85.87 CONNECTED EXTERN
SQL> show parameter undo
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
_gc_undo_affinity boolean FALSE
_undo_autotune boolean FALSE
undo_management string AUTO
undo_retention integer 10800
undo_tablespace string UNDOTBS1
SQL> select 10800/60/60 from dual;
10800/60/60
-----------
3
1.3问题处理
根据查询到的UNDO情况，我们可以发现UNDOTBS1 第一个文件自扩展，当前分配7G,最大可扩展31.9xxG; UNDOTBS2 第一个文件自扩展，当前分配0.02G,最大可扩展31.9XXXG;
根据UNDO EXTENT状态可以发现，UNDOTBS1 30G处于UNEXPIRED 未过期状态，7G处于正在使用；只有少量数量处于已过期状态；
UNDOTBS2 10G处于UNEXPIRED 未过期状态，0G处于正在使用；2.6G已过期状态；
由于UNDOTBS1 有2个UNDO文件，第一个DB建好，自扩展，当前7，最大31.9G； 第二个DB建后，DBA新增的已完全分配，非自动扩展30G；
本次就是由于UNDOTBS1还有扩展的可能性，因此新的UNDO申请需要对UNDOTBS1的第一个自扩展文件进行扩展时，出现严重的争用；
扩容！！！ SQL> alter tablespace UNDOTBS1 add datafile '+DATA' size 30g;
SQL> alter tablespace UNDOTBS2 add datafile '+DATA' size 10g;
业务反馈突然好了！
收尾
select file_id,TABLESPACE_NAME,bytes/1024/1024/1024,MAXbytes/1024/1024/1024 from dba_data_files where tablespace_name in('UNDOTBS1','UNDOTBS2');
FILE_ID TABLESPACE_NAME BYTES/1024/1024/1024 MAXBYTES/1024/1024/1024
---------- ------------------------------ -------------------- -----------------------
3 UNDOTBS1 7.14355469 31.9999847
5 UNDOTBS2 .024414063 31.9999847
6 UNDOTBS1 30 0
7 UNDOTBS2 30 0
18 UNDOTBS1 30 0
19 UNDOTBS2 10 0
6 rows selected.
select file_name from dba_data_files where file_id in(3,5);
SQL> alter database datafile 3,5 autoextend off;
SQL> alter database datafile '+DATA/wglcta/datafile/undotbs1.258.1047648231' resize 10g;
SQL> alter database datafile '+DATA/wglcta/datafile/undotbs2.264.1047648343' resize 10g;
select file_id,TABLESPACE_NAME,bytes/1024/1024/1024,AUTOEXTENSIBLE,MAXbytes/1024/1024/1024 from dba_data_files where tablespace_name in('UNDOTBS1','UNDOTBS2');
FILE_ID TABLESPACE_NAME BYTES/1024/1024/1024 AUT MAXBYTES/1024/1024/1024
---------- ------------------------------ -------------------- --- -----------------------
3 UNDOTBS1 10 NO 0
5 UNDOTBS2 10 NO 0
6 UNDOTBS1 30 NO 0
7 UNDOTBS2 30 NO 0
18 UNDOTBS1 30 NO 0
19 UNDOTBS2 10 NO 0
GROUP_NUMBER NAME TOTAL_GB FREE_GB FREE_PERCENT STATE TYPE CARE
------------ -------------------- ---------- ---------- ------------ --------------- ------ -----
1 ARCH 500.00 499.87 99.97 CONNECTED EXTERN
2 DATA 500.00 26.89 5.38 CONNECTED EXTERN *
1.4事后追踪
1） 确认UNDOTBS1的使用情况，最近几个小时包括问题时段
SQL> select ts#,name from v$tablespace where name='UNDOTBS1';
TS# NAME
---------- --------------------
2 UNDOTBS1 select * from
DBA_HIST_TBSPC_SPACE_USAGE where TABLESPACE_ID=2 order by RTIME;
SNAP_ID DBID TABLESPACE_ID TABLESPACE_SIZE TABLESPACE_MAXSIZE TABLESPACE_USEDSIZE RTIME
---------- ---------- ------------- --------------- ------------------ ------------------- -------------------------
4728 476729316 2 3942400 8126462 1344368 11/12/2020 01:30:01
4729 476729316 2 3942400 8126462 1147240 11/12/2020 02:00:04
4730 476729316 2 3942400 8126462 1001072 11/12/2020 02:30:10
4731 476729316 2 3942400 8126462 923368 11/12/2020 03:00:12
4732 476729316 2 3942400 8126462 959200 11/12/2020 03:30:01
4733 476729316 2 3942400 8126462 973392 11/12/2020 04:00:02
4734 476729316 2 3942400 8126462 990800 11/12/2020 04:30:03
4735 476729316 2 3942400 8126462 1031128 11/12/2020 05:00:04
4736 476729316 2 3942400 8126462 1087312 11/12/2020 05:30:06
4737 476729316 2 3942400 8126462 1168968 11/12/2020 06:00:07
4738 476729316 2 3942400 8126462 1315032 11/12/2020 06:30:08
4739 476729316 2 3942400 8126462 1544672 11/12/2020 07:00:10
4740 476729316 2 3942400 8126462 1795568 11/12/2020 07:30:11
4741 476729316 2 3942400 8126462 2033392 11/12/2020 08:00:12
4742 476729316 2 3942400 8126462 2305024 11/12/2020 08:30:14
4743 476729316 2 3942400 8126462 2605192 11/12/2020 09:00:15
4744 476729316 2 3942400 8126462 2872568 11/12/2020 09:30:16
4745 476729316 2 3942400 8126462 3098480 11/12/2020 10:00:18
4746 476729316 2 3942400 8126462 3246192 11/12/2020 10:30:06
4747 476729316 2 3942400 8126462 3414576 11/12/2020 11:00:11
4748 476729316 2 3942400 8126462 3577952 11/12/2020 11:30:14
4749 476729316 2 3942400 8126462 3580104 11/12/2020 12:00:17
4750 476729316 2 3942400 8126462 3458504 11/12/2020 12:30:19
4751 476729316 2 3942400 8126462 3300168 11/12/2020 13:00:22
4752 476729316 2 3942400 8126462 3024576 11/12/2020 13:30:28
4753 476729316 2 3942400 8126462 2831064 11/12/2020 14:00:29
4754 476729316 2 3942400 8126462 2698656 11/12/2020 14:30:31
4755 476729316 2 3942400 8126462 2644232 11/12/2020 15:00:32
4756 476729316 2 3942400 8126462 2669832 11/12/2020 15:30:37
4757 476729316 2 3942400 8126462 2780168 11/12/2020 16:00:38
4758 476729316 2 3942400 8126462 2911240 11/12/2020 16:30:40
4759 476729316 2 3942400 8126462 2985720 11/12/2020 17:00:41
4760 476729316 2 3942400 8126462 3577632 11/12/2020 17:30:42
4761 476729316 2 3942400 8126462 3697616 11/12/2020 18:00:44
4762 476729316 2 3942400 8126462 3812664 11/12/2020 18:30:44
4763 476729316 2 4071680 8126462 4055680 11/12/2020 19:01:35
4764 476729316 2 8800640 12058622 4935560 11/12/2020 19:30:12
695 rows selected. 参考
https://yq.aliyun.com/articles/283841 我们查询ASH视图看看当时的情况： SELECT D.SQL_ID,CHR(BITAND(P1, -16777216) / 16777215) ||
CHR(BITAND(P1, 16711680) / 65535) "Lock",
BITAND(P1, 65535) "Mode", COUNT(1),COUNT(DISTINCT d.session_id )
FROM DBA_HIST_ACTIVE_SESS_HISTORY D
WHERE D.SAMPLE_TIME BETWEEN TO_DATE('2020-11-12 16:00:00', 'YYYY-MM-DD HH24:MI:SS') AND
TO_DATE('2020-11-12 19:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND D.EVENT = 'enq: US - contention'
GROUP BY D.SQL_ID,(CHR(BITAND(P1, -16777216) / 16777215) ||
CHR(BITAND(P1, 16711680) / 65535)),(BITAND(P1, 65535)); SQL_ID Lock Mode COUNT(1) COUNT(DISTINCTD.SESSION_ID)
------------- -------- ---------- ---------- ---------------------------
q US 6 5472 1161
c US 6 33943 1340select distinct sql_id,sql_text from v$sql where sql_id in('xx')
order by 1;
2）AWR对比报告，看看能否观察一些其它信息
AWR对比报告
@?/rdbms/admin/awrddrpt.sql
Snapshot Set Begin Snap Id Begin Snap Time End Snap Id End Snap Time Avg Active Users Elapsed Time (min) DB time (min)
------------ -------------- ------------------------- ------------ ------------------------- --------------------------
1st 4761 12-Nov-20 18:00:43 (Thu) 4762 12-Nov-20 18:30:00 (Thu) 542.2 29.3 15,883.8
2nd 4760 12-Nov-20 17:30:41 (Thu) 4761 12-Nov-20 18:00:43 (Thu) 113.0 30.0 3,310.2
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ %Diff: -79.7% 2.5% -79.2%
How to correct performance issues with enq: US - contention related to undo segments (Doc ID 1332738.1)
IF: Undo Wait Event - Enq: US - contention (Doc ID 1951680.1)
Update to latest version or apply the Patch 14226599 (included in 11.2.0.4 patchset and above) and Patch 17720709 (included in 12.2 and above)
Setting _ROLLBACK_SEGMENT_COUNT to a high number to keep undo segments online:
ALTER SYSTEM SET "_rollback_segment_count"=<n>;
Note: In databases with high query activity, particularly parallel query and a high setting for _ROLLBACK_SEGMENT_COUNT, you can expect to see wait contention on the row cache for DC_ROLLBACK_SEGS. It is highly recommended in these environments where setting _ROLLBACK_SEGMENT_COUNT to a high value (10s of thousands and higher) apply the patch for Bug:14226599 (as mentioned in solution 1). This will increase the hash buckets on the DC_ROLLBACK_SEGS row cache to help alleviate latch contention.
A fix to Bug:7291739 is to set a new hidden parameter, _HIGHTHRESHOLD_UNDORETENTION to set a high threshold for undo retention completely distinct from maxquerylen:
ALTER SYSTEM SET "_highthreshold_undoretention"=<n>;
性能问题，AWR High Event enq: US - contention的更多相关文章
- Analyzing 'enq: HW - contention' Wait Event (Doc ID 740075.1)
Analyzing 'enq: HW - contention' Wait Event (Doc ID 740075.1) In this Document Symptoms Cause ...
- enq: TM - contention一例
今天下午,有台服务器出现异常,响应特别慢,io等待奇高,awr top 5事件如下: 经回查ash,找到了造成这些事件的sql语句,如下: select * from v$active_session ...
- 【等待事件】序列等待事件总结（enq: SQ - contention、row cache lock、DFS lock handle和enq: SV - contention）
[等待事件]序列等待事件总结(enq: SQ - contention.row cache lock.DFS lock handle和enq: SV - contention) 1 BLOG文档结 ...
- Oracle 性能之 Enq: CF - contention
Oracle 性能之 Enq: CF - contention Table of Contents 1. 原因 2. 解决问题 2.1. 针对持有锁进程类型处理 2.1.1. 查看持有锁会话的进程类型 ...
- 【故障处理】队列等待之enq IV - contention案例
[故障处理]队列等待之enq IV - contention案例 1.1 BLOG文档结构图 1.2 前言部分 1.2.1 导读和注意事项 各位技术爱好者,看完本文后,你可以掌握如下的技能,也 ...
- 等待事件：enq: HW - contention和enq: TM - contention
今天生成了生产库前几日的AWR报告,发现等待事件中出现了一个陌生的event--enq: HW - contention,google一下是ASSM(Auto Segment Space Manage ...
- enq: SQ - contention
--每分钟操作 SQL> select sql_id, mi, count(mi) 2 from (select event, sql_id, to_char(sample_time ...
- oracle 11g enq: JI – contention等待事件
最近使用物化视图同步的环境在大量刷新的时候频繁出现enq: JI – contention等待事件,经查: JI enqueue is acquired in exclusive mode on th ...
- undo表空间居高不下和enq: US - contention
这几天遇到一个错误,我也不知道算不算错误吧,因为没有报错,只是在那突然的短短2分钟内表的操作突然降低了,导致了该软件重新启动.查看alert日志没有报错,而是在ASH里找到了TOP SQL框有一个这样 ...
随机推荐
- Linux系统编程—管道
▋****1. 管道的概念 管道,又名「无名管理」,或「匿名管道」,管道是一种非常基本,也是使用非常频繁的IPC方式. 1.1 管道本质 管道的本质也是一种文件,不过是伪文件,实际上是一块内核缓冲区, ...
- 从面试角度学完 Kafka
Kafka 是一个优秀的分布式消息中间件,许多系统中都会使用到 Kafka 来做消息通信.对分布式消息系统的了解和使用几乎成为一个后台开发人员必备的技能.今天码哥字节就从常见的 Kafka 面试题入手 ...
- pytest文档33-Hooks函数获取用例执行结果(pytest_runtest_makereport)
前言 pytest提供的很多钩子(Hooks)方法方便我们对测试用例框架进行二次开发,可以根据自己的需求进行改造. 先学习下pytest_runtest_makereport这个钩子方法,可以更清晰的 ...
- super函数
Python面向对象中super用法与MRO机制:https://www.cnblogs.com/chenhuabin/p/10058594.html python 中 super函数的使用:http ...
- spring-boot-route（十七）使用aop记录操作日志
在上一章内容中--使用logback管理日志,我们详细讲述了如何将日志生成文件进行存储.但是在实际开发中,使用文件存储日志用来快速查询问题并不是最方便的,一个优秀系统除了日志文件还需要将操作日志进行持 ...
- 【C语言高级编程】你见过长度为0的数组吗？管你信不信，看就完了！
一.什么是零长度数组 零长度数组就是长度为0的数组. ANSI C 标准规定:定义一个数组时,数组的长度必须是一个常数,即数组的长度在编译的时候是确定的.在ANSI C 中定义一个数组的方法如下: 类 ...
- centos8上配置openresty/nginx可访问php
一,创建一个测试站的目录 [root@yjweb data]# mkdir dev [root@yjweb data]# cd dev [root@yjweb dev]# mkdir think_ww ...
- 往with as中写入数据的方法
方法1:直接写入,使用union all,简单直观,但程序运行效率低,几百条就很慢了 with dw_wms_outbound_info_v100 as( select '10700001' as o ...
- 【API管理 APIM】APIM集成内部VNet时，常遇见的关于自定义DNS服务问题。
问题描述 Azure 的APIM集成虚拟网络有两种方式,外部VNET, 内部VNET. 外部VNET,要求低,可以通过APIM访问VNET中的VM等资源,不需要配置自定义DNS服务器,这种方式下,AP ...
- mac 搭建 Robot Framework
前提介绍,我的mac上python2和python3是都要有的,然后大家可以看看我其他的文章,这些文章虽然很多都是连接,是别人的博客或者资料,但都是自己试过没有问题的,只是比较懒然后就没有自己写. r ...