性能问题，AWR High Event enq: US - contention
1.1问题现象
应用反馈业务执行SQL响应超时，需要数据库排除DB是否存在问题，创建AWR观察到top event 新增enq: US - contention ???
1.2问题分析
1） DB Time指标
2020-11-12 17:30 18:00 DB time 3310.19
2020-11-12 18:00 18:30 DB time 15883.77
2020-11-12 18:30 19:00 DB time 19463.52
2020-11-12 19:00 19:30 DB time 16001.57 2） AWR报告
Top 10 Foreground Events by Total Wait Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tota Wait % DB
Event Waits Time Avg(ms) time Wait Class
------------------------------ ------------ ---- ------- ------ ----------
latch: row cache objects 41,636,274 669. 16 57.4 Concurrenc
enq: US - contention 20,130,960 291. 14 25.0 Other
latch: ges resource hash list 11,265,725 55.8 5 4.8 Other
DB CPU 33.2 2.8
buffer busy waits 3,502,842 2301 1 .2 Concurrenc
log file sync 1,831,042 1774 1 .2 Commit
library cache: mutex X 301,887 1170 4 .1 Concurrenc
latch: cache buffers chains 1,222,834 924. 1 .1 Concurrenc
latch: enqueue hash chains 365,389 222. 1 .0 Other
latch: undo global data 505,480 116. 0 .0 Other Statistic Name Time (s) % of DB Time
------------------------------------------ ------------------ ------------
sql execute elapsed time 1,165,594.2 99.8
PL/SQL execution elapsed time 611,956.6 52.4
DB CPU 33,234.1 2.8
parse time elapsed 20.2 .0
connection management call elapsed time 17.0 .0
repeated bind elapsed time 3.2 .0
hard parse elapsed time 0.0 .0
sequence load elapsed time 0.0 .0
hard parse (sharing criteria) elapsed time 0.0 .0
failed parse elapsed time 0.0 .0
DB time 1,167,811.4
background elapsed time 1,515.8
background cpu time 513.7 -> Ordered by statistic name Statistic Total per Second per Trans
-------------------------------- ------------------ -------------- -------------
db block changes 97,449,293 54,125.4 47.7
execute count 14,890,901 8,270.7 7.3
gc cr block receive time 1,277 0.7 0.0
gc cr blocks received 28,544 15.9 0.0
gc current block receive time 24,427 13.6 0.0
gc current blocks received 94,906 52.7 0.1
logons cumulative 2,788 1.6 0.0
opened cursors cumulative 3,649,945 2,027.3 1.8
parse count (total) 2,649,940 1,471.8 1.3
parse time elapsed 1,781 1.0 0.0
physical reads 18,385 10.2 0.0
physical writes 2,808,767 1,560.1 1.4
redo size 23,166,050,528 12,866,904.3 11,345.2
session cursor cache hits 2,292,253 1,273.2 1.1
session logical reads 2,394,937,749 1,330,198.0 1,172.9
user calls 8,341,907 4,633.3 4.1
user commits 1,911,204 1,061.5 0.9
user rollbacks 130,726 72.6 0.1
workarea executions - optimal 206,244 114.6 0.1
------------------------------------------------------ 3）新出现的等待事件是什么？
enq: US - contention 参考如下文档
How to correct performance issues with enq: US - contention related to undo segments (Doc ID 1332738.1)
The wait event "enq: US Contention" is associated with contention on the latch in the row cache (dc_rollback_seg). Enqueue US - Contention can become a bottle-neck for performance if workload dictates that a lot of offlined undo segments must be onlined over a short period of time. The latch on the row cache can be unable to keep up with the workload.
This can happen for a number of reasons and some scenarios are legitimate workload demands.
Solution: ensure that peaks in onlined undo segments do not happen (see workaround #2). That is not always feasible.
从里面我们能掌握的信息有，UNDO的争用, 理想化的解决方法是避免undo段的使用出现峰值？？？ 呵呵。
4）UNDO表空间的使用情况查询
SQL> select status,sum(bytes)/1024/1024/1024,TABLESPACE_NAME from dba_undo_extents group by status,TABLESPACE_NAME;
STATUS SUM(BYTES)/1024/1024/1024 TABLESPACE_NAME
--------- ------------------------- ------------------------------
EXPIRED .194824219 UNDOTBS1
EXPIRED 2.67224121 UNDOTBS2
UNEXPIRED 29.1513672 UNDOTBS1
ACTIVE 7.76220703 UNDOTBS1
UNEXPIRED 10.0617676 UNDOTBS2
GROUP_NUMBER NAME TOTAL_GB FREE_GB FREE_PERCENT STATE TYPE CARE
------------ -------------------- ---------- ---------- ------------ --------------- ------ -----
1 ARCH 500.00 499.87 99.97 CONNECTED EXTERN
2 DATA 500.00 49.74 9.95 CONNECTED EXTERN *
3 OCR 30.00 29.10 96.99 MOUNTED NORMAL
4 REDO_LOG 100.00 85.87 85.87 CONNECTED EXTERN
SQL> show parameter undo
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
_gc_undo_affinity boolean FALSE
_undo_autotune boolean FALSE
undo_management string AUTO
undo_retention integer 10800
undo_tablespace string UNDOTBS1
SQL> select 10800/60/60 from dual;
10800/60/60
-----------
3
1.3问题处理
根据查询到的UNDO情况，我们可以发现UNDOTBS1 第一个文件自扩展，当前分配7G,最大可扩展31.9xxG; UNDOTBS2 第一个文件自扩展，当前分配0.02G,最大可扩展31.9XXXG;
根据UNDO EXTENT状态可以发现，UNDOTBS1 30G处于UNEXPIRED 未过期状态，7G处于正在使用；只有少量数量处于已过期状态；
UNDOTBS2 10G处于UNEXPIRED 未过期状态，0G处于正在使用；2.6G已过期状态；
由于UNDOTBS1 有2个UNDO文件，第一个DB建好，自扩展，当前7，最大31.9G； 第二个DB建后，DBA新增的已完全分配，非自动扩展30G；
本次就是由于UNDOTBS1还有扩展的可能性，因此新的UNDO申请需要对UNDOTBS1的第一个自扩展文件进行扩展时，出现严重的争用；
扩容！！！ SQL> alter tablespace UNDOTBS1 add datafile '+DATA' size 30g;
SQL> alter tablespace UNDOTBS2 add datafile '+DATA' size 10g;
业务反馈突然好了！
收尾
select file_id,TABLESPACE_NAME,bytes/1024/1024/1024,MAXbytes/1024/1024/1024 from dba_data_files where tablespace_name in('UNDOTBS1','UNDOTBS2');
FILE_ID TABLESPACE_NAME BYTES/1024/1024/1024 MAXBYTES/1024/1024/1024
---------- ------------------------------ -------------------- -----------------------
3 UNDOTBS1 7.14355469 31.9999847
5 UNDOTBS2 .024414063 31.9999847
6 UNDOTBS1 30 0
7 UNDOTBS2 30 0
18 UNDOTBS1 30 0
19 UNDOTBS2 10 0
6 rows selected.
select file_name from dba_data_files where file_id in(3,5);
SQL> alter database datafile 3,5 autoextend off;
SQL> alter database datafile '+DATA/wglcta/datafile/undotbs1.258.1047648231' resize 10g;
SQL> alter database datafile '+DATA/wglcta/datafile/undotbs2.264.1047648343' resize 10g;
select file_id,TABLESPACE_NAME,bytes/1024/1024/1024,AUTOEXTENSIBLE,MAXbytes/1024/1024/1024 from dba_data_files where tablespace_name in('UNDOTBS1','UNDOTBS2');
FILE_ID TABLESPACE_NAME BYTES/1024/1024/1024 AUT MAXBYTES/1024/1024/1024
---------- ------------------------------ -------------------- --- -----------------------
3 UNDOTBS1 10 NO 0
5 UNDOTBS2 10 NO 0
6 UNDOTBS1 30 NO 0
7 UNDOTBS2 30 NO 0
18 UNDOTBS1 30 NO 0
19 UNDOTBS2 10 NO 0
GROUP_NUMBER NAME TOTAL_GB FREE_GB FREE_PERCENT STATE TYPE CARE
------------ -------------------- ---------- ---------- ------------ --------------- ------ -----
1 ARCH 500.00 499.87 99.97 CONNECTED EXTERN
2 DATA 500.00 26.89 5.38 CONNECTED EXTERN *
1.4事后追踪
1） 确认UNDOTBS1的使用情况，最近几个小时包括问题时段
SQL> select ts#,name from v$tablespace where name='UNDOTBS1';
TS# NAME
---------- --------------------
2 UNDOTBS1 select * from
DBA_HIST_TBSPC_SPACE_USAGE where TABLESPACE_ID=2 order by RTIME;
SNAP_ID DBID TABLESPACE_ID TABLESPACE_SIZE TABLESPACE_MAXSIZE TABLESPACE_USEDSIZE RTIME
---------- ---------- ------------- --------------- ------------------ ------------------- -------------------------
4728 476729316 2 3942400 8126462 1344368 11/12/2020 01:30:01
4729 476729316 2 3942400 8126462 1147240 11/12/2020 02:00:04
4730 476729316 2 3942400 8126462 1001072 11/12/2020 02:30:10
4731 476729316 2 3942400 8126462 923368 11/12/2020 03:00:12
4732 476729316 2 3942400 8126462 959200 11/12/2020 03:30:01
4733 476729316 2 3942400 8126462 973392 11/12/2020 04:00:02
4734 476729316 2 3942400 8126462 990800 11/12/2020 04:30:03
4735 476729316 2 3942400 8126462 1031128 11/12/2020 05:00:04
4736 476729316 2 3942400 8126462 1087312 11/12/2020 05:30:06
4737 476729316 2 3942400 8126462 1168968 11/12/2020 06:00:07
4738 476729316 2 3942400 8126462 1315032 11/12/2020 06:30:08
4739 476729316 2 3942400 8126462 1544672 11/12/2020 07:00:10
4740 476729316 2 3942400 8126462 1795568 11/12/2020 07:30:11
4741 476729316 2 3942400 8126462 2033392 11/12/2020 08:00:12
4742 476729316 2 3942400 8126462 2305024 11/12/2020 08:30:14
4743 476729316 2 3942400 8126462 2605192 11/12/2020 09:00:15
4744 476729316 2 3942400 8126462 2872568 11/12/2020 09:30:16
4745 476729316 2 3942400 8126462 3098480 11/12/2020 10:00:18
4746 476729316 2 3942400 8126462 3246192 11/12/2020 10:30:06
4747 476729316 2 3942400 8126462 3414576 11/12/2020 11:00:11
4748 476729316 2 3942400 8126462 3577952 11/12/2020 11:30:14
4749 476729316 2 3942400 8126462 3580104 11/12/2020 12:00:17
4750 476729316 2 3942400 8126462 3458504 11/12/2020 12:30:19
4751 476729316 2 3942400 8126462 3300168 11/12/2020 13:00:22
4752 476729316 2 3942400 8126462 3024576 11/12/2020 13:30:28
4753 476729316 2 3942400 8126462 2831064 11/12/2020 14:00:29
4754 476729316 2 3942400 8126462 2698656 11/12/2020 14:30:31
4755 476729316 2 3942400 8126462 2644232 11/12/2020 15:00:32
4756 476729316 2 3942400 8126462 2669832 11/12/2020 15:30:37
4757 476729316 2 3942400 8126462 2780168 11/12/2020 16:00:38
4758 476729316 2 3942400 8126462 2911240 11/12/2020 16:30:40
4759 476729316 2 3942400 8126462 2985720 11/12/2020 17:00:41
4760 476729316 2 3942400 8126462 3577632 11/12/2020 17:30:42
4761 476729316 2 3942400 8126462 3697616 11/12/2020 18:00:44
4762 476729316 2 3942400 8126462 3812664 11/12/2020 18:30:44
4763 476729316 2 4071680 8126462 4055680 11/12/2020 19:01:35
4764 476729316 2 8800640 12058622 4935560 11/12/2020 19:30:12
695 rows selected. 参考
https://yq.aliyun.com/articles/283841 我们查询ASH视图看看当时的情况： SELECT D.SQL_ID,CHR(BITAND(P1, -16777216) / 16777215) ||
CHR(BITAND(P1, 16711680) / 65535) "Lock",
BITAND(P1, 65535) "Mode", COUNT(1),COUNT(DISTINCT d.session_id )
FROM DBA_HIST_ACTIVE_SESS_HISTORY D
WHERE D.SAMPLE_TIME BETWEEN TO_DATE('2020-11-12 16:00:00', 'YYYY-MM-DD HH24:MI:SS') AND
TO_DATE('2020-11-12 19:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND D.EVENT = 'enq: US - contention'
GROUP BY D.SQL_ID,(CHR(BITAND(P1, -16777216) / 16777215) ||
CHR(BITAND(P1, 16711680) / 65535)),(BITAND(P1, 65535)); SQL_ID Lock Mode COUNT(1) COUNT(DISTINCTD.SESSION_ID)
------------- -------- ---------- ---------- ---------------------------
q US 6 5472 1161
c US 6 33943 1340select distinct sql_id,sql_text from v$sql where sql_id in('xx')
order by 1;
2）AWR对比报告，看看能否观察一些其它信息
AWR对比报告
@?/rdbms/admin/awrddrpt.sql
Snapshot Set Begin Snap Id Begin Snap Time End Snap Id End Snap Time Avg Active Users Elapsed Time (min) DB time (min)
------------ -------------- ------------------------- ------------ ------------------------- --------------------------
1st 4761 12-Nov-20 18:00:43 (Thu) 4762 12-Nov-20 18:30:00 (Thu) 542.2 29.3 15,883.8
2nd 4760 12-Nov-20 17:30:41 (Thu) 4761 12-Nov-20 18:00:43 (Thu) 113.0 30.0 3,310.2
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ %Diff: -79.7% 2.5% -79.2%
How to correct performance issues with enq: US - contention related to undo segments (Doc ID 1332738.1)
IF: Undo Wait Event - Enq: US - contention (Doc ID 1951680.1)
Update to latest version or apply the Patch 14226599 (included in 11.2.0.4 patchset and above) and Patch 17720709 (included in 12.2 and above)
Setting _ROLLBACK_SEGMENT_COUNT to a high number to keep undo segments online:
ALTER SYSTEM SET "_rollback_segment_count"=<n>;
Note: In databases with high query activity, particularly parallel query and a high setting for _ROLLBACK_SEGMENT_COUNT, you can expect to see wait contention on the row cache for DC_ROLLBACK_SEGS. It is highly recommended in these environments where setting _ROLLBACK_SEGMENT_COUNT to a high value (10s of thousands and higher) apply the patch for Bug:14226599 (as mentioned in solution 1). This will increase the hash buckets on the DC_ROLLBACK_SEGS row cache to help alleviate latch contention.
A fix to Bug:7291739 is to set a new hidden parameter, _HIGHTHRESHOLD_UNDORETENTION to set a high threshold for undo retention completely distinct from maxquerylen:
ALTER SYSTEM SET "_highthreshold_undoretention"=<n>;
性能问题，AWR High Event enq: US - contention的更多相关文章
- Analyzing 'enq: HW - contention' Wait Event (Doc ID 740075.1)
Analyzing 'enq: HW - contention' Wait Event (Doc ID 740075.1) In this Document Symptoms Cause ...
- enq: TM - contention一例
今天下午,有台服务器出现异常,响应特别慢,io等待奇高,awr top 5事件如下: 经回查ash,找到了造成这些事件的sql语句,如下: select * from v$active_session ...
- 【等待事件】序列等待事件总结（enq: SQ - contention、row cache lock、DFS lock handle和enq: SV - contention）
[等待事件]序列等待事件总结(enq: SQ - contention.row cache lock.DFS lock handle和enq: SV - contention) 1 BLOG文档结 ...
- Oracle 性能之 Enq: CF - contention
Oracle 性能之 Enq: CF - contention Table of Contents 1. 原因 2. 解决问题 2.1. 针对持有锁进程类型处理 2.1.1. 查看持有锁会话的进程类型 ...
- 【故障处理】队列等待之enq IV - contention案例
[故障处理]队列等待之enq IV - contention案例 1.1 BLOG文档结构图 1.2 前言部分 1.2.1 导读和注意事项 各位技术爱好者,看完本文后,你可以掌握如下的技能,也 ...
- 等待事件：enq: HW - contention和enq: TM - contention
今天生成了生产库前几日的AWR报告,发现等待事件中出现了一个陌生的event--enq: HW - contention,google一下是ASSM(Auto Segment Space Manage ...
- enq: SQ - contention
--每分钟操作 SQL> select sql_id, mi, count(mi) 2 from (select event, sql_id, to_char(sample_time ...
- oracle 11g enq: JI – contention等待事件
最近使用物化视图同步的环境在大量刷新的时候频繁出现enq: JI – contention等待事件,经查: JI enqueue is acquired in exclusive mode on th ...
- undo表空间居高不下和enq: US - contention
这几天遇到一个错误,我也不知道算不算错误吧,因为没有报错,只是在那突然的短短2分钟内表的操作突然降低了,导致了该软件重新启动.查看alert日志没有报错,而是在ASH里找到了TOP SQL框有一个这样 ...
随机推荐
- DBA提交脚步规范
工作中需要走脚步流程,申请修改数据库,总结一些常用的语句:)提交时注明为DDL/DML_需求号_日期(各公司标准不一样)//修改字段长度使用;alter table t_task modify tas ...
- 推荐一款IDEA神器！一键查看Java字节码以及其他类信息
由于后面要分享的一篇文章中用到了这篇文章要推荐的一个插件,所以这里分享一下.非常实用!你会爱上它的! 开始推荐 IDEA 字节码查看神器之前,先来回顾一下 Java 字节码是啥. 何为 Java 字节 ...
- 关于Elasticsearch版本升级，Kibana报index迁移与需要x-pack插件问题
关于Elasticsearch版本升级,Kibana报index迁移与需要x-pack插件问题 这个问题是由于elasticsearch旧版残留文件导致,使用下述指令删除即可 查看所有elastics ...
- java 的 callback
Java 本身没有回调这一说,但是面向对象可以模拟出来. 1. 回调接口对象 ICommand package com.git.Cmder; public interface ICommand { v ...
- Java 集合看这一篇就够了
大家好,这里是<齐姐聊数据结构>系列之大集合. 话不多说,直接上图: Java 集合,也称作容器,主要是由两大接口 (Interface) 派生出来的: Collection 和 Map ...
- day32 Pyhton 异常处理
一.内容回顾 反射的另外两个内置函数 setattr delattr a.b=c 与 setattr(a,'b',c)相对 del a.b 与 delattr(a,'b') 两个内置函数 A,B(A) ...
- canal 整合RabbitMQ
环境如下: canal: 1.15-alpha-1 mysql 5.6.49 rabbitmq 3.7.14 Erlang 21.3 canal 安装和启动 见上篇文章 canal快速安装启动 但是 ...
- 对接天猫精灵X1 (https 的申请)
1 起因 公司是做智能家居的,最近公司要求对接天猫精灵的智能家居功能,所以就来对接天猫精灵X1 了. 新产品,大家都懂的,坑是有不少的,正常事. 1 首先,语言是 c#,不要和我讲 php 是世界最好 ...
- MVC注册
前言 最近没什么写的,写个MVC注册巩固一下 HTML @{ Layout = null; } <!DOCTYPE html> <html> <head> < ...
- 通过express快速搭建一个node服务
Node.js 是一个基于Chrome JavaScript 运行时建立的一个平台.可以理解为是运行在服务端的 JavaScript.如果你是一个前端程序员,不太擅长像PHP.Python或Ruby等 ...