数据库hang住 处理过程

当数据库出现严重的性能问题或者hang了的时候，我们非常需要通过systemstate dump来知道进程在做什么，在等待什么，谁是资源的持有者，谁阻塞了别人。在出现上述问题时，及时收集systemstate dump非常有助于问题原因的分析。
在一些情况下，数据库会自动生成systemstate dump, 比如出现了“WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK”。
systemstate dump大部分时候需要手工生成，具体的命令为：

（如果连接很多，比如几千个连接，那么生成dump可能需要几十分钟，而且会占用几百M磁盘空间）

1. 用sysdba登录到数据库上：
$sqlplus / as sysdba
或者
$sqlplus -prelim / as sysdba <==当数据库已经很慢或者hang到无法连接

SQL>oradebug setmypid
SQL>oradebug unlimit;
SQL>oradebug dump systemstate 266;
等1~2分钟
SQL>oradebug dump systemstate 266;
等1~2分钟
SQL>oradebug dump systemstate 266;
SQL>oradebug tracefile_name;==>这是生成的文件名

2. 通常除了systemstate dump，最好同时生成hang analyze来直观地了解数据库进程间的等待关系。

$sqlplus / as sysdba
或者
$sqlplus -prelim / as sysdba <==当数据库已经很慢或者hang到无法连接

SQL>oradebug setmypid
SQL>oradebug unlimit;
SQL>oradebug dump hanganalyze 3
等1~2分钟
SQL>oradebug dump hanganalyze 3
等1~2分钟
SQL>oradebug dump hanganalyze 3
SQL>oradebug tracefile_name;==>这是生成的文件名

对于RAC数据库，需要各个实例在同一时间的systemstate dump,那么登录到任意一个实例（无需在所有实例执行）：

$sqlplus / as sysdba
或者
$sqlplus -prelim / as sysdba <==当数据库已经很慢或者hang到无法连接

SQL>oradebug setmypid
SQL>oradebug unlimit
SQL>oradebug -g all dump systemstate 266 <==-g all 表示针对所有实例生成dump
等1~2分钟
SQL>oradebug -g all dump systemstate 266
等1~2分钟
SQL>oradebug -g all dump systemstate 266

在RAC上生成hang analyze:
SQL>oradebug setmypid
SQL>oradebug unlimit
SQL>oradebug -g all hanganalyze 3
等1~2分钟
SQL>oradebug -g all hanganalyze 3
等1~2分钟
SQL>oradebug -g all hanganalyze 3

上面的命令执行后会在每个实例都生成systemstate dump，生成的信息放到了每个实例的backgroud_dump_dest下的diag trace文件中。

上面的这些命令执行三次是为了比较进程的变化情况，查看是真的hang了，还是很慢。

systemstate dump有多个级别：
2: dump (不包括lock element)
10: dump
11: dump + global cache of RAC
256: short stack （函数堆栈）
258: 256+2 -->short stack +dump(不包括lock element)
266: 256+10 -->short stack+ dump
267: 256+11 -->short stack+ dump + global cache of RAC

level 11和 267会 dump global cache, 会生成较大的trace 文件，一般情况下不推荐。

一般情况下，如果进程不是太多，推荐用266，因为这样可以dump出来进程的函数堆栈，可以用来分析进程在执行什么操作。
但是生成short stack比较耗时，如果进程非常多，比如2000个进程，那么可能耗时30分钟以上。这种情况下，可以生成level 10 或者 level 258， level 258 比 level 10会多收集short short stack, 但比level 10少收集一些lock element data.

另外对于RAC系统，请关注Bug 11800959 - A SYSTEMSTATE dump with level >= 10 in RAC dumps huge BUSY GLOBAL CACHE ELEMENTS - can hang/crash instances (Doc ID 11800959.8)。这个Bug在11.2.0.3上被修复，对于<=11.2.0.2的RAC，当系统中的lock element 很多的时候，如果执行level 10、266或者 267的systemstate dump时，可能会导致数据库hang或者crash，这种情况下可以采用level 258。

下面是生成systemstate dump的测试，用来查看每个level占用的空间：

这个例子中有37个进程：

-rw-r----- 1 oracle oinstall 72721 Aug 31 21:50 rac10g2_ora_31092.trc==>256 (short stack, 每个进程2K)

-rw-r----- 1 oracle oinstall 2724863 Aug 31 21:52 rac10g2_ora_31654.trc==>10 (dump，每个进程72K )
-rw-r----- 1 oracle oinstall 2731935 Aug 31 21:53 rac10g2_ora_32214.trc==>266 (dump + short stack ，每个进程72K)

RAC:
-rw-r----- 1 oracle oinstall 55873057 Aug 31 21:49 rac10g2_ora_30658.trc ==>11 (dump+global cache，每个进程1.4M)
-rw-r----- 1 oracle oinstall 55879249 Aug 31 21:48 rac10g2_ora_28615.trc ==>267 (dump+global cache+short stack，每个进程1.4M)

所以，可以看出如果dump global cache(level 11和267，那么占用的空间比其他级别大很多）。

补充一点:
+++ 在一些非常大SGA或者非常多server process的系统里面,如果level 266花的时间比较久,我们可以收集level 258.它比level 10会多收集callstack,比level 266少收集一些lock element data.
而callstack在很多时候会很有帮助.

1. vmstat的第一行输出没有太大的参考意义,( OSWbb里面vmstat是取3行值的),要看第二行或者之后的
2. 当sqlplus -prelim / as sysdba也hung住,我们可以使用OS的gdb命令来生成systemstate dump.如

$ps -ef |grep $ORACLE_SID ## Use PID of any User Process instead of BG Process
$gdb $ORACLE_HOME/bin/oracle <PID_from_earlier_step>
print ksudss(258)

文档359536.1中有描述

Thank you, Feng Gao

是的，如Gao Feng的回复，我们可以用gdb或dbx来搜集:
比如，我做了一个例子：
$ ps -ef|grep R11202
oracle 28288 1 0 04:42 ? 00:00:00 ora_pmon_R11202
oracle 28290 1 0 04:42 ? 00:00:00 ora_psp0_R11202
...

$ gdb $ORACLE_HOME/bin/oracle 28288
GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...(no debugging symbols found)
Using host libthread_db library "/lib64/tls/libthread_db.so.1".

Attaching to program: /opt/oracle/products/11.2.0.2/bin/oracle, process 28288
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libodm11.so...(no debugging symbols found)...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libodm11.so
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libcell11.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libcell11.so
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libskgxp11.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libskgxp11.so
Reading symbols from /lib64/tls/librt.so.1...done.
Loaded symbols for /lib64/tls/librt.so.1
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libnnz11.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libnnz11.so
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libclsra11.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libclsra11.so
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libdbcfg11.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libdbcfg11.so
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libhasgen11.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libhasgen11.so
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libskgxn2.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libskgxn2.so
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libocr11.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libocr11.so
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libocrb11.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libocrb11.so
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libocrutl11.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libocrutl11.so
Reading symbols from /usr/lib64/libaio.so.1...done.
Loaded symbols for /usr/lib64/libaio.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/tls/libm.so.6...done.
Loaded symbols for /lib64/tls/libm.so.6
Reading symbols from /lib64/tls/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 182919795136 (LWP 28288)]
Loaded symbols for /lib64/tls/libpthread.so.0
Reading symbols from /lib64/libnsl.so.1...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/tls/libc.so.6...done.
Loaded symbols for /lib64/tls/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libnuma.so...done.
Loaded symbols for /usr/lib64/libnuma.so
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /opt/oracle/products/11.2.0.2/hpatch/bug10269503.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/hpatch/bug10269503.so
Reading symbols from /opt/oracle/products/11.2.0.2/lib/libnque11.so...done.
Loaded symbols for /opt/oracle/products/11.2.0.2/lib/libnque11.so
0x00000034de6c0f6f in poll () from /lib64/tls/libc.so.6
(gdb) print ksudss(10)
[Switching to Thread 182919795136 (LWP 28288)]

然后查看这个进程的trace 文件：
$ more R11202_pmon_28288.trc
Trace file /opt/oracle/diag/rdbms/r11202/R11202/trace/R11202_pmon_28288.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /opt/oracle/products/11.2.0.2
System name: Linux
Node name: nascds18.cn.oracle.com
Release: 2.6.9-89.0.0.0.1.ELsmp
Version: #1 SMP Tue May 19 05:31:20 EDT 2009
Machine: x86_64
Instance name: R11202
Redo thread mounted by this instance: 1
Oracle process number: 2
Unix process pid: 28288, image: oracle@nascds18.cn.oracle.com (PMON)

*** 2012-12-15 04:43:30.589
*** SESSION ID:(2.1) 2012-12-15 04:43:30.589
*** CLIENT ID:() 2012-12-15 04:43:30.589
*** SERVICE NAME:(SYS$BACKGROUND) 2012-12-15 04:43:30.589
*** MODULE NAME:() 2012-12-15 04:43:30.589
*** ACTION NAME:() 2012-12-15 04:43:30.589

===================================================
SYSTEM STATE (level=10)
------------
System global information:
processes: base 0x9f63ebe0, size 250, cleanup 0x9f661850
allocation: free sessions 0x9f77ef88, free calls (nil)
control alloc errors: 0 (process), 0 (session), 0 (call)
PMON latch cleanup depth: 0
seconds since PMON's last scan for dead processes: 65
system statistics:
0 OS CPU Qt wait time
41 logons cumulative

文档359536.1提到 （Use PID of any User Process instead of BG Process）

Stone, 欢迎加入中文社区。:)
是的，如你所说，我们有一个内部工具可以对systemstate dump做分析，它可以把每个进程在等待什么，争用的资源是什么，资源的持有者是谁列出来。这样可以对数据库的等待情况有个大概的了解，接下来再对 waiter 进程或者 holder 进程具体查看trace 文件，分析这个进程在做什么。

这个工具只是Oracle内部使用，外部无法下载。

不过没有这个工具，一样可以分析systemstate dump,常见的一些问题，比如：

"WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! "，下面的文档中有从systemstat dump中找到holder的具体方法：
Note 278316.1 Troubleshooting: "WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! "

找到"Library Cache Lock"的持有者：
Note 122793.1 How to Find which Session is Holding a Particular Library Cache Lock

找到"cursor: pin S wait on X"的持有者：
1. 比如 PROCESS 30在等待 'cursor: pin S wait on X'：

PROCESS 30:
......
waiting for 'cursor: pin S wait on X' blocking sess=0x0 seq=4209 wait_time=0 seconds since wait started=0
idn=76572938, value=51d00000000, where|sleeps=50011ea9c

2. 全文搜索“idn 76572938 oper EXCL”， 然后找到了下面的信息：

PROCESS 20:
......
Mutex 7000001ee8b8d09(1309, 0) idn 76572938 oper EXCL 《==排它地持有了
Cursor Pin uid 1309 efd 0 whr 1 slp 0
opr=3 pso=7000001533652f8 flg=0

3. 在往回搜索“PROCESS "，就会找到 PROCESS 20，说明这个持有者是PROCESS 20. 接下来要分析PROCESS 20在执行什么，为什么阻塞了其他进程。