用oradebug short_stack及strace -p分析oracle进程是否dead或出现故障

1,可以采用oradebug或者strace -p跟踪后台或前台进程是否dead或hang住
2,如果进程出现故障，必会在对应的TRC文件写入最新信息，基于此可以获取非常重要的信息进一步分析与诊断
   日志文件在background_dump_dest
3,采用 ll -lhrt *lgwr*|tail -10f  获取最新的进程的TRC文件
4,而且出现故障时，多半会在ALERT日志记录相关信息，此是排除故障重要且首要的方法及思路
5,oradebug setospid ospid
  oradebug short_stack
  会显示进程的堆栈信息，注意：可以间隔多次运行，如果多次显示的堆栈信息一致，可以肯定此进程肯定是dead或出现故障了

6，可以用strace -p ospid跟踪分析，

---hang或故障时的类似信息如下
semtimedop(9273344, 0x7fffe66199d0, 1, {1, 0}) = -1 EAGAIN (Resource temporarily unavailable)

---正常时的类似信息如下
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440015944
semtimedop(9273344, 0x7fffe661b1f0, 1, {1, 800000000}) = -1 EAGAIN (Resource temporarily unavailable)
getrusage(RUSAGE_SELF, {ru_utime={0, 123981}, ru_stime={0, 132979}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={0, 123981}, ru_stime={0, 132979}, ...}) = 0
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016124
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016124
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016124
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016124
semtimedop(9273344, 0x7fffe661b1f0, 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)
getrusage(RUSAGE_SELF, {ru_utime={0, 123981}, ru_stime={0, 132979}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={0, 123981}, ru_stime={0, 132979}, ...}) = 0
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016424
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016424
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016424
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016424
semtimedop(9273344, 0x7fffe661b1f0, 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)
getrusage(RUSAGE_SELF, {ru_utime={0, 123981}, ru_stime={0, 132979}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={0, 123981}, ru_stime={0, 132979}, ...}) = 0
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016725
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016725
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016725
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016725
semtimedop(9273344, 0x7fffe661b1f0, 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)
getrusage(RUSAGE_SELF, {ru_utime={0, 123981}, ru_stime={0, 132979}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={0, 123981}, ru_stime={0, 132979}, ...}) = 0
times({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440017025
open("/proc/4385/stat", O_RDONLY)       = 35
read(35, "4385 (oracle) S 1 4385 4385 0 -1"..., 999) = 225

说白了，就是看信息有没有变化，有变化就说明进程是正常的，否则就说明是不正常的

测试

SQL> select * from v$version where rownum=1;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

查看后台进程
SQL> select pid,spid,pname,username from v$process order by 1;

PID SPID       PNAME      USERNAME
---------- ---------- ---------- ------------------------------
         1
         2 4385       PMON       oracle
         3 4387       VKTM       oracle
         4 4391       GEN0       oracle
         5 4393       DIAG       oracle
         6 4395       DBRM       oracle
         7 4397       PSP0       oracle
         8 4399       DIA0       oracle
         9 4401       MMAN       oracle
        10 4403       DBW0       oracle
        11 4405       LGWR       oracle

PID SPID       PNAME      USERNAME
---------- ---------- ---------- ------------------------------
        12 4407       CKPT       oracle
        13 4409       SMON       oracle
        14 4411       RECO       oracle
        15 4413       MMON       oracle
        16 4415       MMNL       oracle
        17 4417       D000       oracle
        18 4419       S000       oracle
        19 4652       SMCO       oracle
        20 5266       W000       oracle
        21 4936                  oracle
        27 4468       ARC0       oracle
       PID SPID       PNAME      USERNAME
---------- ---------- ---------- ------------------------------
        28 4481       ARC1       oracle
        29 4486       ARC2       oracle
        30 4489       ARC3       oracle
        31 4496       QMNC       oracle
        32 4549       Q000       oracle
        33 4551       Q001       oracle
        34 4568                  oracle

29 rows selected.

SQL> 
---查看TRC文件目录
[oracle@seconary trace]$ ll -lhrt *lgwr*|tail -10f
-rw-r----- 1 oracle oinstall  213 Dec 14 19:05 guowang_lgwr_5297.trm
-rw-r----- 1 oracle oinstall 2.4K Dec 14 19:05 guowang_lgwr_5297.trc
-rw-r----- 1 oracle oinstall 2.3K Dec 15 01:05 guowang_lgwr_22295.trm
-rw-r----- 1 oracle oinstall  27K Dec 15 01:05 guowang_lgwr_22295.trc
-rw-r----- 1 oracle oinstall   63 Dec 15 02:18 guowang_lgwr_31280.trm
-rw-r----- 1 oracle oinstall  903 Dec 15 02:18 guowang_lgwr_31280.trc
-rw-r----- 1 oracle oinstall   63 Dec 15 02:44 guowang_lgwr_32077.trm
-rw-r----- 1 oracle oinstall  906 Dec 15 02:44 guowang_lgwr_32077.trc
-rw-r----- 1 oracle oinstall   62 Dec 15 03:27 guowang_lgwr_1032.trm
-rw-r----- 1 oracle oinstall  887 Dec 15 03:27 guowang_lgwr_1032.trc

---HANG LGWR
SQL> oradebug setospid 4405
Oracle pid: 11, Unix process pid: 4405, image: oracle@seconary (LGWR)
SQL> oradebug suspend
Statement processed.

--ALERT同步记录上述信息
Tue Dec 15 04:46:15 2015
Unix process pid: 4405, image: oracle@seconary (LGWR) flash frozen [ command #1 ]

---TRC目录同步记录上述信息
[oracle@seconary trace]$ ll -lhrt *lgwr*|tail -10f
-rw-r----- 1 oracle oinstall 2.3K Dec 15 01:05 guowang_lgwr_22295.trm
-rw-r----- 1 oracle oinstall  27K Dec 15 01:05 guowang_lgwr_22295.trc
-rw-r----- 1 oracle oinstall   63 Dec 15 02:18 guowang_lgwr_31280.trm
-rw-r----- 1 oracle oinstall  903 Dec 15 02:18 guowang_lgwr_31280.trc
-rw-r----- 1 oracle oinstall   63 Dec 15 02:44 guowang_lgwr_32077.trm
-rw-r----- 1 oracle oinstall  906 Dec 15 02:44 guowang_lgwr_32077.trc
-rw-r----- 1 oracle oinstall   62 Dec 15 03:27 guowang_lgwr_1032.trm
-rw-r----- 1 oracle oinstall  887 Dec 15 03:27 guowang_lgwr_1032.trc
-rw-r----- 1 oracle oinstall   63 Dec 15 04:46 guowang_lgwr_4405.trm
-rw-r----- 1 oracle oinstall  896 Dec 15 04:46 guowang_lgwr_4405.trc
[oracle@seconary trace]$