OGG抽取进程异常问题排查一例

1、问题现象
抽取进程常常running，但是没有新产生trail文件，lag比较大

GGSCI (xxxdb) > info all
Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     EXT1        ::      ::
EXTRACT     RUNNING     PUMP1       ::      ::
GGSCI (xxxdb) >
GGSCI (xxxdb) > info EXT1,showch

EXTRACT    EXT1      Last Started -- :   Status RUNNING
Checkpoint Lag       :: (updated :: ago)
Log Read Checkpoint  Oracle Redo Logs
                     -- ::  Seqno , RBA
                     SCN 3439.4180128660 ()

-- ::  INFO    OGG-  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (dtogg): start EXT1.
-- ::  INFO    OGG-  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from GGSCI on host pacsdb (START EXTRACT EXT1 ).
-- ::  INFO    OGG-  Oracle GoldenGate Manager for Oracle, mgr.prm:  EXTRACT EXT1 starting.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  EXTRACT EXT1 starting.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Operating system character set identified as ISO--. Locale: en
_US, LC_ALL:.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Virtual Memory Facilities for: BR
    anon alloc: mmap(MAP_ANON)  anon free: munmap
    file alloc: mmap(MAP_SHARED)  file free: munmap
    target directories:
    /oracle/10.2/ogg/BR/EXT1.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Virtual Memory Facilities for: COM
    anon alloc: mmap(MAP_ANON)  anon free: munmap
    file alloc: mmap(MAP_SHARED)  file free: munmap
    target directories:
    /oracle/10.2/ogg/dirtmp.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  BOUNDED RECOVERY:  ACTIVE: for object pool : p48169162_extr.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  BOUNDED RECOVERY: recovery start XID: 117.11..
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  BOUNDED RECOVERY: recovery start position: SeqNo: , RBA:
, SCN: 3439.4175624574 (), Timestamp: -- ::55.000000.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  BOUNDED RECOVERY: recovery end position: SeqNo: , RBA:
, SCN: 3439.4180119353 (), Timestamp: -- ::01.000000, Thread: .
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  BOUNDED RECOVERY: VALID BCP: CP.EXT1..
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  BOUNDED RECOVERY: PERSISTED OBJECTS RECOVERED: .
-- ::  INFO    OGG-  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (dtogg):  start ext1.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Positioning to Sequence , RBA , SCN 3439.4175624574.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Positioned to Sequence , RBA , SCN 3439.4175624574, May
 ,  :: PM.

2、问题解决
通过 info ext1 showch 查看抽取进程的Recovery Checkpoint，设置抽取进程到相应时间

GGSCI (xxxdb) > alter extract EXT1 ,tranlog ,begin -- ::
EXTRACT altered.

GGSCI (xxxdb) > start EXT1

Sending START request to MANAGER ...
EXTRACT EXT1 starting

GGSCI (xxxdb) > info EXT1

EXTRACT    EXT1      Initialized   -- :   Status RUNNING
Checkpoint Lag       :: (updated :: ago)
Log Read Checkpoint  Oracle Redo Logs
                     -- ::  Seqno , RBA
                     SCN 0.0 ()

GGSCI (xxxdb) > info EXT1

EXTRACT    EXT1      Last Started -- :   Status RUNNING
Checkpoint Lag       :: (updated :: ago)
Log Read Checkpoint  Oracle Redo Logs
                     -- ::  Seqno , RBA
                     SCN 3439.4180220954 ()

GGSCI (xxxdb) > info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     EXT1        ::     ::
EXTRACT     RUNNING     PUMP1       ::     ::    

GGSCI (xxxdb) >  info EXT1

EXTRACT    EXT1      Last Started -- :   Status RUNNING
Checkpoint Lag       :: (updated :: ago)
Log Read Checkpoint  Oracle Redo Logs
                     -- ::  Seqno , RBA
                     SCN 3439.4180340086 ()

-- ::  INFO    OGG-  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (dtogg): alter extract EXT1  tranlog ,begin -
- ::.
-- ::  INFO    OGG-  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (dtogg): start EXT1.
-- ::  INFO    OGG-  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from GGSCI on host pacsdb (START EXTRACT EXT1 ).
-- ::  INFO    OGG-  Oracle GoldenGate Manager for Oracle, mgr.prm:  EXTRACT EXT1 starting.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  EXTRACT EXT1 starting.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Operating system character set identified as ISO--. Locale: en
_US, LC_ALL:.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  BOUNDED RECOVERY: reset to initial or altered checkpoint.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Virtual Memory Facilities for: BR
    anon alloc: mmap(MAP_ANON)  anon free: munmap
    file alloc: mmap(MAP_SHARED)  file free: munmap
    target directories:
    /oracle/10.2/ogg/BR/EXT1.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Virtual Memory Facilities for: COM
    anon alloc: mmap(MAP_ANON)  anon free: munmap
    file alloc: mmap(MAP_SHARED)  file free: munmap
    target directories:
    /oracle/10.2/ogg/dirtmp.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Positioning to begin time May ,  :: PM.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Positioned to Sequence , RBA , SCN 0.0, May ,
:: PM.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  EXTRACT EXT1 started.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Recovery initialization completed for target file /oradata/ogg/dird
at/e1002586, at RBA .
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Output file /oradata/ogg/dirdat/e1 is using format RELEASE 11.2.
-- ::  WARNING OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Checkpoint marked as from graceful shutdown, but records found afte
r checkpoint in trail /oradata/ogg/dirdat/e1.  Expected EOF Seqno , RBA .  Found Seqno , RBA .
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Rolling over remote file /oradata/ogg/dirdat/e1002586.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Recovery completed for target file /oradata/ogg/dirdat/e1002587, at
 RBA .
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Recovery completed for all targets.
-- ::  INFO    OGG-  Oracle GoldenGate Capture for Oracle, ext1.prm:  Position of first record processed Sequence , RBA , SC
N 3439.4180128606, May ,  :: PM.

shell下通过下列脚本过滤出OGG进程状态，留作监控使用

(echo info all;echo exit)|./ggsci|grep -E "EXTRACT|REPLICAT" |awk 'BEGIN {FS=" +"} {print $1","$3","$2","$4","$5}'