DataGuard备库ORA-01196故障恢复一则

问题现象

在使用shutdown abort停DataGuard备库后。备库不能open，报ORA-01196错误。

详细例如以下：

发现一备库不能应用日志。查看备库日志没发现报错。怀疑是备库应用日志服务停止，于是尝试重新启动备库。

可能由于备库是读业务比較繁忙，在shutdown immediate关闭备库时等时间过长，于是使用了shutdown abort命令；

但后面在启动备库时发生报错，造成数据文件损坏，控制文件和数据文件的scn号不一致。

--启动备库时报错

SQL> startup

ORACLE 例程已经启动。

Total System Global Area 2.0310E+10 bytes

Fixed Size                  2235256 bytes

Variable Size            9328133256 bytes

Database Buffers         1.0939E+10 bytes

Redo Buffers               40894464 bytes

数据库装载完成。

ORA-10458: standby database requiresrecovery

ORA-01196: 文件 1 因为介质恢复会话失败而不一致

ORA-01110: 数据文件 1:'+DATA/htdb5/datafile/system.261.759082693'

--查看日志

alter database open

Data Guard Brokerinitializing...

Data Guard Brokerinitialization complete

Beginning standby crash recovery.

Serial Media Recovery started

Managed Standby Recoverystarting Real Time Apply

Media Recovery Log+FRA/htdb5/archivelog/2015_07_16/thread_1_seq_180068.1541.885192077

Thu Jul 16 12:00:47 2015

Errors in file/u01/app/ora11g/diag/rdbms/htdb5/htdb5/trace/htdb5_ora_10154.trc:

ORA-01013: 用户请求取消当前的操作

ORA-10567: Redo is inconsistentwith data block (file# 47, block# 1187724, file offset is 1139900416 bytes)

ORA-10564: tablespace JDYWP_IDX

ORA-01110: 数据文件 47:'+DATA/htdb5/datafile/jdywp_idx.336.856967805'

ORA-10561: block type'TRANSACTION MANAGED INDEX BLOCK', data object# 251837

Errors in file/u01/app/ora11g/diag/rdbms/htdb5/htdb5/trace/htdb5_ora_10154.trc:

ORA-00339: 归档日志未包括不论什么重做

ORA-00334: 归档日志: '+DATA/htdb5/onlinelog/group_2.280.759082845'

ORA-10567: Redo is inconsistentwith data block (file# 47, block# 1187724, file offset is 1139900416 bytes)

ORA-10564: tablespace JDYWP_IDX

ORA-01110: 数据文件 47:'+DATA/htdb5/datafile/jdywp_idx.336.856967805'

ORA-10561: block type'TRANSACTION MANAGED INDEX BLOCK', data object# 251837

Errors in file/u01/app/ora11g/diag/rdbms/htdb5/htdb5/trace/htdb5_ora_10154.trc  (incident=116743):

ORA-00600: 内部错误代码, 參数: [3020],[47], [1187724], [198320012], [], [], [], [], [], [], [], []

ORA-10567: Redo is inconsistentwith data block (file# 47, block# 1187724, file offset is 1139900416 bytes)

ORA-10564: tablespace JDYWP_IDX

ORA-01110: 数据文件 47:'+DATA/htdb5/datafile/jdywp_idx.336.856967805'

ORA-10561: block type'TRANSACTION MANAGED INDEX BLOCK', data object# 251837

Incident details in:/u01/app/ora11g/diag/rdbms/htdb5/htdb5/incident/incdir_116743/htdb5_ora_10154_i116743.trc

Use ADRCI or Support Workbenchto package the incident.

See Note 411.1 at My OracleSupport for error and packaging details.

Standby crash recovery aborteddue to error 600.

Errors in file/u01/app/ora11g/diag/rdbms/htdb5/htdb5/trace/htdb5_ora_10154.trc:

ORA-00600: 内部错误代码, 參数: [3020],[47], [1187724], [198320012], [], [], [], [], [], [], [], []

ORA-10567: Redo is inconsistentwith data block (file# 47, block# 1187724, file offset is 1139900416 bytes)

ORA-10564: tablespace JDYWP_IDX

ORA-01110: 数据文件 47:'+DATA/htdb5/datafile/jdywp_idx.336.856967805'

ORA-10561: block type'TRANSACTION MANAGED INDEX BLOCK', data object# 251837

Recovery interrupted!

Some recovered datafiles maybeleft media fuzzy

Media recovery may continue butopen resetlogs may fail

Completed standby crashrecovery.

Errors in file/u01/app/ora11g/diag/rdbms/htdb5/htdb5/trace/htdb5_ora_10154.trc:

ORA-10458: standby databaserequires recovery

ORA-01196: 文件 1 因为介质恢复会话失败而不一致

ORA-01110: 数据文件 1:'+DATA/htdb5/datafile/system.261.759082693'

ORA-10458 signalled during:alter database open...

Thu Jul 16 12:00:49 2015

Sweep [inc][116743]: completed

Sweep [inc2][116743]: completed

Thu Jul 16 12:00:49 2015

Dumping diagnostic data indirectory=[cdmp_20150716120049], requested by (instance=1, osid=10154),summary=[incident=116743].

Thu Jul 16 12:01:50 2015

解决的方法：

把备库闪回到正常的状态的时点。

--前提数据库闪回之前已经打开

SQL> select FLASHBACK_ON from v$database;

FLASHBACK_ON

------------------

YES

SQL> Flashback database to timestamp to_timestamp('2015-07-16 4:00:05','yyyy-mm-ddhh24:mi:ss');

--或是使用Flashbackdatabase to scn 947921

SQL> alter database open;

SQL> select open_mode from v$database;

OPEN_MODE

--------------------

READ ONLY

--启动实时应用

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;

SQL> select open_mode from v$database;

OPEN_MODE

--------------------

READ ONLY WITH APPLY

--查看日志看到日志已经从闪回的时点開始应用

Thu Jul 16 13:36:01 2015

Flashback database to timestampto_timestamp('2015-07-16 4:00:05','yyyy-mm-dd hh24:mi:ss')

Flashback Restore Start

Thu Jul 16 13:39:30 2015

Flashback Restore Complete

Flashback Media Recovery Start

started logmerger process

Parallel Media Recovery startedwith 16 slaves

Flashback Media Recovery Log+FRA/htdb5/archivelog/2015_07_16/thread_1_seq_180047.2212.885180637

Thu Jul 16 13:41:54 2015

Flashback Media Recovery Log+FRA/htdb5/archivelog/2015_07_16/thread_1_seq_180061.2611.885182343

Thu Jul 16 13:42:04 2015

Flashback Media Recovery Log+FRA/htdb5/archivelog/2015_07_16/thread_1_seq_180062.2861.885182537

Thu Jul 16 13:42:12 2015

Incomplete Recovery applieduntil change 71489772016 time 07/16/2015 04:00:06

Flashback Media RecoveryComplete

Completed: Flashback databaseto timestamp to_timestamp('2015-07-16 4:00:05','yyyy-mm-dd hh24:mi:ss')

Thu Jul 16 13:43:25 2015

Deleted Oracle managed file+FRA/htdb5/archivelog/2015_07_15/thread_1_seq_179690.2885.885083087

Thu Jul 16 13:43:25 2015

Standby controlfile consistentwith primary

RFS[3]: Selected log 8 forthread 1 sequence 180122 dbid 1083719948 branch 759079182

Archived Log entry 180115 addedfor thread 1 sequence 180121 ID 0x40a48484 dest 1:

Thu Jul 16 13:45:41 2015

alter database open

Data Guard Brokerinitializing...

Data Guard Brokerinitialization complete

SMON: enabling cache recovery

Dictionary check beginning

Dictionary check complete

Database Characterset isZHS16GBK

No Resource Manager plan active

replication_dependency_trackingturned off (no async multimaster replication found)

Physical standby databaseopened for read only access.

Completed: alter database open

Thu Jul 16 13:45:44 2015

ALTER DATABASE RECOVER MANAGEDSTANDBY DATABASE  THROUGH ALL SWITCHOVERDISCONNECT  USING CURRENT LOGFILE

Attempt to start backgroundManaged Standby Recovery process (htdb5)

Thu Jul 16 13:45:44 2015

MRP0 started with pid=51, OSid=14743

MRP0: Background ManagedStandby Recovery process started (htdb5)

started logmerger process

Thu Jul 16 13:45:50 2015

Managed Standby Recoverystarting Real Time Apply

Parallel Media Recovery startedwith 16 slaves

Waiting for all non-currentORLs to be archived...

All non-current ORLs have beenarchived.

Media Recovery Log +FRA/htdb5/archivelog/2015_07_16/thread_1_seq_180062.2861.885182537

Completed: ALTER DATABASERECOVER MANAGED STANDBY DATABASE  THROUGHALL SWITCHOVER DISCONNECT  USING CURRENTLOGFILE

Thu Jul 16 13:46:08 2015

Media Recovery Log+FRA/htdb5/archivelog/2015_07_16/thread_1_seq_180063.3683.885182777

Thu Jul 16 13:46:35 2015

Media Recovery Log+FRA/htdb5/archivelog/2015_07_16/thread_1_seq_180064.2542.885183119

Thu Jul 16 13:47:07 2015

Media Recovery Log+FRA/htdb5/archivelog/2015_07_16/thread_1_seq_180065.2717.885183615