由于坏块导致DG的mrp0进程中断

由于坏块导致DG的mrp0进程中断
1.Environment
11.2.0.4 ADG

2.Symptoms
DG的mrp0进程中断，重启后，还是卡住。查看alert，在1.47存在报错，进一步查看trc，发现0:09存在错误，和mrp0进程卡住的时间吻合。
11.2.0.4 adg
ORA-00600 [3020], [2393], [206563], [2189633251]
ORA-10567
ORA-10564
ORA-01110
ORA-10561

Sun Sep 26 01:47:41 2021
Archived Log entry 1219914 added for thread 1 sequence 1413483 ID 0x64e21606 dest 1:
Sun Sep 26 01:47:52 2021
Errors in file /data/app/oracle/diag/rdbms/XXX/XXX1/trace/XXX1_pr1q_21233804.trc (incident=556535):
ORA-00600: internal error code, arguments: [3020], [2393], [206563], [2189633251], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 2393, block# 206563, file offset is unknown bytes)
ORA-10564: tablespace LIFEINDEX_T_L
ORA-01110: data file 2393: '+DATA/XXX/datafile/lifeindex_t_l.15681.1083761543'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 249021
Incident details in: /data/app/oracle/diag/rdbms/XXX/XXX1/incident/incdir_556535/XXX1_pr1q_21233804_i556535.trc
Sun Sep 26 01:48:04 2021
RFS[10]: Selected log 111 for thread 1 sequence 1413485 dbid 922980069 branch 789829078
Sun Sep 26 01:48:05 2021
Archived Log entry 1219915 added for thread 1 sequence 1413484 ID 0x64e21606 dest 1:
RFS[10]: Selected log 112 for thread 1 sequence 1413486 dbid 922980069 branch 789829078
Sun Sep 26 01:48:15 2021
Archived Log entry 1219916 added for thread 1 sequence 1413485 ID 0x64e21606 dest 1:
Sun Sep 26 01:48:21 2021
RFS[25]: Selected log 122 for thread 2 sequence 1327788 dbid 922980069 branch 789829078

3.Changes
无

4.Cause
出现了坏块

5.Solution
看trc的报错是00:09的时候，刚好是mrp0卡住的时间，应该逻辑坏块导致进程卡住了。和这篇mos能匹配上，查到是一个主键索引，可以跳过去

6.References
Stuck recovery of database ORA-00600[3020] (Doc ID 283269.1)
Resolving ORA-00752 or ORA-600 [3020] During Standby Recovery (Doc ID 1265884.1)

匹配
SQL> select name,value,unit,time_computed from v$dataguard_stats where name in ('transport lag','apply lag');

NAME VALUE UNIT TIME_COMPUTED
-------------------- -------------------- ------------------------------ ------------------------------
transport lag +00 00:00:00 day(2) to second(0) interval 09/26/2021 09:33:16
apply lag +00 09:24:12 day(2) to second(0) interval 09/26/2021 09:33:16

SELECT SEGMENT_NAME FROM DBA_EXTENTS WHERE FILE_ID= &file_number
AND &BLOCK_NUMBER BETWEEN BLOCK_ID AND BLOCK_ID+BLOCKS-1;

SEGMENT_NAME
---------------------------------------------------------------------------------
PK_T_XXXXXXXXODUCT

SELECT OWNER, OBJECT_NAME, SUBOBJECT_NAME, OBJECT_TYPE
FROM DBA_OBJECTS
WHERE DATA_OBJECT_ID = &Object_number;

SELECT OWNER, OBJECT_NAME, SUBOBJECT_NAME, OBJECT_TYPE
FROM DBA_OBJECTS
WHERE OBJECT_NAME = 'PK_T_XXXXXXXXODUCT';

OWNER OBJECT_NAME SUBOBJECT_NAME OBJECT_TYPE
------------------------------ --------------------------------------------------------------------------------------------------------------------------------
xxxxx PK_T_XXXXXXXXODUCT INDEX

操作：
SQL> RECOVER DATABASE ALLOW 1 CORRUPTION;
ORA-00283: recovery session canceled due to errors
ORA-10459: cannot start media recovery on standby database; conflicting state detected

alter database recover automatic standby database allow 1 corruption;
alter database open READ ONLY;
alter database recover managed standby database using current logfile disconnect from session;

--alert
Managed Standby Recovery not using Real Time Apply
ALLOW CORRUPTION option must use serial recovery
Sun Sep 26 11:02:51 2021
Media Recovery Log /data/arch/thread_2_seq_1327403.37691.1084234145
Media Recovery Log /data/arch/thread_1_seq_1413157.55611.1084234147
CORRUPTING BLOCK 206563 OF FILE 2393 AND CONTINUING RECOVERY
Errors in file /data/app/oracle/diag/rdbms/XXX/XXX1/trace/XXX1_ora_5768708.trc:
ORA-10567: Redo is inconsistent with data block (file# 2393, block# 206563, file offset is unknown bytes)
ORA-10564: tablespace LIFEINDEX_T_L
ORA-01110: data file 2393: '+DATA/XXX/datafile/lifeindex_t_l.15681.1083761543'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 249021

进一步分析
---select tracefile from v$process where addr in (select paddr from v$session where sid in (select sid from v$mystat));
Dump of the controlfiles
SQL> alter session set events 'immediate trace name controlf level 8';
Dump of the datafile headers:
SQL> alter session set events 'immediate trace name file_hdrs level 10';
Dump of the redo log headers:
SQL> alter session set events 'immediate trace name redohdr level 10';