ORA-15196: invalid ASM block header

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-15335 ORA-15130 ORA-15066 ORA-15196

作者：惜分飞版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户反馈,数据库无法正常启动，通过分析asm的alert日志发现,data磁盘组mount成功之后,没有一会儿自动dismount掉

Mon Sep 26 16:40:14 2022

SQL> /* ASMCMD */ALTER DISKGROUP data MOUNT 

NOTE: cache registered group DATA number=2 incarn=0x9dfa705f

NOTE: cache began mount (first) of group DATA number=2 incarn=0x9dfa705f

NOTE: Assigning number (2,1) to disk (/dev/oracleasm/disks/DATA02)

NOTE: Assigning number (2,0) to disk (/dev/oracleasm/disks/DATA01)

Mon Sep 26 16:40:20 2022

NOTE: GMON heartbeating for grp 2

GMON querying group 2 at 68 for pid 25, osid 14650

NOTE: cache opening disk 0 of grp 2: DATA_0000 path:/dev/oracleasm/disks/DATA01

NOTE: F1X0 found on disk 0 au 2 fcn 0.0

NOTE: cache opening disk 1 of grp 2: DATA_0001 path:/dev/oracleasm/disks/DATA02

NOTE: cache mounting (first) external redundancy group 2/0x9DFA705F (DATA)

Mon Sep 26 16:40:20 2022

* allocate domain 2, invalid = TRUE

kjbdomatt send to inst 2

Mon Sep 26 16:40:20 2022

NOTE: attached to recovery domain 2

NOTE: cache recovered group 2 to fcn 0.321845

NOTE: redo buffer size is 256 blocks (1053184 bytes)

Mon Sep 26 16:40:20 2022

NOTE: LGWR attempting to mount thread 1 for diskgroup 2 (DATA)

NOTE: LGWR found thread 1 closed at ABA 20.3546

NOTE: LGWR mounted thread 1 for diskgroup 2 (DATA)

NOTE: LGWR opening thread 1 at fcn 0.321845 ABA 21.3547

NOTE: cache mounting group 2/0x9DFA705F (DATA) succeeded

NOTE: cache ending mount (success) of group DATA number=2 incarn=0x9dfa705f

Mon Sep 26 16:40:20 2022

NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2

SUCCESS: diskgroup DATA was mounted

SUCCESS: /* ASMCMD */ALTER DISKGROUP data MOUNT

Mon Sep 26 16:40:22 2022

WARNING: failed to online diskgroup resource ora.DATA.dg (unable to communicate with CRSD/OHASD)

Mon Sep 26 16:40:47 2022

NOTE: client xff1:xff registered, osid 14742, mbr 0x0

Mon Sep 26 16:40:57 2022

WARNING: cache read  a corrupt block: group=2(DATA) dsk=1 blk=257 disk=1 (DATA_0001)

incarn=3916071178 au=113792 blk=1 count=1

Errors in file /opt/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14778.trc:

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

NOTE: a corrupted block from group DATA was dumped to /opt/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14778.trc

WARNING: cache read (retry) a corrupt block: group=2(DATA) dsk=1 blk=257

disk=1 (DATA_0001) incarn=3916071178 au=113792 blk=1 count=1

Errors in file /opt/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14778.trc:

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

ERROR: cache failed to read group=2(DATA) dsk=1 blk=257 from disk(s): 1(DATA_0001)

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

NOTE: cache initiating offline of disk 1 group DATA

NOTE: process _user14778_+asm1 (14778) initiating offline of

disk 1.3916071178 (DATA_0001) with mask 0x7e in group 2

NOTE: initiating PST update: grp = 2, dsk = 1/0xe96a810a, mask = 0x6a, op = clear

Mon Sep 26 16:40:58 2022

GMON updating disk modes for group 2 at 70 for pid 28, osid 14778

ERROR: Disk 1 cannot be offlined, since diskgroup has external redundancy.

ERROR: too many offline disks in PST (grp 2)

Mon Sep 26 16:40:58 2022

NOTE: cache dismounting (not clean) group 2/0x9DFA705F (DATA)

WARNING: Offline for disk DATA_0001 in mode 0x7f failed.

NOTE: messaging CKPT to quiesce pins Unix process pid: 14782, image: oracle@oracle11grac1 (B000)

Mon Sep 26 16:40:58 2022

NOTE: halting all I/Os to diskgroup 2 (DATA)

Errors in file /opt/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14778.trc  (incident=144548):

ORA-15335: ASM metadata corruption detected in disk group 'DATA'

ORA-15130: diskgroup "DATA" is being dismounted

ORA-15066: offlining disk "DATA_0001" in group "DATA" may result in a data loss

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

Incident details in: /opt/grid/diag/asm/+asm/+ASM1/incident/incdir_144548/+ASM1_ora_14778_i144548.trc

Mon Sep 26 16:40:58 2022

Sweep [inc][144548]: completed

System State dumped to trace file /opt/grid/diag/asm/+asm/+ASM1/incident/incdir_144548/+ASM1_ora_14778_i144548.trc

Mon Sep 26 16:40:58 2022

NOTE: AMDU dump of disk group DATA created at /opt/grid/diag/asm/+asm/+ASM1/incident/incdir_144548

Mon Sep 26 16:41:00 2022

NOTE: LGWR doing non-clean dismount of group 2 (DATA)

NOTE: LGWR sync ABA=21.3550 last written ABA 21.3550

Mon Sep 26 16:41:00 2022

Sweep [inc2][144548]: completed

Mon Sep 26 16:41:00 2022

ERROR: ORA-15130 in COD recovery for diskgroup 2/0x9dfa705f (DATA)

ERROR: ORA-15130 thrown in RBAL for group number 2

Errors in file /opt/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_5162.trc:

ORA-15130: diskgroup "DATA" is being dismounted

这里看主要是由于asm 磁盘组需要做COD recovery导致无法正常稳定的mount,主要原因是遭遇到asm disk的逻辑坏块(存储物理上看是ok的,但是实际数据在asm中看是异常的)

数据库alert日志报错

Mon Sep 26 16:40:52 2022

Successful mount of redo thread 1, with mount id 1097279951

Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)

Lost write protection disabled

Completed: alter database mount

alter database open

This instance was first to open

Picked broadcast on commit scheme to generate SCNs

LGWR: STARTING ARCH PROCESSES

Mon Sep 26 16:40:56 2022

ARC0 started with pid=40, OS id=14761

ARC0: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

ARC0: STARTING ARCH PROCESSES

Mon Sep 26 16:40:57 2022

ARC1 started with pid=41, OS id=14764

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_lgwr_14479.trc:

ORA-00313: ??????? 1 (???? 1) ???

Mon Sep 26 16:40:57 2022

ARC2 started with pid=42, OS id=14766

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_lgwr_14479.trc:

ORA-00313: ??????? 2 (???? 1) ???

Mon Sep 26 16:40:57 2022

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc:

ORA-00313: open failed for members of log group 1 of thread 1

Mon Sep 26 16:40:57 2022

ARC3 started with pid=44, OS id=14770

ARC1: Archival started

ARC2: Archival started

ARC1: Becoming the 'no FAL' ARCH

ARC1: Becoming the 'no SRL' ARCH

ARC2: Becoming the heartbeat ARCH

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc:

ORA-00313: open failed for members of log group 1 of thread 1

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc2_14766.trc:

ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc1_14764.trc:

ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc  (incident=180281):

ORA-15335: ASM metadata corruption detected in disk group 'DATA'

ORA-15130: diskgroup "DATA" is being dismounted

ORA-15066: offlining disk "DATA_0001" in group "DATA" may result in a data loss

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

ARC3: Archival started

ARC0: STARTING ARCH PROCESSES COMPLETE

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc0_14761.trc:

ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员

ORA-00312: 联机日志 1 线程 1: '+DATA/xff/onlinelog/group_1.271.1025610215'

ORA-17503: ksfdopn: 2 未能打开文件 +DATA/xff/onlinelog/group_1.271.1025610215

ORA-15130: diskgroup "DATA" is being dismounted

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc3_14770.trc:

ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员

ORA-00312: 联机日志 1 线程 1: '+DATA/xff/onlinelog/group_1.271.1025610215'

ORA-17503: ksfdopn: 2 未能打开文件 +DATA/xff/onlinelog/group_1.271.1025610215

ORA-15130: diskgroup "DATA" is being dismounted

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc0_14761.trc:

ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员

ORA-00312: 联机日志 1 线程 1: '+DATA/xff/onlinelog/group_1.271.1025610215'

ORA-17503: ksfdopn: 2 未能打开文件 +DATA/xff/onlinelog/group_1.271.1025610215

ORA-15130: diskgroup "DATA" is being dismounted

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc3_14770.trc:

ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员

ORA-00312: 联机日志 1 线程 1: '+DATA/xff/onlinelog/group_1.271.1025610215'

ORA-17503: ksfdopn: 2 未能打开文件 +DATA/xff/onlinelog/group_1.271.1025610215

ORA-15130: diskgroup "DATA" is being dismounted

Unable to create archive log file '+DATA'

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc:

ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are not known to database.

ORA-17502: ksfdcre:4 Failed to create file +DATA

ORA-15335: ASM metadata corruption detected in disk group 'DATA'

ORA-15130: diskgroup "DATA" is being dismounted

ORA-15066: offlining disk "DATA_0001" in group "DATA" may result in a data loss

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]

*************************************************************

WARNING: A file of type ARCHIVED LOG may exist in

db_recovery_file_dest that is not known to the database.

Use the RMAN command CATALOG RECOVERY AREA to re-catalog

any such files. If files cannot be cataloged, then manually

delete them using OS command. This is most likely the

result of a crash during file creation.

*************************************************************

ARCH: Error 19504 Creating archive log file to '+DATA'

NOTE: Deferred communication with ASM instance

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc:

ORA-15130: diskgroup "DATA" is being dismounted

NOTE: deferred map free for map id 23

Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc:

ORA-16038: log 1 sequence# 14235 cannot be archived

ORA-19504: failed to create file ""

ORA-00312: online log 1 thread 1: '+DATA/xff/onlinelog/group_1.271.1025610215'

ORA-00312: online log 1 thread 1: '+ARCH/xff/onlinelog/group_1.279.1025610217'

Mon Sep 26 16:40:58 2022

Sweep [inc][180281]: completed

Sweep [inc2][180281]: completed

USER (ospid: 14732): terminating the instance due to error 16038

Mon Sep 26 16:40:59 2022

System state dump requested by (instance=1, osid=14732), summary=[abnormal instance termination].

Instance terminated by USER, pid = 14732

对于这类故障处理相对比较容易,通过patch asm，让data磁盘组稳定mount,然后open库,迁移数据,实现数据0丢失,完美恢复

• ORA-15335: ASM metadata corruption detected in disk group ‘DATA’
• WARNING: Read Failed.导致asm磁盘组异常
• ORA-15196: invalid ASM block header [kfc.c:26368]故障恢复
• ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh]故障处理
• ORA-15130: diskgroup “ORADATA” is being dismounted
• asm 加磁盘导致磁盘组损坏恢复
• ORA-600 kfrValAcd30 恢复
• ORA-15096: lost disk write detected
• ASM中磁盘组权限设置
• ocr磁盘组掉盘故障处理
• 再一起asm disk被格式化成ext3文件系统故障恢复
• 存储精简卷导致asm磁盘组异常