又遇BUG-ORA-01148：数据文件忽然变为recover状态

现象：
RAC环境，数据文件状态变为recover，查看alert日志有如下报错：

Wed Jun 26 02:31:03 2013

Thread 1 advanced to log sequence 33187

 Current log# 1 seq# 33187 mem# 0: +TJDISK/tj/onlinelog/group_1.257.757797483

Wed Jun 26 10:10:03 2013

Errors in file /opt/app/diag/rdbms/tj/tj1/trace/tj1_dbw0_6145.trc:

ORA-01148: cannot refresh file size for datafile 17

ORA-01110: data file 17: '+TJDISK/tj/datafile/ntj_index03.301.757894747'

ORA-01031: insufficient privileges

Automatic datafile offline due to media error on

file 17: +TJDISK/tj/datafile/ntj_index03.301.757894747

Unexpected communication failure with ASM instance:

 error 1031 (ORA-01031: insufficient privileges

)

Wed

分析：

1.查看所有节点的messages系统日志、asm日志均没有出现错误信息。

2.查看DG的raw权限，也没有异常。

/dev/raw/raw6

/dev/raw/raw7

3.该Datafile为autoextend模式。

SQL> select file_name,autoextensible from dba_data_files where file_name like '+TJDISK/tj/datafile/ntj_index03.301.757894747';

FILE_NAME                                                        

--------------------------------------------------------------------------------

AUT

---

+TJDISK/tj/datafile/ntj_index03.301.757894747

YES

最后查了下Metalink，怀疑是命中了Oracle的一个BUG：Bug 16734525或Bug 9357097（Bug 16734525 is the duplicate of Bug 9357097）。

Bug 16734525 : ORA-1148: CANNOT REFRESH FILE SIZE FOR DATAFILE

Hdr: 16734525 10.2.0.5 RDBMS 11.1.0.7 ASM PRODID-5 PORTID-23 ORA-1148 9357097

Abstract: ORA-1148: CANNOT REFRESH FILE SIZE FOR DATAFILE

*** 04/27/13 02:21 am ***

PROBLEM:--------

Fri Apr 26 11:31:28 EDT 2013

Redo Shipping Client Connected as PUBLIC--

Connected User is ValidRedo Shipping Client Connected as PUBLIC--

Connected User is Valid

Fri Apr 26 11:44:55 EDT 2013

Errors in file /home/oracle/admin/ctopprul/bdump/ctopprul1_dbw0_20315.trc:

ORA-1148: cannot refresh file size for datafile 340

ORA-1110: data file 340: '+DATA/ctopprul_rdc/datafile/wires_data.1968.789654733'

ORA-1031: insufficient privilegesFri

Apr 26 11:44:55 EDT 2013

Automatic datafile offline due to media error onfile 340: +DATA/ctopprul_rdc/datafile/wires_data.1968.789654733

Fri Apr 26 11:44:59 EDT 2013

Unexpected communication failure with ASM instance: error 1031

ORA-1031: insufficient privileges)

NOTE: ASMB process state dumped to trace file /home/oracle/admin/ctopprul/bdump/ctopprul1_dbw0_20315.trc

NOTE: force a map free for map id 345 DIAGNOSTIC

ANALYSIS:--------------------

1. Matches the bug 9357097: SMALL BEEHIVE: FAILURE TO REFRESH FILE SIZE DUE TO SPACE OFFLINES DATAFILE

Need to confirm from DEV as audit file space issues were not there

2. Not using role separation and oracle executable is with correct permissions

3. CT is not sure if dbv or rman validate was run on the problematic datafiledue to media error

ORA-1148: cannot refresh file size for datafile 340

ORA-1110: data file 340: '+DATA/ctopprul_rdc/datafile/wires_data.1968.789654733'

ORA-1031: insufficient privileges

Fri Apr 26 11:44:55 EDT 2013

Automatic datafile offline due to media error on  >>>>>>>>>>>> Media error

4. Ulimit was showing nofiles of low value         Customer Visible         

[Open Update screen]     

[Double Click on Activity Text to enable Save operation]         

[Audit]Hi team,   Oracle:-----------  -

Checked if there was any space issues on the server and nothing foundas the above bug is hit when audit files are not able to write-OS watcher logs shows normal

WORKAROUND:-----------

RELATED BUGS:-------------

REPRODUCIBILITY:----------------

TEST CASE:----------

STACK TRACE:------------

SUPPORTING INFORMATION:-----------------------

Uploaded all the relevant info to the bug 24 HOUR CONTACT INFORMATION FOR P1

BUGS:----------------------------------------

DIAL-IN INFORMATION:--------------------

IMPACT DATE:------------

Bug 9357097  ORA-1148 Failure to refresh file size offlines datafile producing ORA-372 ORA-376

Symptoms:

Related To:

1 Error May Occur

2 ORA-1148 / ORA-372 / ORA-376

Range of versions believed to be affected   <-- 12.1下的版本都有可能命中

Versions BELOW 12.1    

Versions confirmed as being affected

?11.2.0.1

?11.1.0.7

?10.2.0.5

?10.2.0.4

Platforms affected

 Generic (all / most platforms affected)

Fixed:

This issue is fixed in                  <-- 12.1.0.1 11.2.0.2中已修复

 ?12.1.0.1 (Base Release)

?11.2.0.2 (Server Patch Set)

DBWR can offline the datafile with message "Automatic datafile offline due to media error"

if file size refresh fails with error ORA-1148.

As the file is offline, subsequent attempts to read the affected file produce

error ORA-372 or ORA-376 requiring media recovery.

解决方法：

临时解决方法是将手动将文件online，

Oracle并没有提供专门的补丁，需要升级到对应版本才能彻底解决（11.2.0.2）。

诊断时在ASM实例中执行了以下脚本。

SPOOL ASM_FIRST<instance#>.HTML

SET MARKUP HTML ON

set echo on

set pagesize 200

alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS';

select 'THIS ASM REPORT WAS GENERATED AT: ==)> ' , sysdate " " from dual;

select 'HOSTNAME ASSOCIATED WITH THIS ASM INSTANCE: ==)> ' , MACHINE " " from v$session where program like '%SMON%';

select * from v$asm_diskgroup;

SELECT * FROM V$ASM_DISK ORDER BY GROUP_NUMBER,DISK_NUMBER;

SELECT * FROM V$ASM_CLIENT;

select * from V$ASM_ATTRIBUTE;

select * from gv$asm_operation;

select * from v$version;

show parameter asm

show parameter cluster

show parameter instance_type

show parameter instance_name

show parameter spfile

show sga

spool off

exit