ORA-27090 故障一例

近期的alert日志中碰到了ORA-27090的错误信息。其错误提示为Unable to reserve kernel resources for asynchronous disk I/O。依据这个提示来看是跟异步I/O相关的内核參数问题。

以下是这个问题的描写叙述与解决。

1、故障错误信息
Wed Sep 10 09:58:17 CST 2014
Errors in file /u01/app/oracle/admin/orclls/bdump/orclls_p000_47020.trc:
ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Additional information: 128
Additional information: 65536
Wed Sep 10 09:58:17 CST 2014
Errors in file /u01/app/oracle/admin/orclls/bdump/orclls_p002_47024.trc:
ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Additional information: 128
Additional information: 65536

2、故障分析
#调用oerr，没有获得比較有价值的信息，例如以下：
suse11b:oracle:orcl102 > oerr ora 27090
27090, 00000, "Unable to reserve kernel resources for asynchronous disk I/O"
// *Cause:  The system call to reserve kernel resources for asynchronous I/O
//          has failed.
// *Action: Check errno

#进一步查看trace文件。注，当前数据库版本号为10.2.0.5。OS为suse 11 sp3
suse11a:oracle:orcl101 > more /u01/app/oracle/admin/orclls/bdump/orclls_p002_47024.trc
/u01/app/oracle/admin/orclls/bdump/orclls_p002_47024.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/10.2/db_1
System name:    Linux
Node name:      suse11a
Release:        3.0.76-0.11-default
Version:        #1 SMP Fri Jun 14 08:21:43 UTC 2013 (ccab990)
Machine:        x86_64
Instance name: orclls
Redo thread mounted by this instance: 1
Oracle process number: 35
Unix process pid: 47024, image: oracle@suse11a (P002)

*** SERVICE NAME:() 2014-09-10 09:58:17.159
*** SESSION ID:(1072.1) 2014-09-10 09:58:17.159
*** 2014-09-10 09:58:17.159
KCRP: blocks claimed = 0, eliminated = 0
----- Recovery Hash Table Statistics ---------
Hash table buckets = 32768
Longest hash chain = 0
Average hash chain = 0/0 = 0.0
Max compares per lookup = 0
Avg compares per lookup = 0/0 = 0.0
----------------------------------------------
ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Additional information: 128
Additional information: 65536
----- Recovery Hash Table Statistics ---------
Hash table buckets = 32768
Longest hash chain = 0
Average hash chain = 0/0 = 0.0
Max compares per lookup = 0
Avg compares per lookup = 0/0 = 0.0
----------------------------------------------
#从trace文件里也没有获得比較有价值的信息

3、故障解决
#依据ORA-27090: MESSAGE 27090 NOT FOUND; (文档 ID 579108.1)文档。我们改动内核參数fs.aio-max-nr
#因为此数据库为RAC环境下，因此同一时候调整了2个节点，以下为调整后的情形。
suse11a:~ # grep max-nr /etc/sysctl.conf
#fs.aio-max-nr = 1048576
fs.aio-max-nr = 3145728

suse11b:~ # grep max-nr /etc/sysctl.conf
#aio-max-nr = 1048576
fs.aio-max-nr = 3145728

suse11a:~ # sysctl -p
suse11b:~ # sysctl -p

#配置完成后能够通过cluvy校验
#几日后观察alert日志，未发现该类错误。

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  suse11a           3145728       3145728       1048576       passed         
  suse11b           3145728       3145728       1048576       passed         
Result: Kernel parameter check passed for "aio-max-nr"