RAC with asm on AIX, ORA-01114 error，with "gipcretAuthFail (22) " in ocssd.log

I/O Errors in Alert log with ORA-29701, with "gipcWait failed with 16" in trace (文档 ID 1496329.1)

1. Database alert log

Fri May  ::
Errors in file /oracle/app/oracle/diag/rdbms/orcl/rocl1/trace/rocl1_ora_65536796.trc:
ORA-: 将块写入文件  时出现 IO 错误 (块 # )
Fri May  :: 

2. trace file

Oracle Database 11g Enterprise Edition Release  - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/app/oracle/product//db_1
System name:    AIX
Node name:      rac1
Release:
Version:
Machine:        00F6E7C84C00
Instance name: rocl1
Redo thread mounted by this instance:
Oracle process number:
Unix process pid: , image: oracle@rac1

*** -- ::58.840
*** SESSION ID:(-- ::58.840
*** CLIENT ID:() -- ::58.840
*** SERVICE NAME:(orcl) -- ::58.840
*** MODULE NAME:(JDBC Thin Client) -- ::58.840
*** ACTION NAME:() -- ::58.840

-- :: ()
-- ::
kgxgncin: CLSS init failed with status
kgxgncin: return status  ( SKGXN not av) from CLSS
kjfmsgr: unable to connect to NM for reg in shared group
ORA-: 将块写入文件  时出现 IO 错误 (块 # )
Dump of memory from 0x070001209CBA0328 to 0x070001209CBA0D3B
70001209CBA0320                    20544F44          [WITH TOD]

3. ocssd.log

-- 检查/oracle/app//grid/log/rac1/cssd/ocssd.log 文件
-- ::]clssgmQueueShare: (11ba99f10) target global grock DBORCL member  type  queued from client (1176496b0), global grock DBORCL, refcount
-- ::]clssgmRegisterShared: global grock DBORCL member  share type , refcount
-- ::] gipcmodMuxTransferAccept: internal accept request failed endp 1112a2970, child 11ba653d0, ret gipcretAuthFail (22) 
-- ::] gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretAuthFail () ]  error during accept on endp 1112a2970
-- ::] gipcmodClscCallback: async request failed req 1172b0bf0 [00000000e3b63bc0] { gipcSendRequest : addr , olen , parentEndp 11abbcef
, ret gipcretConnectionLost (), objFlags )
-- ::] gipcmodMuxTransferAccept: internal accept request failed endp 1112a2970, child 11abbcef0, ret gipcretConnectionInvalid ()
-- ::] gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretConnectionInvalid () ]  error during accept on endp 1112a2970
-- ::]clssscSelect: cookie accept request 11ad57f10
-- ::]clssscevtypSHRCON: getting client with cmproc 11ad57f10
-- ::]clssgmRegisterClient: proc(/11ad57f10), client(/1174aaa90)
-- ::]clssscSelect: cookie accept request 11ba74630
-- ::]clssscevtypSHRCON: getting client with cmproc 11ba74630
-- ::]clssgmRegisterClient: proc(/11ba74630), client(/)
-- ::]clssgmRegisterShared: grp DG_LOCAL_DATA, mbr , type
-- ::]clssgmQueueShare: (11a93a690) target local grock DG_LOCAL_DATA member  type  queued from client (1174aaa90), local grock DG_LOCAL_DATA, refcount
-- ::]clssgmRegisterShared: local grock DG_LOCAL_DATA member  share type , refcount
-- ::]clssgmRegisterShared: grp DBORCL, mbr , type
-- ::]clssgmQueueShare: (11a93ab70) target global grock DBORCL member  type  queued from client (), global grock DBORCL, refcount
-- ::]clssgmRegisterShared: global grock DBORCL member  share type , refcount
-- ::] gipcmodClscCallback: async request failed req 11730eff0 [00000000e3b63c64] { gipcSendRequest : addr , olen , parentEndp 11abbcef
, ret gipcretConnectionLost (), objFlags )
-- ::] gipcmodMuxTransferAccept: internal accept request failed endp 1112a2970, child 11abbcef0, ret gipcretConnectionInvalid ()
-- ::] gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretConnectionInvalid () ]  error during accept on endp 1112a2970
-- ::]clssscSelect: cookie accept request 11ba4a590
-- ::]clssscevtypSHRCON: getting client with cmproc 11ba4a590
-- ::]clssgmRegisterClient: proc(/11ba4a590), client(/11764d8f0)
-- ::]clssscSelect: cookie accept request 1109c2e00
-- ::]clssgmAllocProc: (11bac8dd0) allocated

4. 检查CRS_home空间及文件

目录空间足够。
ls -ld /var/tmp/.oracle
drwxrwxrwt     root     oinstall         Nov    /var/tmp/.oracle
ls -ld /tmp/.oracle
drwxrwxrwt     root     oinstall        Jan  : /tmp/.oracle

5. 数据库此刻出现活动回话剧增，459f3z9u4fb3u语句查询字典视图出现（cursor: pin S wait on X）等待事件，且sga频繁收缩和扩展

SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
GROW        |IMMEDIATE   |shared_pool_size     |       |      |     |COMPLETE |/ :          |
GROW        |IMMEDIATE   |shared_pool_size     |       |      |     |COMPLETE |/ :          |
GROW        |IMMEDIATE   |shared_pool_size     |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
GROW        |IMMEDIATE   |shared_pool_size     |       |      |     |COMPLETE |/ :          |
GROW        |DEFERRED    |db_cache_size        |       |      |     |COMPLETE |/ :          |
SHRINK      |DEFERRED    |shared_pool_size     |       |      |     |COMPLETE |/ :          |
SHRINK      |DEFERRED    |shared_pool_size     |       |      |     |COMPLETE |/ :          |
GROW        |DEFERRED    |db_cache_size        |       |      |     |COMPLETE |/ :          |
GROW        |DEFERRED    |db_cache_size        |       |      |     |COMPLETE |/ :          |
SHRINK      |DEFERRED    |shared_pool_size     |       |      |     |COMPLETE |/ :          |
GROW        |DEFERRED    |db_cache_size        |       |      |     |COMPLETE |/ :          |
SHRINK      |DEFERRED    |shared_pool_size     |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
GROW        |IMMEDIATE   |shared_pool_size     |       |      |     |COMPLETE |/ :          |
GROW        |IMMEDIATE   |shared_pool_size     |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
GROW        |IMMEDIATE   |shared_pool_size     |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
GROW        |IMMEDIATE   |shared_pool_size     |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
SHRINK      |IMMEDIATE   |db_cache_size        |       |      |     |COMPLETE |/ :          |
GROW        |IMMEDIATE   |shared_pool_size     |       |      |     |COMPLETE |/ :          |
GROW        |IMMEDIATE   |shared_pool_size     |       |      |     |COMPLETE |/ :          |
SHRINK      |DEFERRED    |shared_pool_size     |       |      |     |COMPLETE |/ :          |
GROW        |DEFERRED    |db_cache_size        |       |      |     |COMPLETE |/ :          |        

Cause 3. ocssd log has "gipcretAuthFail (22)" (文档 ID 1496329.1)

Example:

-- ::] gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretAuthFail () ]  error during accept on endp 111249b70
gipcretAuthFail () indicates "general security authorization failure".

This could occur for multiple reasons:
* if filesystem is full and there is no space to create file under auth directory. Please check if there is sufficient space in CRS_HOME.
* Also this issue could occur if /var/tmp/.oracle socket is deleted (/tmp/.oracle on some platforms) . Please check on this too.

核查结果与【Cause 3. ocssd log has "gipcretAuthFail (22)" (文档 ID 1496329.1)】情况一致，但我们数据库软件目录空间足够且.oracle文件存在。

问题分析总结：ORA-01114告警是由于数据库SGA出现抖动引起数据库出现性能问题导致。

处理建议：增加SGA大小132G扩大到180G（v$sga_target_advice建议值)