oraagent.bin High Memory Usage as Dependent Listener was Removed/Renamed

Grid Infrastructure oraagent.bin process using huge amount of memory and forking huge number of threads.

<GI_HOME>/log/<node>/agent/crsd/oraagent_grid/oraagent_grid.log

2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 CLSCRS_STAT ret: 200
2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 clscrs_res_get_op_status CLSCRS_STAT status 210 err_msg CRS-0210: Could not find resource 'ora.LISTENER.lsnr'. ====>> resource missing
2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 CLSCRS_STAT ret: 200
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 clscrs_res_get_op_status CLSCRS_STAT status 210 err_msg CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} AsmCommonAgent::setLocalListener cls::Exception CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} ASM Dedicated Thread }

"<GI_HOME>/bin/crsctl stat res -t" shows no 'ora.LISTENER.lsnr' resource

"<GI_HOME>/bin/crsctl stat res ora.asm -p" shows ora.asm depends on 'ora.LISTENER.lsnr' resource

NAME=ora.asm
....
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)

CAUSE

Dependent resource ' ora.LISTENER.lsnr' was removed.

The issue is being investigated in bug 18261183 and bug 18427298

SOLUTION

The workaround is to add the missing "ora.LISTENER.lsnr" listener or set correct "listener" resource for ora.asm as grid user:

In the case of "ora.LISTENER.lsnr" is gone, add the listener back:

$ srvctl add listener -l LISTENER -p <port>

In the case of "ora.LISTENER.lsnr" was renamed to something else, i.e. "ora.ASM_LISTENER.lsnr"

$ srvctl modify asm -l ASM_LISTENER

The issue is fixed in 11.2.0.4 GI PSU5

####isssue

cradmin@sdbdb01:/home/cradmin]# ps -ef|grep 12320912
grid 4259970 12320912 120 0:00 <defunct>
grid 4784202 12320912 28 0:00 <defunct>
grid 4915348 12320912 120 0:00 <defunct>
grid 6947042 12320912 85 0:00 <defunct>
grid 7012576 12320912 29 0:00 <defunct>
grid 7143450 12320912 86 0:00 <defunct>
grid 7209134 12320912 120 0:00 <defunct>
grid 7274710 12320912 31 0:00 <defunct>
grid 7405822 12320912 29 0:00 <defunct>
grid 7536764 12320912 97 0:00 <defunct>
grid 7864386 12320912 64 0:00 <defunct>
grid 8126494 12320912 16 0:00 <defunct>

grid 12320912 1 0 Aug 21 - 98:55 /db/db/oracleapp/11.2.0/grid/bin/oraagent.bin

[cradmin@sdbdb01:/home/cradmin]# svmon -P 12320912

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
12320912 oraagent.bin 2352886 28396 105 2275346 Y Y N

grid@sdbdb01:/db/db/oracleapp/11.2.0/grid/OPatch]$ svmon -P 3277270

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
3277270 *** process does not exist ***
[grid@sdbdb01:/db/db/oracleapp/11.2.0/grid/OPatch]$ svmon -P 5243274

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
5243274 ocssd.bin 71624 57263 312 68341 Y Y N

->08/27 58.2.105.31
1.ps -ef|grep oraagent.bin
svmon -P 3408330 |more

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
3408330 oraagent.bin 66563 28398 0 55579 Y Y N

2.svmon -P 6357426 |more
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
6357426 oraagent.bin 106979 28391 0 97487 Y Y N

->after one minutes
6357426 oraagent.bin 107741 28391 0 98247 Y Y N

->after one minutes
6357426 oraagent.bin 113797 28391 0 104286 Y Y N

->after one minutes
6357426 oraagent.bin 120610 28391 0 111086 Y Y N

3.svmon -P 7274974 |more
[grid@sdbdb01:/home/grid]$ svmon -P 7274974 |more

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
7274974 *** process does not exist ***

->/db/db/oracleapp/11.2.0/grid/log/sdbdb01/agent/crsd/oraagent_grid/oraagent_grid.log

tail -100 /db/db/oracleapp/11.2.0/grid/log/sdbdb01/agent/crsd/oraagent_grid/oraagent_grid.log

2019-08-27 15:17:57.949: [ USRTHRD][62143]{1:10808:2} AsmCommonAgent::setLocalListener cls::Exception CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.

-> crs_stat

crs_stat |grep LISTENER
NAME=ora.LISTENER_SCAN1.lsnr

crsctl stat res ora.asm -p

NAME=ora.asm
TYPE=ora.asm.type
ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=ora.%CRS_CSS_NODENAME%.ASM%CRS_CSS_NODENUMBER%.asm
AUTO_START=never
CHECK_INTERVAL=60
CHECK_TIMEOUT=30
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)
DEGREE=1
DESCRIPTION=Oracle ASM resource
ENABLED=1
GEN_USR_ORA_INST_NAME=
GEN_USR_ORA_INST_NAME@SERVERNAME(sdbdb01)=+ASM1
GEN_USR_ORA_INST_NAME@SERVERNAME(sdbdb02)=+ASM2
LOAD=1
LOGGING_LEVEL=1
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
START_DEPENDENCIES=weak(ora.LISTENER.lsnr)
START_TIMEOUT=900
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=
STOP_TIMEOUT=600
TYPE_VERSION=1.2
UPTIME_THRESHOLD=1d
USR_ORA_ENV=
USR_ORA_INST_NAME=+ASM%CRS_CSS_NODENUMBER%
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.4.0

cause:
Dependent resource ' ora.LISTENER.lsnr' was removed.

The issue is being investigated in bug 18261183 and bug 18427298

fix:

以下操作大部分操作都是grid用户，除了重启CRS

1.确保1527 端口没被占用

[grid@sdbdb01:/home/grid]$ netstat -an|grep 1527
[grid@sdbdb01:/home/grid]$

In the case of "ora.LISTENER.lsnr" is gone, add the listener back:

$ srvctl add listener -l LISTENER -p 1527

In the case of "ora.LISTENER.lsnr" was renamed to something else, i.e. "ora.db.lsnr"
$ lsnrctl status db

$ srvctl modify asm -l db

The issue is fixed in 11.2.0.4 GI PSU5

2.停下MRP ，
alter database recover managed standby database cancel;

3.重启CRS
/db/db/oracleapp/11.2.0/grid/bin/crsctl stop crs
/db/db/oracleapp/11.2.0/grid/bin/crsctl start crs

4.启动MRP
alter database recover managed standby database using current logfile disconnect from session;

5.观察日志oraagent_grid.log是否还有 CRS-0210 错误，理论上应该没有
tail -100 /db/db/oracleapp/11.2.0/grid/log/sdbdb01/agent/crsd/oraagent_grid/oraagent_grid.log

是否还有 CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.

6.删除默认监听
lsnrctl stop
srvctl remove listener -l LISTENER

ps.
7.
->
[opdb@sdbdb01:/home/opdb]$ svmon -P 7078058 |more

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
7078058 *** process does not exist ***

-〉
[opdb@sdbdb01:/home/opdb]$ svmon -P 4653478 |more

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
4653478 oraagent.bin 56155 28385 0 49585 Y Y N

->after one minutes;
svmon -P 4653478 |more
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
4653478 oraagent.bin 56157 28385 0 49587 Y Y N

->
svmon -P 6160876 |more