oraagent.bin High Memory Usage as Dependent Listener was Removed/Renamed
Grid Infrastructure oraagent.bin process using huge amount of memory and forking huge number of threads.
- <GI_HOME>/log/<node>/agent/crsd/oraagent_grid/oraagent_grid.log
2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 clscrs_res_get_op_status CLSCRS_STAT status 210 err_msg CRS-0210: Could not find resource 'ora.LISTENER.lsnr'. ====>> resource missing
2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 CLSCRS_STAT ret: 200
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 clscrs_res_get_op_status CLSCRS_STAT status 210 err_msg CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} AsmCommonAgent::setLocalListener cls::Exception CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} ASM Dedicated Thread }
- "<GI_HOME>/bin/crsctl stat res -t" shows no 'ora.LISTENER.lsnr' resource
- "<GI_HOME>/bin/crsctl stat res ora.asm -p" shows ora.asm depends on 'ora.LISTENER.lsnr' resource
....
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)
CAUSE
Dependent resource ' ora.LISTENER.lsnr' was removed.
The issue is being investigated in bug 18261183 and bug 18427298
SOLUTION
The workaround is to add the missing "ora.LISTENER.lsnr" listener or set correct "listener" resource for ora.asm as grid user:
In the case of "ora.LISTENER.lsnr" is gone, add the listener back:
In the case of "ora.LISTENER.lsnr" was renamed to something else, i.e. "ora.ASM_LISTENER.lsnr"
The issue is fixed in 11.2.0.4 GI PSU5
####isssue
cradmin@sdbdb01:/home/cradmin]# ps -ef|grep 12320912
grid 4259970 12320912 120 0:00 <defunct>
grid 4784202 12320912 28 0:00 <defunct>
grid 4915348 12320912 120 0:00 <defunct>
grid 6947042 12320912 85 0:00 <defunct>
grid 7012576 12320912 29 0:00 <defunct>
grid 7143450 12320912 86 0:00 <defunct>
grid 7209134 12320912 120 0:00 <defunct>
grid 7274710 12320912 31 0:00 <defunct>
grid 7405822 12320912 29 0:00 <defunct>
grid 7536764 12320912 97 0:00 <defunct>
grid 7864386 12320912 64 0:00 <defunct>
grid 8126494 12320912 16 0:00 <defunct>
grid 12320912 1 0 Aug 21 - 98:55 /db/db/oracleapp/11.2.0/grid/bin/oraagent.bin
[cradmin@sdbdb01:/home/cradmin]# svmon -P 12320912
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
12320912 oraagent.bin 2352886 28396 105 2275346 Y Y N
grid@sdbdb01:/db/db/oracleapp/11.2.0/grid/OPatch]$ svmon -P 3277270
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
3277270 *** process does not exist ***
[grid@sdbdb01:/db/db/oracleapp/11.2.0/grid/OPatch]$ svmon -P 5243274
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
5243274 ocssd.bin 71624 57263 312 68341 Y Y N
->08/27 58.2.105.31
1.ps -ef|grep oraagent.bin
svmon -P 3408330 |more
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
3408330 oraagent.bin 66563 28398 0 55579 Y Y N
2.svmon -P 6357426 |more
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
6357426 oraagent.bin 106979 28391 0 97487 Y Y N
->after one minutes
6357426 oraagent.bin 107741 28391 0 98247 Y Y N
->after one minutes
6357426 oraagent.bin 113797 28391 0 104286 Y Y N
->after one minutes
6357426 oraagent.bin 120610 28391 0 111086 Y Y N
3.svmon -P 7274974 |more
[grid@sdbdb01:/home/grid]$ svmon -P 7274974 |more
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
7274974 *** process does not exist ***
->/db/db/oracleapp/11.2.0/grid/log/sdbdb01/agent/crsd/oraagent_grid/oraagent_grid.log
tail -100 /db/db/oracleapp/11.2.0/grid/log/sdbdb01/agent/crsd/oraagent_grid/oraagent_grid.log
2019-08-27 15:17:57.949: [ USRTHRD][62143]{1:10808:2} AsmCommonAgent::setLocalListener cls::Exception CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
-> crs_stat
crs_stat |grep LISTENER
NAME=ora.LISTENER_SCAN1.lsnr
->
crsctl stat res ora.asm -p
NAME=ora.asm
TYPE=ora.asm.type
ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=ora.%CRS_CSS_NODENAME%.ASM%CRS_CSS_NODENUMBER%.asm
AUTO_START=never
CHECK_INTERVAL=60
CHECK_TIMEOUT=30
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)
DEGREE=1
DESCRIPTION=Oracle ASM resource
ENABLED=1
GEN_USR_ORA_INST_NAME=
GEN_USR_ORA_INST_NAME@SERVERNAME(sdbdb01)=+ASM1
GEN_USR_ORA_INST_NAME@SERVERNAME(sdbdb02)=+ASM2
LOAD=1
LOGGING_LEVEL=1
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
START_DEPENDENCIES=weak(ora.LISTENER.lsnr)
START_TIMEOUT=900
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=
STOP_TIMEOUT=600
TYPE_VERSION=1.2
UPTIME_THRESHOLD=1d
USR_ORA_ENV=
USR_ORA_INST_NAME=+ASM%CRS_CSS_NODENUMBER%
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.4.0
cause:
Dependent resource ' ora.LISTENER.lsnr' was removed.
The issue is being investigated in bug 18261183 and bug 18427298
fix:
以下操作大部分操作都是grid用户,除了重启CRS
1.确保1527 端口没被占用
[grid@sdbdb01:/home/grid]$ netstat -an|grep 1527
[grid@sdbdb01:/home/grid]$
In the case of "ora.LISTENER.lsnr" is gone, add the listener back:
$ srvctl add listener -l LISTENER -p 1527
In the case of "ora.LISTENER.lsnr" was renamed to something else, i.e. "ora.db.lsnr"
$ lsnrctl status db
$ srvctl modify asm -l db
The issue is fixed in 11.2.0.4 GI PSU5
2.停下MRP ,
alter database recover managed standby database cancel;
3.重启CRS
/db/db/oracleapp/11.2.0/grid/bin/crsctl stop crs
/db/db/oracleapp/11.2.0/grid/bin/crsctl start crs
4.启动MRP
alter database recover managed standby database using current logfile disconnect from session;
5.观察日志oraagent_grid.log是否还有 CRS-0210 错误,理论上应该没有
tail -100 /db/db/oracleapp/11.2.0/grid/log/sdbdb01/agent/crsd/oraagent_grid/oraagent_grid.log
是否还有 CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
6.删除默认监听
lsnrctl stop
srvctl remove listener -l LISTENER
ps.
7.
->
[opdb@sdbdb01:/home/opdb]$ svmon -P 7078058 |more
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
7078058 *** process does not exist ***
-〉
[opdb@sdbdb01:/home/opdb]$ svmon -P 4653478 |more
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
4653478 oraagent.bin 56155 28385 0 49585 Y Y N
->after one minutes;
svmon -P 4653478 |more
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
4653478 oraagent.bin 56157 28385 0 49587 Y Y N
->
svmon -P 6160876 |more
oraagent.bin High Memory Usage as Dependent Listener was Removed/Renamed的更多相关文章
- SHELL:Find Memory Usage In Linux (统计每个程序内存使用情况)
转载一个shell统计linux系统中每个程序的内存使用情况,因为内存结构非常复杂,不一定100%精确,此shell可以在Ghub上下载. [root@db231 ~]# ./memstat.sh P ...
- Shell script for logging cpu and memory usage of a Linux process
Shell script for logging cpu and memory usage of a Linux process http://www.unix.com/shell-programmi ...
- 5 commands to check memory usage on Linux
Memory Usage On linux, there are commands for almost everything, because the gui might not be always ...
- Why does the memory usage increase when I redeploy a web application?
That is because your web application has a memory leak. A common issue are "PermGen" memor ...
- Reducing and Profiling GPU Memory Usage in Keras with TensorFlow Backend
keras 自适应分配显存 & 清理不用的变量释放 GPU 显存 Intro Are you running out of GPU memory when using keras or ten ...
- GPU Memory Usage占满而GPU-Util却为0的调试
最近使用github上的一个开源项目训练基于CNN的翻译模型,使用THEANO_FLAGS='floatX=float32,device=gpu2,lib.cnmem=1' python run_nn ...
- 【转】C++ Incorrect Memory Usage and Corrupted Memory(模拟C++程序内存使用崩溃问题)
http://www.bogotobogo.com/cplusplus/CppCrashDebuggingMemoryLeak.php Incorrect Memory Usage and Corru ...
- Memory usage of a Java process java Xms Xmx Xmn
http://www.oracle.com/technetwork/java/javase/memleaks-137499.html 3.1 Meaning of OutOfMemoryError O ...
- Redis: Reducing Memory Usage
High Level Tips for Redis Most of Stream-Framework's users start out with Redis and eventually move ...
随机推荐
- workerman——消息推送(web-msg-send)
前言 说下场景,当后台将号码池的号码分配给指定客服的时候,需要给指定的客户推送一条消息告诉该客户,通讯录有新增数据. 步骤 下载 https://www.workerman.net/web-sende ...
- dosbox+masm5.0编译汇编文件
在去年写过如何bc3.1编译ucos,不过现在很少去用到,但是那是用dosbox也是懵懵懂懂的,参见https://blog.csdn.net/liming0931/article/details/8 ...
- car配置篇
这没有什么好说的 1.5t的涡轮增压 CVT(Continuously Variable Transmission)技术即无级变速技术,它采用传动带和工作直径可变的主.从动轮相配合来传递动力,可以实现 ...
- Codeforces Global Round 6[A,B,C]
题意:给一个字符串,对它重新排列使得它是60的倍数. 模拟一下,需要能整除60 字符串中需要 能整除2 3 10,所以需要字符串各位数之和能整除3 并且有 一个偶数和一个0 或者两个0也行[没考虑 ...
- man、whatis、apropos命令
man命令类似于Linux的帮助文档. 1.man1提供给普通用户的可执行命令说明: 输入man 1 ls 后,结果如下: 结果分析: (1)NAME:命令的名称: (2)SYNOPSIS:参数的使用 ...
- Luogu P3810 【模板】三维偏序(陌上花开) CDQ分治 树状数组
https://www.luogu.org/problemnew/show/P3810 复习板子,重要的题就真的要写三遍???之前写过一篇博客了,不过之前写的那个板子的sort用的规则真是沙雕 #in ...
- P1453 城市环路
题目背景 一座城市,往往会被人们划分为几个区域,例如住宅区.商业区.工业区等等.B市就被分为了以下的两个区域——城市中心和城市郊区.在着这两个区域的中间是一条围绕B市的环路,环路之内便是B市中心. 题 ...
- Vijos 1057 盖房子
二次联通门 : Vijos 1057 盖房子 /* Vijos 1057 盖房子 简单的dp 当前点(i, j)所能构成的最大的正方形的边长 为点(i - 1, j - 1)与(i, j - 1), ...
- Loadrunner11录制过程中报错:由于另一个程序正在运行中,此操作无法完成
第一次安装LR11录制没有问题,第二次录制也没有问题,后面第二天录制就出现了如下截图的现象 报错内容:由于另一个程序正在运行中,此操作无法完成.请选择“切换到”来激活正在运行中的的程序,并更正问题 ...
- Android中活动的最佳实践(如何很快的看懂别人的代码activity)
这种方法主要在你拿到别人的代码时候很多activity一时半会儿看不懂,用了这个方法以后就可以边实践操作就能够知道具体哪个activity是干什么用的 1.新建一个BaseActivity的类,让他继 ...