前几天,一台Oracle数据库(Oracle Database 10g Release 10.2.0.4.0 - 64bit Production)监控出现"PMON failed to acquire latch, see PMON dump"错误,连接数据库出现短暂异常,告警日志中具体错误如下所示:

Tue Dec 20 09:13:16 2016

PMON failed to acquire latch, see PMON dump

Tue Dec 20 09:14:16 2016

PMON failed to acquire latch, see PMON dump

Tue Dec 20 09:15:55 2016

PMON failed to acquire latch, see PMON dump

Tue Dec 20 09:17:15 2016

PMON failed to acquire latch, see PMON dump

Tue Dec 20 09:17:24 2016

WARNING: inbound connection timed out (ORA-3136)

Tue Dec 20 09:18:23 2016

PMON failed to acquire latch, see PMON dump

Tue Dec 20 09:19:24 2016

PMON failed to acquire latch, see PMON dump

在生成的epps_pmon_4988.trc 跟踪文件里面,发现有些详细信息,你会发现PMON进程不能获取'Child shared pool' latch,它被一个pid = 19 ospid=5022的进程给阻塞了。而ospid这个进程是一个Dispatcher的进程。

*** 2016-12-20 09:14:16.575

PMON unable to acquire latch  600edfa8 Child shared pool level=7 child#=1 

        Location from where latch is held: kghfrunp: alloc: session dur: 

        Context saved from call: 0

        state=busy, wlstate=free

    waiters [orapid (seconds since: put on list, posted, alive check)]:

     33 (3, 1482196555, 3)

     10 (3, 1482196555, 3)

     25 (3, 1482196555, 3)

     13 (3, 1482196555, 3)

     waiter count=4

    gotten 861091119 times wait, failed first 7114074 sleeps 1392223

    gotten 0 times nowait, failed: 0

  possible holder pid = 19 ospid=5022

----------------------------------------

SO: 0x40979aec8, type: 2, owner: (nil), flag: INIT/-/-/0x00

  (process) Oracle pid=19, calls cur/top: (nil)/0x409c92608, flag: (80) DISPATCHER

            int error: 0, call error: 0, sess error: 0, txn error 0

  (post info) last post received: 0 0 236

              last post received-location: kmcpdp

              last process to post me: 4097a64a0 106 64

              last post sent: 0 0 229

              last post sent-location: kmcmbf: not KMCVCFTOS

              last process posted by me: 4097a64a0 106 64

  (latch info) wait_event=0 bits=a0

    holding    (efd=4) 600edfa8 Child shared pool level=7 child#=1 

        Location from where latch is held: kghfrunp: alloc: session dur: 

        Context saved from call: 0

        state=busy, wlstate=free

        waiters [orapid (seconds since: put on list, posted, alive check)]:

         33 (3, 1482196555, 3)

         10 (3, 1482196555, 3)

         25 (3, 1482196555, 3)

         13 (3, 1482196555, 3)

         waiter count=4

    holding    (efd=4) 3fff78210 Child library cache level=5 child#=2 

        Location from where latch is held: kghfrunp: clatch: wait: 

        Context saved from call: 0

        state=busy, wlstate=free

        waiters [orapid (seconds since: put on list, posted, alive check)]:

         15 (3, 1482196555, 3)

         17 (3, 1482196555, 3)

         12 (3, 1482196555, 3)

         waiter count=3

    Process Group: DEFAULT, pseudo proc: 0x4098bc190

    O/S info: user: oracle, term: UNKNOWN, ospid: 5022 

    OSD pid info: Unix process pid: 5022, image: oracle@xx.xxx.xxx.com (D007)

    Short stack dump: 

ksdxfstk()+32<-ksdxcb()+1547<-sspuser()+111<-__restore_rt()+0<-kghfrunp()+1506<-kghfnd()+1389<-kghalo()+587<-kmnsbm()+578<-nsb

al()+428<-nsbalc()+123<-nsdo()+17278<-nsopen()+2315<-nsanswer()+512<-kmnans()+37<-kmdahd()+385<-kmdmai()+5220<-kmmrdp()+564<-o

pirip()+1193<-opidrv()+582<-sou2o()+114<-opimai_real()+317<-main()+116<-__libc_start_main()+244<-_start()+41

Dump of memory from 0x0000000409747C68 to 0x0000000409747E70

409747C60                   00000001 00000000          [........]

409747C70 FE9BEE10 00000003 0000003A 0003129B  [........:.......]

409747C80 FEA7D5D0 00000003 0000003A 0003129B  [........:.......]

409747C90 FE9DAD30 00000003 0000003A 0003129B  [0.......:.......]

        Repeat 2 times

409747CC0 FEAB01F0 00000003 0000003A 0003129B  [........:.......]

409747CD0 FE9DAD30 00000003 0000003A 0003129B  [0.......:.......]

409747CE0 FEA44E70 00000003 0000003A 0003129B  [pN......:.......]

409747CF0 FEAA6FF0 00000003 0000003A 0003129B  [.o......:.......]

409747D00 FEAB8AD0 00000003 0000003A 0003129B  [........:.......]

409747D10 FEA14FF0 00000003 0000003A 0003129B  [.O......:.......]

409747D20 FE9A77F0 00000003 0000003A 0003129B  [.w......:.......]

        Repeat 1 times

409747D40 FEA3CEB0 00000003 0000003A 0003129B  [........:.......]

        Repeat 1 times

409747D60 FE9C64B0 00000003 0000003A 0003129B  [.d......:.......]

        Repeat 1 times

409747D80 FEA062B0 00000003 0000003A 0003129B  [.b......:.......]

        Repeat 3 times

409747DC0 FEAA6FF0 00000003 0000003A 0003129B  [.o......:.......]

409747DD0 FEA8F9D0 00000003 0000003A 0003129B  [........:.......]

409747DE0 FE9F7570 00000003 0000003A 0003129B  [pu......:.......]

409747DF0 FEA91530 00000003 0000003A 0003129B  [0.......:.......]

409747E00 FE9BEE10 00000003 0000003A 0003129B  [........:.......]

409747E10 FE9BB750 00000003 0000003A 0003129B  [P.......:.......]

409747E20 FEA90C10 00000003 0000003A 0003129B  [........:.......]

409747E30 FEA8B9F0 00000003 0000003A 0003129B  [........:.......]

409747E40 FE9C5270 00000003 0000003A 0003129B  [pR......:.......]

409747E50 FEAE12B0 00000003 0000003A 0003129B  [........:.......]

409747E60 FE9C5270 00000003 0000003A 0003129B  [pR......:.......]

由于当时没有出现问题时,并没有及时发现,没有Collect HangAnalyze traces,所以再深入一点的挖掘root case已经很难了。当时手工生成了一个快照(9:26),也就是说9:00 ~ 9:26这段时间生成的快照刚刚覆盖了出现问题的时间段。生成了这个时段的AWR报告,在这个时间段latch:library cache 和latch:shared pool等待事件是主要等待事件。

出现问题的时间段,数据库服务器是比较空闲的。

生成了20-Dec-16 09:11:16到20-Dec-16 09:21:16时段的ASH报告。如下所示,latch:library cache 和latch:shared pool为主要等待事件,但是Avg Active Sessions很小。

所以觉得很有可能是跟Bug有关系,后面在Oracle MetaLink查了一下是否有相关Bug,如下一些相关资料:

Bug 7039896 Spin under kghquiesce_regular_extent holding shared pool latch with AMM

Bug 6488694 - DATABASE HUNG WITH PMON FAILED TO ACQUIRE LATCH MESSAGE

Note 7039896.8 - Bug 7039896 - Spin under kghquiesce_regular_extent holding shared pool latch with AMM

Pmon Failed To Acquire Latch" Messages in Alert Log -Database Hung (文档 ID 468740.1)

 

 

Hang (Involving Shared Resource)

A process may hold a shared resource a lot longer than normally expected leading to many other processes having to wait for that resource. Such a resource could be a lock, a library cache pin, a latch etc.. The overall symptom is that typically numerous processes all appear to be stuck, although some processes may continue unhindered if they do not need the blocked resource.

Hang (Process Spins)

A process enters a tight CPU loop so appears to hang but is actually consuming CPU.

Latch Contention

This issue can result in latch contention within the database.

Waits for "latch: shared pool"

 

我们数据库版本为Oracle Database 10g Release 10.2.0.4.0 - 64bit Production, 所以Bug 7039896是会影响的这个数据库的, 而出现的现象也很符合,但是有一点就是并没有涉及MMAN进程。而且查过V$SGA_RESIZE_OPS,那个时间段并没有相关组件的增长、收缩。另外跟Bug 也非常类似,但是trc文件并没有发现跟MMAN进程有关系。 这个问题还是第一次出现,而且出现过一次后,最近几天都没有出现,所以更加确信是Bug引起的。当然是要找个时间应用Bug 7039896的相关补丁。

另外,在查找这个问题的时候,在官方文档看到一个如何处理、诊断'PMON failed to acquire latch, see PMON dump'的详细文档,本想收录于此,不过还是保持为PDF文件较好,需要可从下面链接下载。

SRDC - How to Collect Standard Information for Issues Where 'PMON failed to acquire latch, see PMON dump' Warnings are Seen in the Alert Log (文档 ID 1951971.1)

PMON failed to acquire latch, see PMON dump的更多相关文章

  1. kafka.common.KafkaException: Failed to acquire lock on file .lock in /tmp/kafka-logs. A Kafka instance in another process or thread is using this directory.

    1.刚才未启动zookeeper集群的时候,直接启动kafka脚本程序,kafka报错了,但是进程号启动起来来,再次启动出现如下所示的问题,这里先将进程号杀死,再启动脚本程序. [hadoop@sla ...

  2. Failed to acquire lock on file .lock in /tmp/kafka-logs. A Kafka instance in another process or thread is using this directory.

    1. 问题现象 启动 kafka 时报错:Failed to acquire lock on file .lock in /tmp/kafka-logs. A Kafka instance in an ...

  3. Failed to acquire connection "SAP_PRD_NEW.SAPSR3". Connection may not be configured correctly or you may not have the right permissions

    SQLSERVER JOB无法执行 错误提示: Message Executed as user: WORKGROUP\NSDZHSCMFP01$. Microsoft (R) SQL Server ...

  4. 关于使用C3P0程序报错Having failed to acquire a resource, com.mchange.v2.resourcepool的问题

    由于是新手的问题,C3P0的使用时严格跟着视频来的,但是问题却来的很突然 在导入了三个包以及创建了路径以后 进行测试 class JdbcutilsTest { @Test void TestGetC ...

  5. kafka启动报错:kafka.common.KafkaException: Failed to acquire lock on file .lock

    kafka 异常退出后重启时遇到的问题 解决: 执行 netstat -lnp|grep 9092 在执行结果中找到进程号执行 kill -9 进程号再尝试启动Kafka  

  6. CASE:DB shutdown/open 过程中发生异常导致JOB不能自动执行

    CASE:DB shutdown/open 过程中发生异常导致JOB不能自动执行 现象: 一个DB中的所有JOB在3月25日之后就不再自动运行,查询DBA_JOBS,发现LAST_DATE定格在3月2 ...

  7. 高并发压力下导致数据库bug

    环境信息:  linux 6.1 + oracle11.2.0.3 RAC     问题现象: 学校晚上6点选课,人数大概有3000,7点时,数据库报错如下(数据库到6点多还是可以连接的),数据库hu ...

  8. loongson官方PMON使用

    目录 [隐藏]  1 PMON使用介绍 1.1 进入PMON控制界面 1.2 Pmon的图形界面 1.3 Pmon的基本命令 1.3.1 Boot and Load 启动与加载内核 1.3.2 MyC ...

  9. PMON

    PMON是一个兼有BIOS和boot loader部分功能的开放源码软件,多用于嵌入式系统. 与BIOS相比功能不足,与常见的bootloader 相比,功能要丰富的多.基于龙芯的系统采用 pmon ...

随机推荐

  1. JavaScript性能优化

    如今主流浏览器都在比拼JavaScript引擎的执行速度,但最终都会达到一个理论极限,即无限接近编译后程序执行速度. 这种情况下决定程序速度的另一个重要因素就是代码本身. 在这里我们会分门别类的介绍J ...

  2. 【SQLServer】记一次数据迁移-标识重复的简单处理

    汇总篇:http://www.cnblogs.com/dunitian/p/4822808.html#tsql 今天在数据迁移的时候因为手贱遇到一个坑爹问题,发来大家乐乐,也传授新手点经验 迁移惯用就 ...

  3. zookeeper源码分析之三客户端发送请求流程

    znode 可以被监控,包括这个目录节点中存储的数据的修改,子节点目录的变化等,一旦变化可以通知设置监控的客户端,这个功能是zookeeper对于应用最重要的特性,通过这个特性可以实现的功能包括配置的 ...

  4. PHP之时间和日期函数

    // 时间日期函数 Time <?php date_default_timezone_set('UTC'); // 获取当前时间的时间戳 $time0 = mktime(); $time1 = ...

  5. 七牛云:ckeditor JS SDK 结合 C#实现多图片上传。

    成功了,搞了2天.分享一下经验. 首先是把官方的那个例子下载下来,然后照如下的方式修改. 其中tempValue是一个全局变量. function savetoqiniu() { var upload ...

  6. Windows下MySQL无法启动

    问题描述: 从网上下了5.7 的MySQL,在bin目录下执行 start mysqld ,弹出个cmd窗口一闪就没了,也看不清是什么报错.mysqld --install安装了服务,也启动不了.   ...

  7. [Hadoop in Action] 第5章 高阶MapReduce

    链接多个MapReduce作业 执行多个数据集的联结 生成Bloom filter   1.链接MapReduce作业   [顺序链接MapReduce作业]   mapreduce-1 | mapr ...

  8. 【转】 XenServer的架构

    一.Dom0或者控制域 XenServer从安装上看起来像是一个Linux,所以有些同学认为XenServer就是Linux或者是Linux修改的.真实的XenServer和Linux的关系是这样的: ...

  9. BZOJ 3083: 遥远的国度 [树链剖分 DFS序 LCA]

    3083: 遥远的国度 Time Limit: 10 Sec  Memory Limit: 1280 MBSubmit: 3127  Solved: 795[Submit][Status][Discu ...

  10. 整体二分QAQ

    POJ 2104 K-th Number 时空隧道 题意: 给出一个序列,每次查询区间第k小 分析: 整体二分入门题? 代码: #include<algorithm> #include&l ...