每年年底，系统管理员都要组织一次容灾方案的测试、演练。会在一个与生产环境网络隔离的DR环境中，启动各个“生产环境服务器”，然后让各路人员参与其中测试、演练容灾方案是否可靠。这次演练中，一台Oracle数据库服务器启动的时候遇到了问题。如下所示，启动的时候遇到ORA-03113: end-of-file on communication channel错误。

[oracle@mylnx6 ~]$ sqlplus / as sysdba

 

SQL*Plus: Release 10.2.0.5.0 - Production on Fri Dec 21 09:42:11 2018

 

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

 

Connected to an idle instance.

 

SQL> startup

ORA-03113: end-of-file on communication channel

SQL>

检查告警日志，发现数据库在启动的时候，报“ORA-00471: DBWR process terminated with error”错误。如下所示：

PMON started with pid=2, OS id=25005

PSP0 started with pid=3, OS id=25007

MMAN started with pid=4, OS id=25009

DBW0 started with pid=5, OS id=25011

LGWR started with pid=6, OS id=25013

CKPT started with pid=7, OS id=25016

SMON started with pid=8, OS id=25018

RECO started with pid=9, OS id=25020

CJQ0 started with pid=10, OS id=25022

MMON started with pid=11, OS id=25024

Fri Dec 21 09:44:36 CST 2018

starting up 8 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

MMNL started with pid=12, OS id=25026

Fri Dec 21 09:45:12 CST 2018

starting up 24 shared server(s) ...

Fri Dec 21 09:46:43 CST 2018

Errors in file /u01/app/oracle/admin/SCM2/bdump/scm2_pmon_25005.trc:

ORA-00471: DBWR process terminated with error

Fri Dec 21 09:46:43 CST 2018

PMON: terminating instance due to error 471

Instance terminated by PMON, pid = 25005

启动数据库实例的时候，报“ORA-00471: DBWR process terminated with error”这个错误，这个很蹊跷，很有可能是进程被系统给Kill掉了，检查操作系统的错误日志，发现出现了oom_kill_process,也就是说数据库实例启动的时候，由于系统内存资源紧张，DBWR进程被系统选作了牺牲品。具体错误日志如下所示：

Dec 21 09:46:39 mylnx6 kernel: oracle invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0

Dec 21 09:46:39 mylnx6 kernel: oracle cpuset=/ mems_allowed=0

Dec 21 09:46:39 mylnx6 kernel: Pid: 25026, comm: oracle Not tainted 2.6.32-200.13.1.el5uek #1

Dec 21 09:46:39 mylnx6 kernel: Call Trace:

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810a0b66>] ? cpuset_print_task_mems_allowed+0x92/0x9e

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810d9ae6>] oom_kill_process+0x85/0x25b

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810d9fbc>] ? select_bad_process+0xbc/0x102

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810da03f>] __out_of_memory+0x3d/0x86

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810da30f>] out_of_memory+0xfc/0x195

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810dd75e>] __alloc_pages_nodemask+0x487/0x595

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff811075ac>] alloc_page_vma+0xb9/0xc8

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810ff0a7>] read_swap_cache_async+0x52/0xf1

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810ff1a3>] swapin_readahead+0x5d/0x9c

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810d725a>] ? find_get_page+0x22/0x69

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810f1ea3>] handle_mm_fault+0x44b/0x80f

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff8106d7cd>] ? getrusage+0x2b1/0x2ce

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff8101270e>] ? common_interrupt+0xe/0x13

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff81043696>] ? should_resched+0xe/0x2f

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff81456006>] do_page_fault+0x210/0x299

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff81453fd5>] page_fault+0x25/0x30

Dec 21 09:46:39 mylnx6 kernel: Mem-Info:

Dec 21 09:46:39 mylnx6 kernel: Node 0 DMA per-cpu:

Dec 21 09:46:39 mylnx6 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    4: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    5: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    6: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    7: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: Node 0 DMA32 per-cpu:

Dec 21 09:46:39 mylnx6 kernel: CPU    0: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    1: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    2: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    3: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    4: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    5: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    6: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    7: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: Node 0 Normal per-cpu:

Dec 21 09:46:39 mylnx6 kernel: CPU    0: hi:  186, btch:  31 usd:   0

Dec 21 09:46:40 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...

Dec 21 09:46:41 mylnx6 kernel: CPU    1: hi:  186, btch:  31 usd:   0

Dec 21 09:46:40 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...

Dec 21 09:46:41 mylnx6 kernel: CPU    2: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: CPU    3: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: CPU    4: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: CPU    5: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: CPU    6: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: CPU    7: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: active_anon:1764 inactive_anon:209 isolated_anon:64

Dec 21 09:46:41 mylnx6 kernel:  active_file:349 inactive_file:1710 isolated_file:0

Dec 21 09:46:41 mylnx6 kernel:  unevictable:5377 dirty:0 writeback:4 unstable:0

Dec 21 09:46:41 mylnx6 kernel:  free:29838 slab_reclaimable:2400 slab_unreclaimable:119491

Dec 21 09:46:41 mylnx6 kernel:  mapped:2703 shmem:830 pagetables:9849 bounce:0

Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA free:15652kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15172kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 3000 24210 24210

Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA32 free:86296kB min:2464kB low:3080kB high:3696kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 0 21210 21210

Dec 21 09:46:41 mylnx6 kernel: Node 0 Normal free:17404kB min:17440kB low:21800kB high:26160kB active_anon:7056kB inactive_anon:836kB active_file:1396kB inactive_file:6840kB unevictable:21508kB isolated(anon):256kB isolated(file):0kB present:21719040kB mlocked:21504kB dirty:0kB writeback:16kB mapped:10812kB shmem:3320kB slab_reclaimable:9600kB slab_unreclaimable:477964kB kernel_stack:2800kB pagetables:39396kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:544 all_unreclaimable? no

Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 0 0 0

Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA: 1*4kB 2*8kB 1*16kB 0*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15652kB

Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA32: 12*4kB 13*8kB 2*16kB 5*32kB 5*64kB 11*128kB 3*256kB 7*512kB 6*1024kB 4*2048kB 16*4096kB = 86296kB

Dec 21 09:46:41 mylnx6 kernel: Node 0 Normal: 420*4kB 1917*8kB 49*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17800kB

Dec 21 09:46:41 mylnx6 kernel: 4722 total pagecache pages

Dec 21 09:46:41 mylnx6 kernel: 694 pages in swap cache

Dec 21 09:46:41 mylnx6 kernel: Swap cache stats: add 589182, delete 588488, find 343370/443306

Dec 21 09:46:41 mylnx6 kernel: Free swap  = 66723056kB

Dec 21 09:46:41 mylnx6 kernel: Total swap = 67108856kB

Dec 21 09:46:41 mylnx6 kernel: 6291440 pages RAM

Dec 21 09:46:41 mylnx6 kernel: 107316 pages reserved

Dec 21 09:46:41 mylnx6 kernel: 24060 pages shared

Dec 21 09:46:41 mylnx6 kernel: 77648 pages non-shared

Dec 21 09:46:41 mylnx6 kernel: Out of memory: kill process 25011 (oracle) score 8425150 or a child

Dec 21 09:46:41 mylnx6 kernel: Killed process 25011 (oracle)

Dec 21 09:47:20 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...

检查这个系统的内存，发现DR环境下，这个服务器只分配了24G内存，而实际生产环境的内存为64G（设置了Linux标准大页，而且SGA_MAX_SIZE大小为32G），而且这个环境是生产环境的“克隆体”，只是由于资源限制的缘故，系统管理员只分配24G内存。如下所示：

[root@mylnx6 ~]# free -m

             total       used       free     shared    buffers     cached

Mem:         24156      24033        123          0          0          6

-/+ buffers/cache:      24026        130

Swap:        65535         41      65494

[root@mylnx6 ~]# ps -ef | grep ora_

root     11759 11490  0 16:10 pts/1    00:00:00 grep ora_

[root@mylnx6 ~]# ipcs -m

 

------ Shared Memory Segments --------

key        shmid      owner      perms      bytes      nattch     status      

0x00000000 3080192    root      644        80         2                       

0x00000000 3112961    root      644        16384      2                       

0x00000000 3145730    root      644        280        2                       

0x00000000 4096003    gdm       600        393216     0                       

0x2cd12178 3866628    oracle    640        34361835520 0                       

0x00000000 5210117    gdm       600        393216     2          dest  

如上所示，可以看到oracle用户的共享内存段为34361835520字节。所以引起这个错误的原因是因为在系统层面配置了标准大页的缘故（内存资源变化了，但是配置没有随之修改），为了快速解决问题，我们取消标准大页的相关设置。如下所示：

修改limits.conf参数，注释soft memlock和hard memlock参数。

vi /etc/security/limits.conf

然后修改sysctl.conf，将vm.nr_hugepages注释掉。然后重启一下（DR测试环境，可以随时重启）。然后启动Oracle数据库实例，一切正常，当然还需调整相关参数，继续后续测试~。

ORA-00471: DBWR process terminated with error案例的更多相关文章

  1. [SOLVED] “Error 1067: The process terminated unexpectedly” on Windows 10, 7 & 8

    Windows background services enable Windows features function properly. If some errors happen to serv ...

  2. 诊断：MRP0: Background Media Recovery terminated with error 1111

    表现: 灾备环境,无法继续应用日志. 日志: MRP0: Background Media Recovery terminated with error 1111 Fri Jan 18 15:55:2 ...

  3. Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details

    thinkphp 在Apache上配置启用伪静态,重启Apache1 restart 竟然失败了,报错 Job for httpd.service failed because the control ...

  4. kkjcre1p: unable to spawn jobq slave process, slot 0, error 1089(Linux x86_64)补丁

    在shutdown immediately的时候,alert Log出现如下错误信息,并且不能正常关闭 kkjcre1p: unable to spawn jobq slave process, sl ...

  5. Web自动化框架之五一套完整demo的点点滴滴（excel功能案例参数化+业务功能分层设计+mysql数据存储封装+截图+日志+测试报告+对接缺陷管理系统+自动编译部署环境+自动验证false、error案例）

    标题很大,想说的很多,不知道从那开始~~直接步入正题吧 个人也是由于公司的人员的现状和项目的特殊情况,今年年中后开始折腾web自动化这块:整这个原因很简单,就是想能让自己偷点懒.也让减轻一点同事的苦力 ...

  6. protractor protractor.conf.js [launcher] Process exited with error code 1 undefined:1190

    y@y:karma-t01$ protractor protractor.conf.js [launcher] Process exited with error code undefined: vl ...

  7. CentOS启动docker1.13失败(Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.)

    一.启动失败 1.启动docker [root@localhost ~]# systemctl start docker Job for docker.service failed because t ...

  8. Job for network.service failed because the control process exited with error code

    转自:https://blog.csdn.net/dongfei2033/article/details/81124465 今天在centOS 7下更改完静态ip后发现network服务重启不了,翻遍 ...

  9. Job for docker.service failed because the control process exited with error

    Docker 无法启动 报错信息:Job for docker.service failed because the control process exited with error 找了很久才解决 ...

随机推荐

  1. 封装nodeJS中 $on $emit $off 事件

    事件绑定一个事件名称对应多个事件函数 应此它们的关系是一对多的关系 数据类型采用对象的形式 key:val 因为函数有多个 所以val选用数组   事件仓库 eventList = { key:val ...

  2. React 实现拖拽功能

    实现效果:(可戳 https://codepen.io/wenr/pen/EGEQxp 查看) 因为工作中会用到 JIRA 所以想实现一下相似的功能,顺便学习一下 H5 的拖拽.不支持拖拽改变顺序,感 ...

  3. Kafka对Java程序员有多重要？连阿里都再用它处理亿万级数据统计

    一．了解淘宝Kafka架构 在ActiveMQ.RabbitMQ.RocketMQ.Kafka消息中间件之间,我们为什么要选择Kafka?下面详细介绍一下,2012年9月份我在支付宝做余额宝研发,20 ...

  4. ThinkPHP 数据库操作(五) : 存储过程、数据集、分布式数据库

    存储过程 5.0支持存储过程,如果我们定义了一个数据库存储过程 sp_query ,可以使用下面的方式调用: $result = Db::query('call sp_query(8)'); 返回的是 ...

  5. 【Spark篇】--Spark中Standalone的两种提交模式

    一.前述 Spark中Standalone有两种提交模式,一个是Standalone-client模式,一个是Standalone-master模式. 二.具体         1.Standalon ...

  6. 『Two 树的直径求解及其运用』

    树的直径 我们先来认识一下树的直径. 树是连通无环图,树上任意两点之间的路径是唯一的.定义树上任意两点\(u, v\)的距离为\(u\)到\(v\)路径上边权的和.树的直径\(MN\)为树上最长路径, ...

  7. oauth2.0通过JdbcClientDetailsService从数据库读取相应的配置

    oauth2.0通过JdbcClientDetailsService从数据库读取相应的配置 在上一节我们讲述的配置是把授权码存储在redis中,把相应的请求的路径用使用in-memory存储 ,这个是 ...

  8. SpringBoot入门教程(九)定时任务Schedule

    在日常项目运行中,我们总会有需求在某一时间段周期性的执行某个动作.比如每天在某个时间段导出报表,或者每隔多久统计一次现在在线的用户量.在springboot中可以有很多方案去帮我们完成定时器的工作,有 ...

  9. Linux 的进程间通信：管道

    本文由云+社区发表 作者:邹立巍 版权声明: 本文章内容在非商业使用前提下可无需授权任意转载.发布. 转载.发布请务必注明作者和其微博.微信公众号地址,以便读者询问问题和甄误反馈,共同进步. 微博ID ...

  10. mysql数据库备份并且实现远程复制

    一.实现ssh 远程登陆 机器环境: 192.167.33.108 clent 用户:crawler 192.167.33.77 server 用户:crawler 1.客户端 生成密钥 /home/ ...