每年年底，系统管理员都要组织一次容灾方案的测试、演练。会在一个与生产环境网络隔离的DR环境中，启动各个“生产环境服务器”，然后让各路人员参与其中测试、演练容灾方案是否可靠。这次演练中，一台Oracle数据库服务器启动的时候遇到了问题。如下所示，启动的时候遇到ORA-03113: end-of-file on communication channel错误。

[oracle@mylnx6 ~]$ sqlplus / as sysdba

 

SQL*Plus: Release 10.2.0.5.0 - Production on Fri Dec 21 09:42:11 2018

 

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

 

Connected to an idle instance.

 

SQL> startup

ORA-03113: end-of-file on communication channel

SQL>

检查告警日志，发现数据库在启动的时候，报“ORA-00471: DBWR process terminated with error”错误。如下所示：

PMON started with pid=2, OS id=25005

PSP0 started with pid=3, OS id=25007

MMAN started with pid=4, OS id=25009

DBW0 started with pid=5, OS id=25011

LGWR started with pid=6, OS id=25013

CKPT started with pid=7, OS id=25016

SMON started with pid=8, OS id=25018

RECO started with pid=9, OS id=25020

CJQ0 started with pid=10, OS id=25022

MMON started with pid=11, OS id=25024

Fri Dec 21 09:44:36 CST 2018

starting up 8 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

MMNL started with pid=12, OS id=25026

Fri Dec 21 09:45:12 CST 2018

starting up 24 shared server(s) ...

Fri Dec 21 09:46:43 CST 2018

Errors in file /u01/app/oracle/admin/SCM2/bdump/scm2_pmon_25005.trc:

ORA-00471: DBWR process terminated with error

Fri Dec 21 09:46:43 CST 2018

PMON: terminating instance due to error 471

Instance terminated by PMON, pid = 25005

启动数据库实例的时候，报“ORA-00471: DBWR process terminated with error”这个错误，这个很蹊跷，很有可能是进程被系统给Kill掉了，检查操作系统的错误日志，发现出现了oom_kill_process,也就是说数据库实例启动的时候，由于系统内存资源紧张，DBWR进程被系统选作了牺牲品。具体错误日志如下所示：

Dec 21 09:46:39 mylnx6 kernel: oracle invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0

Dec 21 09:46:39 mylnx6 kernel: oracle cpuset=/ mems_allowed=0

Dec 21 09:46:39 mylnx6 kernel: Pid: 25026, comm: oracle Not tainted 2.6.32-200.13.1.el5uek #1

Dec 21 09:46:39 mylnx6 kernel: Call Trace:

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810a0b66>] ? cpuset_print_task_mems_allowed+0x92/0x9e

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810d9ae6>] oom_kill_process+0x85/0x25b

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810d9fbc>] ? select_bad_process+0xbc/0x102

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810da03f>] __out_of_memory+0x3d/0x86

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810da30f>] out_of_memory+0xfc/0x195

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810dd75e>] __alloc_pages_nodemask+0x487/0x595

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff811075ac>] alloc_page_vma+0xb9/0xc8

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810ff0a7>] read_swap_cache_async+0x52/0xf1

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810ff1a3>] swapin_readahead+0x5d/0x9c

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810d725a>] ? find_get_page+0x22/0x69

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff810f1ea3>] handle_mm_fault+0x44b/0x80f

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff8106d7cd>] ? getrusage+0x2b1/0x2ce

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff8101270e>] ? common_interrupt+0xe/0x13

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff81043696>] ? should_resched+0xe/0x2f

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff81456006>] do_page_fault+0x210/0x299

Dec 21 09:46:39 mylnx6 kernel:  [<ffffffff81453fd5>] page_fault+0x25/0x30

Dec 21 09:46:39 mylnx6 kernel: Mem-Info:

Dec 21 09:46:39 mylnx6 kernel: Node 0 DMA per-cpu:

Dec 21 09:46:39 mylnx6 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    4: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    5: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    6: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    7: hi:    0, btch:   1 usd:   0

Dec 21 09:46:39 mylnx6 kernel: Node 0 DMA32 per-cpu:

Dec 21 09:46:39 mylnx6 kernel: CPU    0: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    1: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    2: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    3: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    4: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    5: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    6: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: CPU    7: hi:  186, btch:  31 usd:   0

Dec 21 09:46:39 mylnx6 kernel: Node 0 Normal per-cpu:

Dec 21 09:46:39 mylnx6 kernel: CPU    0: hi:  186, btch:  31 usd:   0

Dec 21 09:46:40 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...

Dec 21 09:46:41 mylnx6 kernel: CPU    1: hi:  186, btch:  31 usd:   0

Dec 21 09:46:40 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...

Dec 21 09:46:41 mylnx6 kernel: CPU    2: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: CPU    3: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: CPU    4: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: CPU    5: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: CPU    6: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: CPU    7: hi:  186, btch:  31 usd:   0

Dec 21 09:46:41 mylnx6 kernel: active_anon:1764 inactive_anon:209 isolated_anon:64

Dec 21 09:46:41 mylnx6 kernel:  active_file:349 inactive_file:1710 isolated_file:0

Dec 21 09:46:41 mylnx6 kernel:  unevictable:5377 dirty:0 writeback:4 unstable:0

Dec 21 09:46:41 mylnx6 kernel:  free:29838 slab_reclaimable:2400 slab_unreclaimable:119491

Dec 21 09:46:41 mylnx6 kernel:  mapped:2703 shmem:830 pagetables:9849 bounce:0

Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA free:15652kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15172kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 3000 24210 24210

Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA32 free:86296kB min:2464kB low:3080kB high:3696kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 0 21210 21210

Dec 21 09:46:41 mylnx6 kernel: Node 0 Normal free:17404kB min:17440kB low:21800kB high:26160kB active_anon:7056kB inactive_anon:836kB active_file:1396kB inactive_file:6840kB unevictable:21508kB isolated(anon):256kB isolated(file):0kB present:21719040kB mlocked:21504kB dirty:0kB writeback:16kB mapped:10812kB shmem:3320kB slab_reclaimable:9600kB slab_unreclaimable:477964kB kernel_stack:2800kB pagetables:39396kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:544 all_unreclaimable? no

Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 0 0 0

Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA: 1*4kB 2*8kB 1*16kB 0*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15652kB

Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA32: 12*4kB 13*8kB 2*16kB 5*32kB 5*64kB 11*128kB 3*256kB 7*512kB 6*1024kB 4*2048kB 16*4096kB = 86296kB

Dec 21 09:46:41 mylnx6 kernel: Node 0 Normal: 420*4kB 1917*8kB 49*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17800kB

Dec 21 09:46:41 mylnx6 kernel: 4722 total pagecache pages

Dec 21 09:46:41 mylnx6 kernel: 694 pages in swap cache

Dec 21 09:46:41 mylnx6 kernel: Swap cache stats: add 589182, delete 588488, find 343370/443306

Dec 21 09:46:41 mylnx6 kernel: Free swap  = 66723056kB

Dec 21 09:46:41 mylnx6 kernel: Total swap = 67108856kB

Dec 21 09:46:41 mylnx6 kernel: 6291440 pages RAM

Dec 21 09:46:41 mylnx6 kernel: 107316 pages reserved

Dec 21 09:46:41 mylnx6 kernel: 24060 pages shared

Dec 21 09:46:41 mylnx6 kernel: 77648 pages non-shared

Dec 21 09:46:41 mylnx6 kernel: Out of memory: kill process 25011 (oracle) score 8425150 or a child

Dec 21 09:46:41 mylnx6 kernel: Killed process 25011 (oracle)

Dec 21 09:47:20 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...

检查这个系统的内存，发现DR环境下，这个服务器只分配了24G内存，而实际生产环境的内存为64G（设置了Linux标准大页，而且SGA_MAX_SIZE大小为32G），而且这个环境是生产环境的“克隆体”，只是由于资源限制的缘故，系统管理员只分配24G内存。如下所示：

[root@mylnx6 ~]# free -m

             total       used       free     shared    buffers     cached

Mem:         24156      24033        123          0          0          6

-/+ buffers/cache:      24026        130

Swap:        65535         41      65494

[root@mylnx6 ~]# ps -ef | grep ora_

root     11759 11490  0 16:10 pts/1    00:00:00 grep ora_

[root@mylnx6 ~]# ipcs -m

 

------ Shared Memory Segments --------

key        shmid      owner      perms      bytes      nattch     status      

0x00000000 3080192    root      644        80         2                       

0x00000000 3112961    root      644        16384      2                       

0x00000000 3145730    root      644        280        2                       

0x00000000 4096003    gdm       600        393216     0                       

0x2cd12178 3866628    oracle    640        34361835520 0                       

0x00000000 5210117    gdm       600        393216     2          dest  

如上所示，可以看到oracle用户的共享内存段为34361835520字节。所以引起这个错误的原因是因为在系统层面配置了标准大页的缘故（内存资源变化了，但是配置没有随之修改），为了快速解决问题，我们取消标准大页的相关设置。如下所示：

修改limits.conf参数，注释soft memlock和hard memlock参数。

vi /etc/security/limits.conf

然后修改sysctl.conf，将vm.nr_hugepages注释掉。然后重启一下（DR测试环境，可以随时重启）。然后启动Oracle数据库实例，一切正常，当然还需调整相关参数，继续后续测试~。

ORA-00471: DBWR process terminated with error案例的更多相关文章

  1. [SOLVED] “Error 1067: The process terminated unexpectedly” on Windows 10, 7 & 8

    Windows background services enable Windows features function properly. If some errors happen to serv ...

  2. 诊断：MRP0: Background Media Recovery terminated with error 1111

    表现: 灾备环境,无法继续应用日志. 日志: MRP0: Background Media Recovery terminated with error 1111 Fri Jan 18 15:55:2 ...

  3. Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details

    thinkphp 在Apache上配置启用伪静态,重启Apache1 restart 竟然失败了,报错 Job for httpd.service failed because the control ...

  4. kkjcre1p: unable to spawn jobq slave process, slot 0, error 1089(Linux x86_64)补丁

    在shutdown immediately的时候,alert Log出现如下错误信息,并且不能正常关闭 kkjcre1p: unable to spawn jobq slave process, sl ...

  5. Web自动化框架之五一套完整demo的点点滴滴（excel功能案例参数化+业务功能分层设计+mysql数据存储封装+截图+日志+测试报告+对接缺陷管理系统+自动编译部署环境+自动验证false、error案例）

    标题很大,想说的很多,不知道从那开始~~直接步入正题吧 个人也是由于公司的人员的现状和项目的特殊情况,今年年中后开始折腾web自动化这块:整这个原因很简单,就是想能让自己偷点懒.也让减轻一点同事的苦力 ...

  6. protractor protractor.conf.js [launcher] Process exited with error code 1 undefined:1190

    y@y:karma-t01$ protractor protractor.conf.js [launcher] Process exited with error code undefined: vl ...

  7. CentOS启动docker1.13失败(Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.)

    一.启动失败 1.启动docker [root@localhost ~]# systemctl start docker Job for docker.service failed because t ...

  8. Job for network.service failed because the control process exited with error code

    转自:https://blog.csdn.net/dongfei2033/article/details/81124465 今天在centOS 7下更改完静态ip后发现network服务重启不了,翻遍 ...

  9. Job for docker.service failed because the control process exited with error

    Docker 无法启动 报错信息:Job for docker.service failed because the control process exited with error 找了很久才解决 ...

随机推荐

  1. [Swift]LeetCode504. 七进制数 | Base 7

    Given an integer, return its base 7 string representation. Example 1: Input: 100 Output: "202&q ...

  2. Ubuntu12.04下安装NS3.25

    实验室项目的需求,要使用到NS3做仿真,使用实验室的电脑,Ubuntu12.04,版本比较低,建议大家使用16.04,这样安装依赖文件时不会出现版本过低的问题 (没管最后也安装成功了).下面就安装步骤 ...

  3. LeetCode算法题-Image Smoother（Java实现）

    这是悦乐书的第282次更新,第299篇原创 01 看题和准备 今天介绍的是LeetCode算法题中Easy级别的第150题(顺位题号是661).给定表示图像灰度的2D整数矩阵M,您需要设计一个平滑器以 ...

  4. 第一次c语言上机

    实验结论 part1: 第一部分的内容是按照书上所给的例题,进行简单的验证.虽然听起来很简单,但是由于之前并未接触过这方面的内容,还是犯了很多微小的错误.主要是在进行编程语言的输入时会输错字母,会忘记 ...

  5. Java8 新特性 | 如何风骚走位防止空指针异常

    文章整理翻译自 https://winterbe.com/posts/2015/03/15/avoid-null-checks-in-java/ 文章首发于个人网站: https://www.exce ...

  6. ECMAScript 6 学习（一）generator函数

    1.ES2017标准引入async函数,那么async函数到底是个什么函数呢? async 是一个generator函数的语法糖. 2.那么generator函数到底是什么函数ne? generato ...

  7. Elasticsearch Search API

    当执行一个搜索时,它将这个搜索请求广播给所有的索引分片.可以通过提供路由参数来控制要搜索哪些分片.例如,当检索tweets这个索引时,路由参数可以设置为用户名: curl -X POST " ...

  8. java基础(一)-----java的三大特性之封装

    面向对象编程有三大特性:封装.继承.多态.本文将介绍java的三大特性之封装 封装 封装从字面上来理解就是包装的意思,专业点就是信息隐藏,是指利用抽象数据类型将数据和基于数据的操作封装在一起,使其构成 ...

  9. 从锅炉工到AI专家(9)

    无监督学习 前面已经说过了无监督学习的概念.无监督学习在实际的工作中应用还是比较多见的. 从典型的应用上说,监督学习比较多用在"分类"上,利用给定的数据,做出一个决策,这个决策在有 ...

  10. source map 的原理探究

    线上产品代码一般是编译过的,前端的编译处理过程包括不限于 转译器/Transpilers (Babel, Traceur) 编译器/Compilers (Closure Compiler, TypeS ...