1.背景

OS:SUSE 12SP3

DB:12.2.0.1.190115 2节点RAC


Q:crs alert日志一直刷如下报错

2019-02-12 12:46:18.163 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:21.161 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:24.168 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:27.167 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:39.163 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:42.161 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:48.167 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:51.165 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:51.724 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:51.789 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:54.166 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:46:57.167 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:47:00.158 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:47:06.160 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:47:09.167 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:47:12.166 [ORAAGENT(91622)]CRS-5013: Agent "ORAAGENT" failed to start process "/oracle/app/12.2.0/grid/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:47:13.588 [ORAAGENT(91622)]CRS-5016: Process "/oracle/app/12.2.0/grid/opmn/bin/onsctli" spawned by agent "ORAAGENT" for action "check" failed: details at "(:CLSN00010:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:47:18.743 [ORAAGENT(91622)]CRS-5016: Process "/oracle/app/12.2.0/grid/opmn/bin/onsctli" spawned by agent "ORAAGENT" for action "start" failed: details at "(:CLSN00010:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2019-02-12 12:47:23.852 [ORAAGENT(91622)]CRS-5016: Process "/oracle/app/12.2.0/grid/opmn/bin/onsctli" spawned by agent "ORAAGENT" for action "check" failed: details at "(:CLSN00010:)" in "/oracle/app/grid/diag/crs/ssng3mcs-db2/crs/trace/crsd_oraagent_grid.trc"

2.检查DB、ASM alert日志

DB alert

Errors in file /oracle/app/oracle/diag/rdbms/mcsdb/MCSDB2/trace/MCSDB2_psp0_93951.trc:

ORA-27300: OS system dependent operation:fork failed with status: 11

ORA-27301: OS failure message: Resource temporarily unavailable

ORA-27302: failure occurred at: skgpspawn5

ASM alert

2019-02-12T16:06:08.891616+08:00

Process startup failed, error stack:

2019-02-12T16:06:08.891871+08:00

Errors in file /oracle/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_psp0_92477.trc:

ORA-27300: OS system dependent operation:fork failed with status: 11

ORA-27301: OS failure message: Resource temporarily unavailable

ORA-27302: failure occurred at: skgpspawn3

2019-02-12T16:06:09.889920+08:00

Process m000 died, see its trace file

3.搜MOS

  • MOS
1.Database And ASM Instance Ora-27300 OS System Dependent Operation Fork Failed With Status 11 (文档 ID 2331884.1)

2.SLES 12: Database Startup Error with ORA-27300 ORA-27301 ORA-27303 While Starting using Srvctl (文档 ID 2340986.1)
  • 参考链接
http://feed.askmaclean.com/archives/suse-12-redhat-7%E4%B8%AD%E7%9A%84ora-27300-os-system-dependent-operationfork-failed-with-status-11%E7%9A%84%E6%95%85%E9%9A%9C%E5%A4%84%E7%90%86.html
  • 根据上面的参考，检查本环境
cat /etc/security/limits.conf

ps h -Led -o user | sort | uniq -c | sort -n


cat /etc/systemd/system.conf|grep DefaultTasksMax

#DefaultTasksMax=512

# systemctl status ohasd

● ohasd.service - LSB: Start and Stop Oracle High Availability Service

   Loaded: loaded (/etc/init.d/ohasd; bad; vendor preset: disabled)

   Active: active (exited) since Tue 2019-02-12 12:42:37 CST; 4h 3min ago

     Docs: man:systemd-sysv-generator(8)

  Process: 78777 ExecStart=/etc/init.d/ohasd start (code=exited, status=0/SUCCESS)

    Tasks: 512 (limit: 512)    ========>限制是512

Feb 12 12:42:32 SSNG3MCS-DB2 systemd[1]: Starting LSB: Start and Stop Oracle High Availability Service...

Feb 12 12:42:32 SSNG3MCS-DB2 ohasd[78777]: Starting ohasd:

Feb 12 12:42:37 SSNG3MCS-DB2 ohasd[78777]: CRS-4123: Oracle High Availability Services has been started.

Feb 12 12:42:37 SSNG3MCS-DB2 systemd[1]: Started LSB: Start and Stop Oracle High Availability Service.

4.解决方案

1.vi /etc/systemd/system.conf

Set DefaultTasksMax to 'infinity'

2.Restart OS

5.问题原因

  在Linux 7或Suse 12上,使用了systemd新的启动方式,这个在Linux 6和SUSE 11上是没有的,当这个启动之后,就会忽略掉/etc/security/limits.conf下的设置。

  而该文件的一个参数，DefaultTasksMax设置为默认值（512），限制了可在节点上创建的最大任务数。此设置还影响OS上的maxpid值。

6. SUSE 12sp3修改方法及参数检查

#################添加limits参数###################

在/etc/security/limits.conf文件中添加参数：

###################For HW  soft #################

*               soft     nofile         1200000

*               hard     nofile         1220000

*               soft     memlock        32

*               soft     core           10485760

*               soft     data           -1

*               soft     nproc          148270

*               soft     stack          -1

*               soft     as             -1

*               soft     rss            -1

*               hard    nproc       1200000

############添加内核参数##################

在/etc/sysctl.conf中添加参数：

fs.aio-max-nr = 1048576

fs.file-max = 6815744

fs.nr_open = 2000000

fs.inotify.max_user_watches = 2000000

kernel.acct = 100 100 30

kernel.msgmax = 1048576

kernel.msgmnb = 8388608

kernel.msgmni = 256

kernel.sem = 1250 320000 100 256

##############512G#####################

kernel.shmmax = 433517314048

kernel.shmmni = 4096

##############512G#####################

kernel.shmall = 549755813888

kernel.suid_dumpable = 1

kernel.sysrq = 8

kernel.core_pattern = /corefiles/core.%p.%e

vm.min_free_kbytes = 3000000

vm.swappiness = 10

net.core.rmem_default = 8388608

net.core.rmem_max = 16777216

net.core.wmem_default = 8388608

net.core.wmem_max = 41943040

net.ipv4.ip_local_port_range = 50000 65000

net.ipv4.tcp_fin_timeout = 5

net.ipv4.tcp_keepalive_time = 1200

net.ipv4.tcp_rmem = 8388608 8388608 33554432

net.ipv4.tcp_sack = 1

net.ipv4.tcp_timestamps = 1

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_wmem = 524288 524288 33554432

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_retries2 = 5

net.ipv4.tcp_syn_retries = 1

net.ipv4.icmp_echo_ignore_broadcasts = 1

net.ipv4.conf.all.rp_filter = 1

net.ipv4.conf.default.promote_secondaries = 1

net.ipv4.conf.all.promote_secondaries = 1

net.ipv4.conf.all.arp_ignore = 1

net.ipv4.conf.default.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 1

net.ipv4.conf.default.arp_announce = 1

net.ipv4.tcp_max_syn_backlog = 8192

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_synack_retries = 5

vm.overcommit_memory = 1

vm.drop_caches = 1

vm.zone_reclaim_mode = 0

vm.max_map_count = 655360

vm.dirty_background_ratio = 60

vm.dirty_ratio = 60

vm.page-cluster = 3

vm.dirty_writeback_centisecs = 360000

net.ipv4.conf.all.accept_source_route = 0

net.ipv4.conf.all.accept_redirects = 0

net.ipv4.icmp_ignore_bogus_error_responses = 1

kernel.sched_child_runs_first = 1

kernel.sched_latency_ns = 40000000

kernel.sched_nr_migrate = 64

net.ipv4.tcp_moderate_rcvbuf = 1

kernel.sched_compat_yield = 1

net.ipv4.tcp_max_tw_buckets = 5000

kernel.sched_migration_cost = 250000

kernel.sched_min_granularity_ns = 8000000

kernel.sched_wakeup_granularity_ns = 2500000

kernel.sched_rt_period_us = 2000000

kernel.pid_max = 1310720

/sbin/sysctl -p

#####################修改system.conf参数#####################

在/etc/systemd/system.conf中将#DefaultTasksMax=512修改为DefaultTasksMax=infinity

...重启OS...


# systemctl status ohasd

● ohasd.service - LSB: Start and Stop Oracle High Availability Service

   Loaded: loaded (/etc/init.d/ohasd; bad; vendor preset: disabled)

   Active: active (exited) since Thu 2019-02-21 14:27:32 CST; 2h 9min ago

     Docs: man:systemd-sysv-generator(8)

  Process: 86737 ExecStart=/etc/init.d/ohasd start (code=exited, status=0/SUCCESS)

    Tasks: 959

Feb 21 14:27:26 SSNG3MCS-DB1 systemd[1]: Starting LSB: Start and Stop Oracle High Availability Service...

Feb 21 14:27:26 SSNG3MCS-DB1 ohasd[86737]: Starting ohasd:

Feb 21 14:27:31 SSNG3MCS-DB1 ohasd[86737]: CRS-4123: Oracle High Availability Services has been started.

Feb 21 14:27:32 SSNG3MCS-DB1 systemd[1]: Started LSB: Start and Stop Oracle High Availability Service.

# cat /etc/systemd/system.conf|grep DefaultTasksMax

#DefaultTasksMax=512

DefaultTasksMax=infinity

Oracle 12C CRS-5013的更多相关文章

  1. 安装oracle 12c RAC遇到的一些问题

    (1) 安装grid软件,停止在38%很长时间不动,日志显示正常   解决方法: 由于是虚拟机安装,设置的内存为600M,关闭虚拟机,把内存调成1GB,问题解决~在38%Linking RMAN Ut ...

  2. Oracle 12c RAC 静默安装文档

    参考文档: https://docs.oracle.com/en/database/oracle/oracle-database/12.2/cwlin/index.html https://docs. ...

  3. ORACLE 12c RAC的常用管理命令

    ORACLE 12c RAC的常用管理命令 一.查询 1.RAC查询 1.1.查询节点 [grid@swnode1 ~]$ su - grid [grid@swnode1 ~]$ /u01/app/1 ...

  4. ORACLE 12C 之集群日志位置变化

    如果你还是使用 oracle 11g RAC 的目录结构方式寻找集群的日志,你会发现目录中所有的日志都是空的.actdb21:/oracle/app/12.2.0/grid/log/actdb21(+ ...

  5. Downgrade ASM DATABASE_COMPATIBILITY (from 11.2.0.4.0 to 11.2.0.0.0) on 12C CRS stack.

    使用Onecommand完成快速Oracle 12c RAC部署后 发现ASM database compatibilty无法设置,默认为11.2.0.4.0. 由于我们还有些数据库低于这个版本,所以 ...

  6. ORACLE 12C新特性——CDB与PDB

    Oracle 12C引入了CDB与PDB的新特性,在ORACLE 12C数据库引入的多租用户环境(Multitenant Environment)中,允许一个数据库容器(CDB)承载多个可插拔数据库( ...

  7. Oracle 12c 使用scott等普通用户的方法

    目录: 一.前言 二.使用普通用户 三.自动启动PDB 一.前言 最近电脑上安装了oracle 12c数据库,想体验下新特性.安装完后,便像11g一样在dos窗口进行下面的操作: SQL Produc ...

  8. [原创]Oracle 12c 抢先安装手迹

    [前言] Oracle 12c 终于投放市场了,唉,等了很久了.据官方说这是一个为云计算平台量身定做的版本....且不管真的假的,先让我们把它装上再说. 注:笔者在安装的过程中发现12c的安装过程,较 ...

  9. java开发连接Oracle 12c采用PDB遇到问题记录

    今天初次使用java连接Oracle 12c,遇到各种问题,为方便后续查询,在汇总了问题记录及解决方案如下. ORA-28040: No matching authentication protoco ...

  10. oracle 12c 创建PDB用户即Local User (PDB与CDB)

    Oracle 12C用户创建与表空间分配  数据库安装完成后,首先用系统用户链接数据库容器(CDB), 在数据库容器(CDB)中创建表空间‘imei’ SQL>create tablespace ...

随机推荐

  1. C/S,B/S应用的区别

    ——————C/S,B/S应用的区别 C/S是指客户端/服务器架构,是一种典型的两层架构,可充分发挥客户端pc的性能,相对减轻服务器的压力. B/S是指浏览器/服务器架构,区别于传统的C/S(客户端/ ...

  2. Centos下安装配置Mongodb3.6

    首先更新系统 yum -y update 1.安装Mongodb 编辑Mongodb安装源 vim /etc/yum.repos.d/mongodb-org-3.6.repo 编辑内容如下: [mon ...

  3. arraylist 为什么 删除元素时要使用迭代器而不能使用遍历

    因为你要是遍历了,arraylist 的长度就变了,容易数组越界和下标问题 public class Test {     public static void main(String[] args) ...

  4. 理解上下文Context

    --摘自<Android进阶解密> 知识点: 1.Context的使用场景 1)使用Context调用方法,比如启动Activity.访问资源.调用系统级服务等 2)调用方法时传入Cont ...

  5. centos没有可用软件包 libgtk2

    在编写一个C程序,使用基于文本的终端图形编程库(curses)或图形界面(QT/GTK),分窗口显示三个并发进程的运行,因为centos并未安装gtk,所以运行命令行安装gtksudo yum -y ...

  6. POJ 2391 Ombrophobic Bovines (二分答案+floyd+最大流)

    <题目链接> 题目大意: 给定一个有$n$个顶点和$m$条边的无向图,点$i$ 处有$A_i$头牛,点$i$ 处的牛棚能容纳$B_i$头牛,每条边有一个时间花费$t_i$(表示从一个端点走 ...

  7. Vue学习陷阱

    v-for在嵌套时index没办法重复用,内循环与外循环不能共用一个index <swiper-item v-for="(items,index) in swiperList" ...

  8. js 把 json 转为以 ‘&’ 连接的字符串

    /** * URL编码; * @param {参数} param */ export function toParams(param) { var result = "" for ...

  9. ECMA Script 6_行为重定义 Proxy

    行为重定义 Proxy 在目标对象之前架设一层“拦截”,外界对该对象的访问,都必须先通过这层拦截 因此提供了一种机制,可以对外界的访问进行过滤和改写 Proxy 这个词的原意是代理,用在这里表示由它来 ...

  10. css3图形绘制

    以下几个例子主要是运用了css3中border.bordr-radius.transform.伪元素等属性来完成的,我们先了解下它们的基本原理. border:简单的来说border语法主要包含(bo ...