由于dns服务为启动导致的GI集群启动故障
1、物业由于突然断电导致grid集群重新启动后rac数据库无法正常启动，对集群进行检查，结果如下，发现其中有4个数据库状态为instance shutdown。
[root@node1 ~]# su - grid
[grid@node1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.FLASH.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.GRIDDG.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.LISTENER.lsnr
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.LTDG.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.ORADG.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.asm
ONLINE ONLINE node1 Started
ONLINE ONLINE node2 Started
ONLINE ONLINE node3 Started
ONLINE ONLINE node4 Started
ONLINE ONLINE node5 Started
ONLINE ONLINE node6 Started
ora.gsd
OFFLINE OFFLINE node1
OFFLINE OFFLINE node2
OFFLINE OFFLINE node3
OFFLINE OFFLINE node4
OFFLINE OFFLINE node5
OFFLINE OFFLINE node6
ora.net1.network
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.ons
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.registry.acfs
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE node3
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE node1
ora.cvu
1 ONLINE ONLINE node3
ora.efmisdb.db
1 ONLINE OFFLINE Instance Shutdown
ora.efmisdb.efmissrv.svc
1 ONLINE OFFLINE
ora.faspdb.db
1 ONLINE OFFLINE Instance Shutdown
ora.faspdb.faspsvc.svc
1 ONLINE OFFLINE
ora.ltdb.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE OFFLINE Instance Shutdown
ora.node1.vip
1 ONLINE ONLINE node1
ora.node2.vip
1 ONLINE ONLINE node2
ora.node3.vip
1 ONLINE ONLINE node3
ora.node4.vip
1 ONLINE ONLINE node4
ora.node5.vip
1 ONLINE ONLINE node5
ora.node6.vip
1 ONLINE ONLINE node6
ora.oadb.db
1 ONLINE ONLINE node6 Open
ora.oc4j
1 ONLINE ONLINE node3
ora.orcl.db
1 ONLINE ONLINE node5 Open
ora.scan1.vip
1 ONLINE ONLINE node2
ora.scan2.vip
1 ONLINE ONLINE node3
ora.scan3.vip
1 ONLINE ONLINE node1
2、asmcmd查看asm磁盘是否可以正常访问，确认asm正常：
[grid@node1 ~]$ asmcmd -p
ASMCMD [+] > ls
FLASH/
GRIDDG/
LTDG/
ORADG/
ASMCMD [+] > ls ltdg
ASMCMD [+] > cd oradg
ASMCMD [+oradg] > ls
EFMISDB/
FASPDB/
LTDB/
OADB/
ORCL/
ASMCMD [+oradg] > cd ltdb
ASMCMD [+oradg/ltdb] > ls
CONTROLFILE/
DATAFILE/
ONLINELOG/
PARAMETERFILE/
TEMPFILE/
spfileltdb.ora
ASMCMD [+oradg/ltdb] > cd datafile
ASMCMD [+oradg/ltdb/datafile] > ls
BSIP_JLPT.335.923932111
EFMIS.291.920050869
EFMIS.292.920050843
EFMIS.293.920050823
EFMIS.294.920050793
EFMIS.338.926285547
EFMIS.339.926285561
EFMIS_YS.327.922529787
EFMIS_YS.DBF
FASP2.290.920053427
FASP2.325.922112969
FASP2.326.922112977
FASP2.328.923068707
FASP2.329.923571833
FASP2.330.923571847
FASP2.340.926285581
FASP2TEST.334.923932089
SYSAUX.257.919523823
SYSTEM.256.919523823
UNDOTBS1.258.919523825
UNDOTBS2.264.919523987
USERS.259.919523825
ASMCMD [+oradg/ltdb/datafile] > exit
3、登录数据库，尝试启动数据库，此间打开alert日志：
[oracle@node1 ~]$ export ORACLE_SID=ltdb1
[oracle@node1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Tue Nov 1 12:32:15 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00132: syntax error or unresolved network name 'lt-cluster:1521'
SQL> exit
发现数据库启动报错119，查看以下alert日志发现进程kill因为119：
Tue Nov 01 12:32:18 2016
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = 32 KB
Total Shared Global Region in Large Pages = 0 KB (0%)
Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB
RECOMMENDATION:
Total System Global Area size is 3010 MB. For optimal performance,
prior to the next instance restart:
1. Increase the number of unused large pages by
at least 1505 (page size 2048 KB, total size 3010 MB) system wide to
get 100% of the System Global Area allocated with large pages
2. Large pages are automatically locked into physical memory.
Increase the per process memlock (soft) limit to at least 3018 MB to lock
100% System Global Area's large pages into physical memory
********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 4
Number of processor cores in the system is 4
Number of processor sockets in the system is 2
Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.149.234, mac=00-50-56-80-52-8e, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
[name='eth0', type=1, ip=192.168.100.61, mac=00-50-56-80-2b-89, net=192.168.100.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth0:1' configured from GPnP for use as a public interface.
[name='eth0:1', type=1, ip=192.168.100.70, mac=00-50-56-80-2b-89, net=192.168.100.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth0:6' configured from GPnP for use as a public interface.
[name='eth0:6', type=1, ip=192.168.100.81, mac=00-50-56-80-2b-89, net=192.168.100.0/24, mask=255.255.255.0, use=public/1]
CELL communication is configured to use 0 interface(s):
CELL IP affinity details:
NUMA status: non-NUMA system
cellaffinity.ora status: N/A
CELL communication will use 1 IP group(s):
Grp 0:
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options.
ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_1
System name: Linux
Node name: node1
Release: 2.6.18-308.el5
Version: #1 SMP Fri Jan 27 17:17:51 EST 2012
Machine: x86_64
VM name: VMWare Version: 6
Using parameter settings in server-side pfile /u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/initltdb1.ora
System parameters with non-default values:
processes = 800
sessions = 1224
spfile = "+ORADG/ltdb/spfileltdb.ora"
sga_target = 3008M
control_files = "+ORADG/ltdb/controlfile/current.260.919523907"
control_files = "+FLASH/ltdb/controlfile/current.256.919523907"
db_block_size = 8192
compatible = "11.2.0.4.0"
cluster_database = TRUE
db_create_file_dest = "+ORADG"
db_recovery_file_dest = "+FLASH"
db_recovery_file_dest_size= 4407M
thread = 1
undo_tablespace = "UNDOTBS1"
_partition_large_extents = "FALSE"
_index_partition_large_extents= "FALSE"
instance_number = 1
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=ltdbXDB)"
remote_listener = "lt-cluster:1521"
audit_file_dest = "/u01/app/oracle/admin/ltdb/adump"
audit_trail = "DB"
db_name = "ltdb"
open_cursors = 300
pga_aggregate_target = 998M
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
169.254.149.234
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Tue Nov 01 12:33:04 2016
USER (ospid: 31080): terminating the instance due to error 119
Instance terminated by USER, pid = 31080
4、目前为止可以看到GI集群启动正常，但是数据库无法打开，报错119，通过查看参数文件配置并未发现异常。通过nslookup检查scan ip解析情况，发现dns解析出现问题，查看dbs服务器，发现dns服务器并未启动。启动dns服务器后重启数据库服务器，发现可以正常打开数据库。
由于dns服务为启动导致的GI集群启动故障的更多相关文章
- SQL Server AG集群启动不起来的临时自救大招
SQL Server AG集群启动不起来的临时自救大招 背景 前晚一朋友遇到AG集群发生来回切换不稳定的情况,情急之下,朋友在命令行使用命令重启WSFC集群 结果重启WSFC集群之后,非但没有好转,导 ...
- Hadoop ha CDH5.15.1-hadoop集群启动后，两个namenode都是standby模式
Hadoop ha CDH5.15.1-hadoop集群启动后,两个namenode都是standby模式 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一说起周五,想必大家都特别 ...
- Deinstall卸载RAC之Oracle软件及数据库+GI集群软件
Deinstall卸载Oracle软件及数据库+GI集群软件 1. 本篇文档应用场景: 需要安装新的ORACLE RAC产品,系统没有重装,需要对原环境中的RAC进行卸载: #本篇文档,在AIX 6. ...
- 庐山真面目之十二微服务架构基于Docker搭建Consul集群、Ocelot网关集群和IdentityServer版本实现
庐山真面目之十二微服务架构基于Docker搭建Consul集群.Ocelot网关集群和IdentityServer版本实现 一.简介 在第七篇文章<庐山真面目之七微服务架构Consul ...
- SpringCloud升级之路2020.0.x版-20. 启动一个 Eureka Server 集群
本系列代码地址:https://github.com/HashZhang/spring-cloud-scaffold/tree/master/spring-cloud-iiford 我们的业务集群结构 ...
- zookeeper源码 — 二、集群启动—leader选举
上一篇介绍了zookeeper的单机启动,集群模式下启动和单机启动有相似的地方,但是也有各自的特点.集群模式的配置方式和单机模式也是不一样的,这一篇主要包含以下内容: 概念介绍:角色,服务器状态 服务 ...
- Hadoop ha CDH5.15.1-hadoop集群启动后，集群容量不正确，莫慌，这是正常的表现！
Hadoop ha CDH5.15.1-hadoop集群启动后,集群容量不正确,莫慌,这是正常的表现! 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.集群启动顺序 1>. ...
- 【ELK】【docker】6.Elasticsearch 集群启动多节点 + 解决ES节点集群状态为yellow
本章其实是ELK第二章的插入章节. 本章ES集群的多节点是docker启动在同一个虚拟机上 ====================================================== ...
- 转载：oracle RAC集群启动和关闭
http://www.cnblogs.com/yhfssp/p/8184761.html oracle 11G RAC集群启动和关闭: 1.停止数据库 $srvctl stop database –d ...
随机推荐
- R read.table函数的check.names参数
今天用cummeRbund 对cuffdiff的结果进行可视化, 一直报错,之前跑的好好的,找了半天原因, 原来出现在read.table这个函数上: read.table有一个参数check.nam ...
- RAC DBCA 找不到共享磁盘
(一) 前言: 通过vmware workstation 走iscsi协议.安装RAC 集群架构,DBCA 时不能识别ASM 共享存储(按理来说这一版都是权限的问题).同一时候,本想通过RMAN ...
- strip() 、lstrip() 、rstrip()
strip() 用于移除字符串开头和结尾的空格或换行符,如果指定参数,则表示移除指定的字符lstrip() 用于移除字符串开头的空格或换行符,如果指定参数,则表示移除指定的字符rstrip() 用于移 ...
- IT教程视频
声明:以下视频均来自与互联网各个高级培训机构内部视频,我们能保证大部分的链接均可用.但不能保证所有的视频内容都是最新的.如果想要实时跟进各个培训机构的内部视频建议您关注微信公众号(BjieCoder) ...
- 关于MCU的烧录，下载与其他接口的比较（一）
今天呢,犯了一个很严重的错误,我不知道这会产生什么样的影响,但我知道,如果我以后再没有具体的了解,仔细认真地观察,认证,只会滑到无底的深渊.做技术来不得半点虚假,切记一知半解,凡事都要弄得清楚明白,认 ...
- 使用类/结构体时关于ZeroMomery用法错误
今天同事在写了如下结构体: typedef struct _tagInfo { std::list<int> lst; std::vector<int> nVec; } INF ...
- Spring学习笔记--通过构造方法创建Bean
如果一个bean没有默认的构造函数,那么可以通过工厂方法来构造一个bean.Spring通过<bean>元素的factory-method属性来装配工厂创建的Bean.下面例子中的Stag ...
- piblog 0.2
在一个Web App中,所有的数据,包括用户的信息,日志,评论等,都存在数据库中.在piblog中使用MySQL作为数据库.Web App中由很多地方需要使用数据库.访问数据库需要创建数据库连接.游标 ...
- 【GIT】Git Flow最佳实践
Git Flow 工作流一共包含五种分支: 两个长期分支: 主分支 master:用于存放对外发布的版本,任何时候在这个分支拿到的,都是稳定的分布版 开发分支 develop:用于日常开发,存放最新的 ...
- LeetCode——Kth Largest Element in an Array
Description: Find the kth largest element in an unsorted array. Note that it is the kth largest elem ...