由于dns服务为启动导致的GI集群启动故障
1、物业由于突然断电导致grid集群重新启动后rac数据库无法正常启动，对集群进行检查，结果如下，发现其中有4个数据库状态为instance shutdown。
[root@node1 ~]# su - grid
[grid@node1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.FLASH.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.GRIDDG.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.LISTENER.lsnr
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.LTDG.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.ORADG.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.asm
ONLINE ONLINE node1 Started
ONLINE ONLINE node2 Started
ONLINE ONLINE node3 Started
ONLINE ONLINE node4 Started
ONLINE ONLINE node5 Started
ONLINE ONLINE node6 Started
ora.gsd
OFFLINE OFFLINE node1
OFFLINE OFFLINE node2
OFFLINE OFFLINE node3
OFFLINE OFFLINE node4
OFFLINE OFFLINE node5
OFFLINE OFFLINE node6
ora.net1.network
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.ons
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
ora.registry.acfs
ONLINE ONLINE node1
ONLINE ONLINE node2
ONLINE ONLINE node3
ONLINE ONLINE node4
ONLINE ONLINE node5
ONLINE ONLINE node6
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE node3
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE node1
ora.cvu
1 ONLINE ONLINE node3
ora.efmisdb.db
1 ONLINE OFFLINE Instance Shutdown
ora.efmisdb.efmissrv.svc
1 ONLINE OFFLINE
ora.faspdb.db
1 ONLINE OFFLINE Instance Shutdown
ora.faspdb.faspsvc.svc
1 ONLINE OFFLINE
ora.ltdb.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE OFFLINE Instance Shutdown
ora.node1.vip
1 ONLINE ONLINE node1
ora.node2.vip
1 ONLINE ONLINE node2
ora.node3.vip
1 ONLINE ONLINE node3
ora.node4.vip
1 ONLINE ONLINE node4
ora.node5.vip
1 ONLINE ONLINE node5
ora.node6.vip
1 ONLINE ONLINE node6
ora.oadb.db
1 ONLINE ONLINE node6 Open
ora.oc4j
1 ONLINE ONLINE node3
ora.orcl.db
1 ONLINE ONLINE node5 Open
ora.scan1.vip
1 ONLINE ONLINE node2
ora.scan2.vip
1 ONLINE ONLINE node3
ora.scan3.vip
1 ONLINE ONLINE node1
2、asmcmd查看asm磁盘是否可以正常访问，确认asm正常：
[grid@node1 ~]$ asmcmd -p
ASMCMD [+] > ls
FLASH/
GRIDDG/
LTDG/
ORADG/
ASMCMD [+] > ls ltdg
ASMCMD [+] > cd oradg
ASMCMD [+oradg] > ls
EFMISDB/
FASPDB/
LTDB/
OADB/
ORCL/
ASMCMD [+oradg] > cd ltdb
ASMCMD [+oradg/ltdb] > ls
CONTROLFILE/
DATAFILE/
ONLINELOG/
PARAMETERFILE/
TEMPFILE/
spfileltdb.ora
ASMCMD [+oradg/ltdb] > cd datafile
ASMCMD [+oradg/ltdb/datafile] > ls
BSIP_JLPT.335.923932111
EFMIS.291.920050869
EFMIS.292.920050843
EFMIS.293.920050823
EFMIS.294.920050793
EFMIS.338.926285547
EFMIS.339.926285561
EFMIS_YS.327.922529787
EFMIS_YS.DBF
FASP2.290.920053427
FASP2.325.922112969
FASP2.326.922112977
FASP2.328.923068707
FASP2.329.923571833
FASP2.330.923571847
FASP2.340.926285581
FASP2TEST.334.923932089
SYSAUX.257.919523823
SYSTEM.256.919523823
UNDOTBS1.258.919523825
UNDOTBS2.264.919523987
USERS.259.919523825
ASMCMD [+oradg/ltdb/datafile] > exit
3、登录数据库，尝试启动数据库，此间打开alert日志：
[oracle@node1 ~]$ export ORACLE_SID=ltdb1
[oracle@node1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Tue Nov 1 12:32:15 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00132: syntax error or unresolved network name 'lt-cluster:1521'
SQL> exit
发现数据库启动报错119，查看以下alert日志发现进程kill因为119：
Tue Nov 01 12:32:18 2016
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = 32 KB
Total Shared Global Region in Large Pages = 0 KB (0%)
Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB
RECOMMENDATION:
Total System Global Area size is 3010 MB. For optimal performance,
prior to the next instance restart:
1. Increase the number of unused large pages by
at least 1505 (page size 2048 KB, total size 3010 MB) system wide to
get 100% of the System Global Area allocated with large pages
2. Large pages are automatically locked into physical memory.
Increase the per process memlock (soft) limit to at least 3018 MB to lock
100% System Global Area's large pages into physical memory
********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 4
Number of processor cores in the system is 4
Number of processor sockets in the system is 2
Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.149.234, mac=00-50-56-80-52-8e, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
[name='eth0', type=1, ip=192.168.100.61, mac=00-50-56-80-2b-89, net=192.168.100.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth0:1' configured from GPnP for use as a public interface.
[name='eth0:1', type=1, ip=192.168.100.70, mac=00-50-56-80-2b-89, net=192.168.100.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth0:6' configured from GPnP for use as a public interface.
[name='eth0:6', type=1, ip=192.168.100.81, mac=00-50-56-80-2b-89, net=192.168.100.0/24, mask=255.255.255.0, use=public/1]
CELL communication is configured to use 0 interface(s):
CELL IP affinity details:
NUMA status: non-NUMA system
cellaffinity.ora status: N/A
CELL communication will use 1 IP group(s):
Grp 0:
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options.
ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_1
System name: Linux
Node name: node1
Release: 2.6.18-308.el5
Version: #1 SMP Fri Jan 27 17:17:51 EST 2012
Machine: x86_64
VM name: VMWare Version: 6
Using parameter settings in server-side pfile /u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/initltdb1.ora
System parameters with non-default values:
processes = 800
sessions = 1224
spfile = "+ORADG/ltdb/spfileltdb.ora"
sga_target = 3008M
control_files = "+ORADG/ltdb/controlfile/current.260.919523907"
control_files = "+FLASH/ltdb/controlfile/current.256.919523907"
db_block_size = 8192
compatible = "11.2.0.4.0"
cluster_database = TRUE
db_create_file_dest = "+ORADG"
db_recovery_file_dest = "+FLASH"
db_recovery_file_dest_size= 4407M
thread = 1
undo_tablespace = "UNDOTBS1"
_partition_large_extents = "FALSE"
_index_partition_large_extents= "FALSE"
instance_number = 1
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=ltdbXDB)"
remote_listener = "lt-cluster:1521"
audit_file_dest = "/u01/app/oracle/admin/ltdb/adump"
audit_trail = "DB"
db_name = "ltdb"
open_cursors = 300
pga_aggregate_target = 998M
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
169.254.149.234
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Tue Nov 01 12:33:04 2016
USER (ospid: 31080): terminating the instance due to error 119
Instance terminated by USER, pid = 31080
4、目前为止可以看到GI集群启动正常，但是数据库无法打开，报错119，通过查看参数文件配置并未发现异常。通过nslookup检查scan ip解析情况，发现dns解析出现问题，查看dbs服务器，发现dns服务器并未启动。启动dns服务器后重启数据库服务器，发现可以正常打开数据库。
由于dns服务为启动导致的GI集群启动故障的更多相关文章
- SQL Server AG集群启动不起来的临时自救大招
SQL Server AG集群启动不起来的临时自救大招 背景 前晚一朋友遇到AG集群发生来回切换不稳定的情况,情急之下,朋友在命令行使用命令重启WSFC集群 结果重启WSFC集群之后,非但没有好转,导 ...
- Hadoop ha CDH5.15.1-hadoop集群启动后，两个namenode都是standby模式
Hadoop ha CDH5.15.1-hadoop集群启动后,两个namenode都是standby模式 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一说起周五,想必大家都特别 ...
- Deinstall卸载RAC之Oracle软件及数据库+GI集群软件
Deinstall卸载Oracle软件及数据库+GI集群软件 1. 本篇文档应用场景: 需要安装新的ORACLE RAC产品,系统没有重装,需要对原环境中的RAC进行卸载: #本篇文档,在AIX 6. ...
- 庐山真面目之十二微服务架构基于Docker搭建Consul集群、Ocelot网关集群和IdentityServer版本实现
庐山真面目之十二微服务架构基于Docker搭建Consul集群.Ocelot网关集群和IdentityServer版本实现 一.简介 在第七篇文章<庐山真面目之七微服务架构Consul ...
- SpringCloud升级之路2020.0.x版-20. 启动一个 Eureka Server 集群
本系列代码地址:https://github.com/HashZhang/spring-cloud-scaffold/tree/master/spring-cloud-iiford 我们的业务集群结构 ...
- zookeeper源码 — 二、集群启动—leader选举
上一篇介绍了zookeeper的单机启动,集群模式下启动和单机启动有相似的地方,但是也有各自的特点.集群模式的配置方式和单机模式也是不一样的,这一篇主要包含以下内容: 概念介绍:角色,服务器状态 服务 ...
- Hadoop ha CDH5.15.1-hadoop集群启动后，集群容量不正确，莫慌，这是正常的表现！
Hadoop ha CDH5.15.1-hadoop集群启动后,集群容量不正确,莫慌,这是正常的表现! 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.集群启动顺序 1>. ...
- 【ELK】【docker】6.Elasticsearch 集群启动多节点 + 解决ES节点集群状态为yellow
本章其实是ELK第二章的插入章节. 本章ES集群的多节点是docker启动在同一个虚拟机上 ====================================================== ...
- 转载：oracle RAC集群启动和关闭
http://www.cnblogs.com/yhfssp/p/8184761.html oracle 11G RAC集群启动和关闭: 1.停止数据库 $srvctl stop database –d ...
随机推荐
- vector 排序
#include <vector> #include <algorithm> 一.vector保存的是基础数据类型(int.char.float等) vector<int ...
- 解决ubuntu下mysql不能远程连接数据库的问题【转】
Ubuntu10.04上自带的MySQL,执行了root@ubuntu:~#sudo apt-get install mysql安装完mysql-server 启动mysqlroot@ubuntu:~ ...
- Mysql经常使用基本命令汇总及默认账户权限与改动
一直仅仅是在浅显利用数据库存储数据.也被windows惯坏了.非常多命令使用的时候记不起来.so,换LINUX系统!不再使用GUI管理数据库!也想深入学习下Mysql.从权限管理開始.也就诞生了这篇学 ...
- php date strtotime的用法
1.上个月第一天及最后一天. echo date('Y-m-01', strtotime('-1 month')); echo strtotime(date('Y-m-01 0:00:00', str ...
- java基础——Collections.sort的两种用法
Collections是一个工具类,sort是其中的静态方法,是用来对List类型进行排序的,它有两种参数形式: public static <T extends Comparable<? ...
- Linux下oracle11g 导入导出操作详细
//用dba匿名登录 [oracle@enfo212 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.1.0 Production on Wed Ma ...
- SVN入门 TortoiseSVN 检出
1. SVN检出(SVN Checkout) 检出项目文件. 新建或者进入目录下(比如qianduan1),右键 --> Svn 检出-->其中版本库URL我可以在SVN服务器获取到,将复 ...
- android 开发者资源下载地址记录（转+补充）
https如果无法下载的话将下面的:https://dl-ssl 部分改为 http://dl (1)Android SDK (Android SDK主安装包,包含SDK Manager.AVD Ma ...
- 第七篇：几个经典的TCP通信函数
前言 在TCP通信中要使用到几个非常经典的函数,本文将对这几个函数进行一个简短的使用说明. socket()函数 函数作用:创建一个网际字节流套接字 包含头文件:sys/socket.h ( 后面几个 ...
- 关于C中I/O缓冲区的解释
用户程序调用C标准I/O库函数读写文件或设备,而这些库函数要通过系统调用把读写请求传给内核,最终由内核驱动磁盘或设备完成I/O操作.C标准库为每个打开的文件分配一个I/O缓冲区以加速读写操作,通过文件 ...