换了网线异常了,CRS无法正常启动,clssnmSendingThread: sending status msg to all nodes
同事换网线前我将节点2正常关闭了,换完网线告诉我,发现节点2死活起不来了,看上面的日志和一些帖子最后也没解决,尝试过重启、网线拔掉重新插上、查看过存储是否正常和存储重新挂载。。。。看过一个帖子说可能是OCR信息发生了改变,不过之前没备份,也没忘这方面深入考虑。
最后还是没搞定,主要是技术有限,没准确的定位出具体问题也不敢轻易乱动。。。
20xx-12-16 19:01:05.792: [ CSSD][3786819328]clssnmSendingThread: sending join msg to all nodes
20xx-12-16 19:01:05.792: [ CSSD][3786819328]clssnmSendingThread: sent 5 join msgs to all nodes
20xx-12-16 19:01:06.295: [GIPCHALO][3811858176] gipchaLowerProcessNode: no valid interfaces found to node for 7286464 ms, node 0x7fecd0028450 { host 'myrac1', haName 'CSS_myrac-cluster', srcLuid fac66ea4-f1a960af, dstLuid 00000000-00000000 numInf 0, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [249 : 249], createTime 7037424, sentRegister 1, localMonitor 1, flags 0x4 }
20xx-12-16 19:01:06.303: [ CSSD][3789973248]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
20xx-12-16 19:01:06.420: [ CSSD][3799754496]clssnmvDHBValidateNcopy: node 1, myrac1, has a disk HB, but no network HB, DHB has rcfg 471981092, wrtcnt, 211618800, LATS 7286584, lastSeqNo 211618797, uniqueness 1576485880, timestamp 1576494065/8540734
20xx-12-16 19:01:06.435: [ CSSD][3804591872]clssnmvDHBValidateNcopy: node 1, myrac1, has a disk HB, but no network HB, DHB has rcfg 471981092, wrtcnt, 211618802, LATS 7286594, lastSeqNo 211618799, uniqueness 1576485880, timestamp 1576494066/8541524
20xx-12-16 19:01:07.304: [ CSSD][3789973248]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
20xx-12-16 19:01:07.421: [ CSSD][3799754496]clssnmvDHBValidateNcopy: node 1, myrac1, has a disk HB, but no network HB, DHB has rcfg 471981092, wrtcnt, 211618803, LATS 7287584, lastSeqNo 211618800, uniqueness 1576485880, timestamp 1576494066/8541734
20xx-12-16 19:01:07.435: [ CSSD][3804591872]clssnmvDHBValidateNcopy: node 1, myrac1, has a disk HB, but no network HB, DHB has rcfg 471981092, wrtcnt, 211618805, LATS 7287604, lastSeqNo 211618802, uniqueness 1576485880, timestamp 1576494067/8542524
20xx-12-16 19:01:08.304: [ CSSD][3789973248]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
20xx-12-16 19:01:08.422: [ CSSD][3799754496]clssnmvDHBValidateNcopy: node 1, myrac1, has a disk HB, but no network HB, DHB has rcfg 471981092, wrtcnt, 211618806, LATS 7288584, lastSeqNo 211618803, uniqueness 1576485880, timestamp 1576494067/8542734
20xx-12-16 19:01:08.436: [ CSSD][3804591872]clssnmvDHBValidateNcopy: node 1, myrac1, has a disk HB, but no network HB, DHB has rcfg 471981092, wrtcnt, 211618808, LATS 7288604, lastSeqNo 211618805, uniqueness 1576485880, timestamp 1576494068/8543524
20xx-12-16 19:01:09.304: [ CSSD][3789973248]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
20xx-12-16 19:01:09.422: [ CSSD][3799754496]clssnmvDHBValidateNcopy: node 1, myrac1, has a disk HB, but no network HB, DHB has rcfg 471981092, wrtcnt, 211618809, LATS 7289584, lastSeqNo 211618806, uniqueness 1576485880, timestamp 1576494068/8543744
20xx-12-16 19:01:09.437: [ CSSD][3804591872]clssnmvDHBValidateNcopy: node 1, myrac1, has a disk HB, but no network HB, DHB has rcfg 471981092, wrtcnt, 211618811, LATS 7289604, lastSeqNo 211618808, uniqueness 1576485880, timestamp 1576494069/8544524
20xx-12-16 19:01:09.803: [ CSSD][3785242368]clssnmRcfgMgrThread: Local Join
20xx-12-16 19:01:09.803: [ CSSD][3785242368]clssnmLocalJoinEvent: begin on node(2), waittime 193000
20xx-12-16 19:01:09.803: [ CSSD][3785242368]clssnmLocalJoinEvent: set curtime (7289964) for my node
20xx-12-16 19:01:09.803: [ CSSD][3785242368]clssnmLocalJoinEvent: scanning 32 nodes
20xx-12-16 19:01:09.803: [ CSSD][3785242368]clssnmLocalJoinEvent: Node myrac1, number 1, is in an existing cluster with disk state 3
20xx-12-16 19:01:09.803: [ CSSD][3785242368]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
20xx-12-16 19:01:10.305: [ CSSD][3789973248]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
20xx-12-16 19:01:10.423: [ CSSD][3799754496]clssnmvDHBValidateNcopy: node 1, myrac1, has a disk HB, but no network HB, DHB has rcfg 471981092, wrtcnt, 211618812, LATS 7290584, lastSeqNo 211618809, uniqueness 1576485880, timestamp 1576494069/8544744
20xx-12-16 19:01:10.437: [ CSSD][3804591872]clssnmvDHBValidateNcopy: node 1, myrac1, has a disk HB, but no network HB, DHB has rcfg 471981092, wrtcnt, 211618814, LATS 7290604, lastSeqNo 211618811, uniqueness 1576485880, timestamp 1576494070/8545524
20xx-12-16 19:01:10.794: [ CSSD][3786819328]clssnmSendingThread: sending join msg to all nodes
20xx-12-16 19:01:10.794: [ CSSD][3786819328]clssnmSendingThread: sent 5 join msgs to all nodes

20xx-12-16 20:36:02.919: [ CSSD][2756265728]clssgmUpdateGrpData: grock(CLSN.ONSNETPROC.MASTER), commissioner(-1/0)
20xx-12-16 20:36:02.919: [ CSSD][2756265728]clssgmHandleGrockRcfgUpdate: grock(CLSN.ONSNETPROC.MASTER), updateseq(118), status(0), sendresp(1)
20xx-12-16 20:36:02.920: [ CSSD][2756265728]clssgmTestSetLastGrockUpdate: grock(CLSN.ONSNETPROC.MASTER), updateseq(118) msgseq(119), lastupdt<0x7fbb58031e10>, ignoreseq(0)
20xx-12-16 20:36:02.920: [ CSSD][2756265728]clssgmGrockOpTagProcess: Request to commission member(1) using key(1) for grock(CLSN.ONSNETPROC.MASTER)
20xx-12-16 20:36:02.920: [ CSSD][2756265728]clssgmUpdateGrpData: grock(CLSN.ONSNETPROC.MASTER), commissioner(1/1)
20xx-12-16 20:36:02.920: [ CSSD][2756265728]clssgmHandleGrockRcfgUpdate: grock(CLSN.ONSNETPROC.MASTER), updateseq(119), status(0), sendresp(1)
20xx-12-16 20:36:02.921: [ CSSD][2756265728]clssgmTestSetLastGrockUpdate: grock(CLSN.ONSNETPROC.MASTER), updateseq(119) msgseq(120), lastupdt<0x7fbb5804d490>, ignoreseq(0)
20xx-12-16 20:36:02.921: [ CSSD][2756265728]clssgmUpdateGrpData: grock(CLSN.ONSNETPROC.MASTER), private data(2052), incarn(40)
20xx-12-16 20:36:02.921: [ CSSD][2756265728]clssgmHandleGrockRcfgUpdate: grock(CLSN.ONSNETPROC.MASTER), updateseq(120), status(0), sendresp(1)
20xx-12-16 20:36:02.922: [ CSSD][2756265728]clssgmTestSetLastGrockUpdate: grock(CLSN.ONSNETPROC.MASTER), updateseq(120) msgseq(121), lastupdt<0x7fbb5803dee0>, ignoreseq(0)
20xx-12-16 20:36:02.922: [ CSSD][2756265728]clssgmGrockOpTagProcess: Request to commission member(-1) using key(1) for grock(CLSN.ONSNETPROC.MASTER)
20xx-12-16 20:36:02.922: [ CSSD][2756265728]clssgmUpdateGrpData: grock(CLSN.ONSNETPROC.MASTER), commissioner(-1/0)
20xx-12-16 20:36:02.922: [ CSSD][2756265728]clssgmHandleGrockRcfgUpdate: grock(CLSN.ONSNETPROC.MASTER), updateseq(121), status(0), sendresp(1)
20xx-12-16 20:36:05.064: [ CSSD][2753111808]clssnmSendingThread: sending status msg to all nodes
20xx-12-16 20:36:05.064: [ CSSD][2753111808]clssnmSendingThread: sent 5 status msgs to all nodes
20xx-12-16 20:36:09.065: [ CSSD][2753111808]clssnmSendingThread: sending status msg to all nodes
20xx-12-16 20:36:09.065: [ CSSD][2753111808]clssnmSendingThread: sent 4 status msgs to all nodes
20xx-12-16 20:36:14.066: [ CSSD][2753111808]clssnmSendingThread: sending status msg to all nodes
...

根据日志能判断出bond信息变了吗?我当时没发现也没分析出来,最后同事说改了bond!当时不是说只换根网线重新排下线吗?我说改回去试试,果然如此,重启一切正常了

胡乱重启了下,没起来。。。
[root@myrac2 bin]# ./crsctl query crs activeversion
Oracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage
ORA-15077: could not locate ASM instance serving a required diskgroup

[root@myrac2 bin]# ./ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage
ORA-15077: could not locate ASM instance serving a required diskgroup

[grid@myrac2 ~]$ cd /u01/app/11.2.0/grid/bin/
[grid@myrac2 bin]$ srvctl start nodeapps -n myrac2
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.net1.network is registered
Cannot communicate with crsd
PRCR-1035 : Failed to look up CRS resource myrac2 for ora.cluster_vip.type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

[grid@myrac2 bin]$ srvctl start asm -n myrac2
PRCR-1070 : Failed to check if resource ora.asm is registered
Cannot communicate with crsd

[grid@myrac2 bin]$ srvctl start database -d testdb2
PRCD-1027 : Failed to retrieve database testdb2
PRCR-1115 : Failed to find entities of type resource that match filters ((NAME == ora.testdb2.db) && (TYPE == ora.database.type)) and contain attributes VERSION,ORACLE_HOME,DATABASE_TYPE
Cannot communicate with crsd
[grid@myrac2 bin]$

节点2被修改的bond,明显跟1不一样
[root@myrac2 11.2.0]# service network status
Configured devices:
lo bond0 bond1 em1 em2 em3 em4
Currently active devices:
lo em1 em2 em3 em4 bond0 bond1
[root@myrac2 11.2.0]#

节点1
[root@myrac1 ~]# service network status
Configured devices:
lo bond0 em1 em2 em3 em4 idrac
Currently active devices:
lo em1 em2 em3 bond0

抛开技术行不行先不说,单这件事来说,同事之间的合作有时候更重要。一不小心你就会给别人挖个坑或掉到别人给你挖的坑

换了网线异常了,CRS无法正常启动,clssnmSendingThread: sending status msg to all nodes的更多相关文章

  1. 异常System.Web.HttpException (0x80004005): Server cannot set status after HTTP headers have been sent.

    在用mvc 的AuthorizeAttribute做身份验证,重写HandleUnauthorizedRequest方法,在Application_Error方法里出现异常System.Web.Htt ...

  2. Linux异常关机后,Mysql启动出错ERROR 2002 (HY000)

    Linux异常关机后,Mysql启动或訪问时,出错: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/ ...

  3. AIX下禁止crs随ha启动而启动

    /etc/init.crs enable /etc/init.crs disable 查看目前crs是enable还是disable状态 状态记录在一个文本文件里  /etc/oracle/scls_ ...

  4. 异常-CDH的service无法启动并抛出异常-org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connection refused)

    1 详细异常 org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connectio ...

  5. WPF App.xaml.cs常用模板,包括:异常捕获,App只能启动一次

    App.xaml.cs中的代码每次都差不多,故特地将其整理出来直接复用: using System; using System.Configuration; using System.Diagnost ...

  6. MyEclipse异常关闭导致Tomcat不能启动的问题

    由于MyEclipse的异常关闭从而导致Tomcat并没有关闭,所以再次启动Tomcat当然是无法启动的啦,解决方法:在任务管理器中关闭一个叫javaw.exe的进程,如果你这时已经启动了MyEcli ...

  7. 左右RAC CRS 自己主动启动

    左右CRS自己主动重新启动实验 一.检验ASM [root@rac1 ~]# /etc/init.d/oracleasm status Checking if ASM is loaded: yes C ...

  8. 联想ideapad 310s如何进BIOS,换固态硬盘SSD,配置U盘启动,重装Win10系统

    1. 如何进BIOS 关机情况下,捅一下Novo键,即可进入BIOS 2. 安装固态硬盘 Ideadpad 310S 本身自带的硬盘是5400转的机械硬盘,容量小速度慢.换的新的固态硬盘是SATA接口 ...

  9. 换了XCode版本之后,iOS应用启动时不占满全屏,上下有黑边

    原因是没有Retina4对应的启动图片,解决方法很简单,就是把Retina4对应的图片给补上就只可以了

随机推荐

  1. Selenium+Java(二)Selenium打开IE浏览器

    前言 已在Eclipse中配置完成Selenium的相关配置,不知道如何配置的可参考我的另一篇博文:https://www.cnblogs.com/yogouo/p/11946940.html 打开I ...

  2. IIS部署.net core网站

    1 安装   Windows8.1-KB2999226-x64 2 安装 DotNetCore.1.0.4_1.1.1-WindowsHosting http://download.microsoft ...

  3. 10-kubernetes serveraccount RBAC

    目录 认证安全 serviceAccountName 和 userAccount serviceaccount 创建 使用admin 的SA 测试 URL访问kubernetes资源 APIserve ...

  4. div水平垂直居中的六种方法

    在平时,我们经常会碰到让一个div框针对某个模块上下左右都居中(水平垂直居中),其实针对这种情况,我们有多种方法实现. 方法一: 绝对定位方法:不确定当前div的宽度和高度,采用 transform: ...

  5. Zip压缩工具、tar打包、打包并压缩

    第5周第2次课(4月17日) 课程内容: 6.5 zip压缩工具6.6 tar打包6.7 打包并压缩 6.5 zip压缩工具 Zip压缩工具最大的特点就是可以支持压缩目录,也能够压缩文件,Window ...

  6. 华为OSPF与ACL综合应用实例讲解

    OSPF与ACL综合应用实例讲解 项目案例要求: 1.企业内网运行OSPF路由协议,区域规划如图所示:2.财务和研发所在的区域不受其他区域链路不稳定性影响:3.R1.R2.R3只允许被IT登录管理:4 ...

  7. 5G,仅仅是更快的网速吗?

    前不久参加了华为的Dev Summit 2020开发者大会,听到了关于5G的一些分享,刚好最近对5G有一些自己的思考,在此分享给大家. 什么是5G 在这里我不想列举各种晦涩难懂的术语,简单说来,5G就 ...

  8. node.js调试入门

    1-1 Inspector介绍 使用Inspector调试Node.js的优势 可查看当前上下文的变量 可观察当前函数调用堆栈 不侵入代码 可在暂停状态下执行指定代码 Inspector的构成以及原理 ...

  9. [TimLinux] CSS 计数功能实现递归目录

    内容引用自<css世界>: count-reset 与 counter 为父子关系,兄弟关系会导致序号混乱 调用一次 count-increment 将给序号进行一次报数,调用 count ...

  10. 2019CCPC秦皇岛D题 Decimal

    Decimal Time Limit: 2000/1000 MS (Java/Others)    Memory Limit: 131072/131072 K (Java/Others)Total S ...