rac的一次问题 ORA-01565: error in identifying file '+DATA/bol/spfilebol.ora'

昨天安装的测试环境的rac--2节点

CentOS release 6.8 (Final)

SQL*Plus: Release 11.2.0.4.0 Production

今天测试突然出现问题

在rac01节点上面操作报错

Connected.
SQL> select * from v$instance;
select * from v$instance
*
ERROR at line 1:
ORA-01012: not logged on
Process ID: 0
Session ID: 0 Serial number: 0

这个时候创建表空间，用户均报错

用toad工具连接vip 209，进行创建表空间，用户均可以成功，于是怀疑rac01节点有问题

果然

执行就报错

检查节点1的权限和路径

[root@rac11 ~]# chown -R grid:oinstall /u01/app/11.2.0/grid/
[root@rac11 ~]# ll /u01/app/11.2.0/grid/

这时候asmcmd可以执行

然后登陆sqlplus

[oracle@rac11 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Jun 5 15:48:44 2019

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected.
SQL> select * from v$log;
select * from v$log
*
ERROR at line 1:
ORA-01012: not logged on
Process ID: 0
Session ID: 0 Serial number: 0

查看错误日志

[oracle@rac11 trace]$ tail -n 100 /u01/app/oracle/diag/rdbms/bol/bol1/trace/alert_bol1.log
Fatal NI connect error 12547, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1_o000_bol1)(ENVS='ORACLE_HOME=/u01/app/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser)(CONNECT_DATA=(CID=(PROGRAM=oracle@rac11)(HOST=rac11)(USER=oracle))))

  VERSION INFORMATION:
    TNS for Linux: Version 11.2.0.4.0 - Production
    Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
  Time: 05-JUN-2019 15:51:55
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12547

TNS-12547: TNS:lost contact
    ns secondary err code: 12560
    nt main err code: 517

TNS-00517: Lost contact
    nt secondary err code: 32
    nt OS err code: 0
ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1_o000_bol1)(ENVS='ORACLE_HOME=/u01/app/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
WARNING: ASM communication error: op 18 state 0x40 (12547)
ERROR: slave communication error with ASM
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_pmon_840.trc:
ORA-12547: TNS:lost contact

[oracle@rac11 trace]$ more /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_pmon_840.trc
WARNING: ASM communication error: op 18 state 0x40 (1034)
ERROR: slave communication error with ASM
NOTE: Deferred communication with ASM instance
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 26
Additional information: 360454
WARNING: ASM communication error: op 18 state 0x40 (1034)
ERROR: slave communication error with ASM
NOTE: Deferred communication with ASM instance
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 26
Additional information: 360454

查看db状态

[grid@rac11 ~]$ srvctl status asm -a
ASM is running on rac11,rac12
ASM is enabled.
[grid@rac11 ~]$ srvctl status database -d bol
Instance bol1 is not running on node rac11
Instance bol2 is running on node rac12

节点1没有启动

尝试启动报错

[grid@rac11 ~]$ srvctl start instance -d bol -i bol1
PRCR-1013 : Failed to start resource ora.bol.db
PRCR-1064 : Failed to start resource ora.bol.db on node rac11
CRS-5017: The resource action "ora.bol.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/bol/spfilebol.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/bol/spfilebol.ora
ORA-12547: TNS:lost contact
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac11/agent/crsd/oraagent_oracle//oraagent_oracle.log".

CRS-2674: Start of 'ora.bol.db' on 'rac11' failed

没遇到这样的问题，一阵baidu，google骚操作，尝试执行

[grid@rac11 ~]$ ls -l /u01/app/11.2.0/grid/bin/oracle
-rwxr-x--x 1 grid oinstall 209914479 Jun  4 15:32 /u01/app/11.2.0/grid/bin/oracle
[grid@rac11 ~]$ chmod 6751 /u01/app/11.2.0/grid/bin/oracle
[grid@rac11 ~]$ ls -l /u01/app/11.2.0/grid/bin/oracle
-rwsr-s--x 1 grid oinstall 209914479 Jun  4 15:32 /u01/app/11.2.0/grid/bin/oracle

[oracle@rac11 ~]$ ls -l /u01/app/oracle/product/11.2.0/db_1/bin/oracle
-rwxr-s--x 1 oracle asmadmin 239626641 Jun  4 16:00 /u01/app/oracle/product/11.2.0/db_1/bin/oracle

---rac02
[grid@rac12 ~]$ ls -l /u01/app/11.2.0/grid/bin/oracle
-rwsr-s--x 1 grid oinstall 209914479 Jun  4 15:33 /u01/app/11.2.0/grid/bin/oracle
[oracle@rac12 ~]$ ls -l /u01/app/oracle/product/11.2.0/db_1/bin/oracle
-rwsr-s--x 1 oracle asmadmin 239626641 Jun  4 16:03 /u01/app/oracle/product/11.2.0/db_1/bin/oracle

--rac01
[grid@rac11 ~]$ cd $ORACLE_HOME/bin/
[grid@rac11 bin]$ ll setasmgid
-rwxr-x--x 1 grid oinstall 2279833 Jun  4 15:32 setasmgid
[grid@rac11 bin]$ set setasmgid setasmgidwrap setfattr setkeycodes setmetamode setsid setup setarch setasmgid0 setfacl setfont setleds setserial setterm setxkbmap
[grid@rac11 bin]$ setasmgidwrap o=/u01/app/oracle/product/11.2.0/db_1/bin/oracle
[grid@rac11 bin]$ ls -l /u01/app/oracle/product/11.2.0/db_1/bin/oracle
-rwxr-s--x 1 oracle asmadmin 239626641 Jun  4 16:00 /u01/app/oracle/product/11.2.0/db_1/bin/oracle
[启动实例bol1][grid@rac11 bin]$ srvctl status database -d bol
Instance bol1 is running on node rac11
Instance bol2 is running on node rac12

节点01可以启动

--很费劲为什么会出现这样的问题，看了一下history，01,02节点的命令都是一样的，执行的时候也没出什么错误

~~~~~