重新配置ocr voting

由于存储空间不足，下线的数据库需要把存储空间腾出来，关闭集群资源，主机工程师收回lun需要（包括ocr 和 voting data 磁盘组），新的应用需要上线需要新的数据库，新的hitach存储到位需要重新安装数据库，上次删除gi 和database软件都在，这次只需要重新配置即可。参考How to Deconfigure/Reconfigure(Rebuild OCR) or Deinstall Grid Infrastructure (文档 ID 1377349.1)

last node  删除配置
[root@qdcx-db2 /]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force -verbose
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.crf' on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'qdcx-db2'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'qdcx-db2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'qdcx-db2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'qdcx-db2' succeeded
CRS-2677: Stop of 'ora.crf' on 'qdcx-db2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'qdcx-db2'
CRS-2677: Stop of 'ora.gipcd' on 'qdcx-db2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'qdcx-db2'
CRS-2677: Stop of 'ora.gpnpd' on 'qdcx-db2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'qdcx-db2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'qdcx-db2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
Successfully deconfigured Oracle clusterware stack on this node

first node
[root@qdcx-db1 /]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force -verbose   -lastnode
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
CRS-5702: Resource 'ora.cssd' is already running on 'qdcx-db1'
CRS-4000: Command Start failed, or completed with errors.
CSS startup failed with return code 1
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-2673: Attempting to stop 'ora.ctssd' on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.asm' on 'qdcx-db1'
CRS-2677: Stop of 'ora.ctssd' on 'qdcx-db1' succeeded
CRS-2677: Stop of 'ora.asm' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.cssd' on 'qdcx-db1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'qdcx-db1'
CRS-2676: Start of 'ora.cssdmonitor' on 'qdcx-db1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'qdcx-db1'
CRS-2672: Attempting to start 'ora.diskmon' on 'qdcx-db1'
CRS-2676: Start of 'ora.diskmon' on 'qdcx-db1' succeeded
CRS-2676: Start of 'ora.cssd' on 'qdcx-db1' succeeded
This ASM diskgroup does not contain voting disks to be deleted
ASM de-configuration trace file location: /tmp/asmcadc_clean2018-07-16_04-34-19-PM.log
ASM Clean Configuration START
ASM Clean Configuration END

ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2018-07-16_04-34-19-PM.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'qdcx-db1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'qdcx-db1'
CRS-2677: Stop of 'ora.ctssd' on 'qdcx-db1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.mdnsd' on 'qdcx-db1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.gipcd' on 'qdcx-db1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'qdcx-db1'
CRS-2677: Stop of 'ora.gpnpd' on 'qdcx-db1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'qdcx-db1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
Successfully deconfigured Oracle clusterware stack on this node

重新配置
节点1
[root@qdcx-db1 /]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
This may take several minutes. Please wait ...
/oracle/app/crs/install/usm/V7x/powerpc/bin//udefacfsctl[75]: /oracle/app/crs/install/usm/V7x/powerpc/bin//udefacfsctl.bin: 0403-006 Execute permission denied.
ACFS-9361: Removing device 'acfsctl' failed with error code '32256'.
Unable to uninstall ADVM/ACFS

解决
chmod -R  755 /oracle/app/crs/install/usm/V7x/powerpc/bin

/oracle/app/crs/crs/install/rootcrs.pl -deconfig -force
/oracle/app/crs/root.sh

节点2
[root@qdcx-db2 /etc/oracle]# /oracle/app/crs/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
****Unable to retrieve Oracle Clusterware home.
Start Oracle Clusterware stack and try again.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CLSU-00100: Operating System function: opendir failed with error data: 2
CLSU-00101: Operating System error message: No such file or directory
CLSU-00103: error location: scrsearch1
CLSU-00104: additional error information: cant open scr home dir scls_scr_getval
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
This may take several minutes. Please wait ...
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Successfully deconfigured Oracle clusterware stack on this node

[root@qdcx-db2 /etc/oracle]# /oracle/app/crs/root.sh
Performing root user operation for Oracle 11g 

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /oracle/app/crs

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/app/crs/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
User grid has the required capabilities to run CSSD in realtime mode
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'qdcx-db2'
CRS-2676: Start of 'ora.mdnsd' on 'qdcx-db2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'qdcx-db2'
CRS-2676: Start of 'ora.gpnpd' on 'qdcx-db2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'qdcx-db2'
CRS-2672: Attempting to start 'ora.gipcd' on 'qdcx-db2'
CRS-2676: Start of 'ora.cssdmonitor' on 'qdcx-db2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'qdcx-db2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'qdcx-db2'
CRS-2672: Attempting to start 'ora.diskmon' on 'qdcx-db2'
CRS-2676: Start of 'ora.diskmon' on 'qdcx-db2' succeeded
CRS-2676: Start of 'ora.cssd' on 'qdcx-db2' succeeded

Disk Group ocr creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/rhdisk6' matches no disks
ORA-15025: could not open disk "/dev/rhdisk6"
ORA-27041: unable to open file

Configuration of ASM ... failed
see asmca logs at /oracle/app/grid/cfgtoollogs/asmca for details
Did not succssfully configure and start ASM at /oracle/app/crs/crs/install/crsconfig_lib.pm line 6912.
/oracle/app/crs/perl/bin/perl -I/oracle/app/crs/perl/lib -I/oracle/app/crs/crs/install /oracle/app/crs/crs/install/rootcrs.pl execution failed

查看hdisk6 是否同一个lun
[root@qdcx-db1 /]# lscfg -vpl hdisk6
  hdisk6           U5802.001.RCH5723-P1-C2-T1-W50060E8016609092-L4000000000000  Hitachi Disk Array (Fibre)

        Manufacturer................HITACHI
        Machine Type and Model......OPEN-V
        Part Number.................. .....
        ROS Level and ID............37303036
        Serial Number...............50 16090
        EC Level....................
        FRU Number..................
        Device Specific.(Z0)........00000332EF000002
        Device Specific.(Z1)........0B37 AC ....
        Device Specific.(Z2)............
        Device Specific.(Z3).........
        Device Specific.(Z4)............
        Device Specific.(Z5)........
        Device Specific.(Z6)........

  PLATFORM SPECIFIC

  Name:  disk
    Node:  disk
    Device Type:  block

[root@qdcx-db2 /]# lscfg -vpl hdisk6
  hdisk6           U5802.001.RCH4330-P1-C2-T1-W50060E8016609092-L4000000000000  Hitachi Disk Array (Fibre)

        Manufacturer................HITACHI
        Machine Type and Model......OPEN-V
        Part Number.................. .....
        ROS Level and ID............37303036
        Serial Number...............50 16090
        EC Level....................
        FRU Number..................
        Device Specific.(Z0)........00000332EF000002
        Device Specific.(Z1)........0B37 AC ....
        Device Specific.(Z2)............
        Device Specific.(Z3).........
        Device Specific.(Z4)............
        Device Specific.(Z5)........
        Device Specific.(Z6)........

  PLATFORM SPECIFIC

  Name:  disk
    Node:  disk
    Device Type:  block

从Device Specific.(Z1)........0B37 AC可知是同一个lun
查看hdisk6磁盘属性
[root@qdcx-db2 /]# ls -l /dev/rhdisk6
crw-rw----    1 grid     asmadmin     15,  5 Jul 16 14:39 /dev/rhdisk6
[+ASM2]@qdcx-db2[/home/grid]$lsattr -E -l hdisk6|grep reserve_policy
reserve_policy PR_exclusive          Reserve Policy                   True
磁盘属性reserve_policy不对，系统管理员忘记修改，运行脚本修改
[root@qdcx-db2 /tmp]# cat chmodall
begin=18
end=22

i=$begin
while [ $i -le $end ];
do
        chmod 660 /dev/rhdisk$i
        chown grid:asmadmin /dev/rhdisk$i
        chdev -l hdisk$i -a reserve_policy=no_reserve
        chdev -l hdisk$i -a queue_depth=8
        i=`expr $i + 1`
done

ls -l /dev/rhdisk*
继续运行root.sh 这次没报错