Oracle 11g RAC环境下Private IP修改方法及异常处理

Oracle 11g RAC环境下Private IP修改方法及异常处理

Oracle 11g RAC环境下Private IP修改方法及异常处理
一、 修改方法

1. 确认所有节点CRS服务以启动
# olsnodes -s -n –i
host1 1 host1-vip Active
host2 2 host2-vip Active
2. 修改Private IP配置信息

如果之前只有一个私有网卡，则直接删除时会报错，如：PRIF-31: Failed to delete the
specified network interface because it is the last private interface.所以如果只是修改Private IP
而不用更改网卡时，可以参考以下操作命令：
oifcfg getif //查看Private IP配置
oifcfg setif -global en1/10.10.12.0:cluster_interconnect //添加en1新的Private IP
oifcfg delif -global en1/10.10.11.0 //删除en1旧的Private IP
oifcfg getif //查看Private IP配置

如果需要更改Private IP网卡时，可以参考如下命令：
oifcfg getif //查看Private IP配置
oifcfg setif -global en2/10.10.12.0:cluster_interconnect //增加新的en2 上的Private IP
oifcfg delif -global en1 //删除旧的en1上的Private IP
oifcfg getif //查看Private IP配

3. 关闭CRS服务（所有节点都要执行）：
参考命令：
#crsctl stop crs -f

4. 修改私有网络物理网卡IP地址和/etc/hosts文件（所有节点都要执行）：
参考命令：
# smit tcpip-> Minimum Configuration & Startup->en1
#vi /etc/hosts

5. 启动和查看CRS服务
参考命令：
#crsctl start crs
#crs_stat -t

二、异常处理
1. 问题描述
如果使用oifcfg setif命令修改Private IP配置信息时，subnet与实际配置的物理IP不致，
则重启CRS服务时会导致CRS无法启动，如下所示：
oifcfg setif命令配置：
oifcfg setif -global en1/10.10.12.0:cluster_interconnect //subnet为10.10.12.0
物理IP配置：
10.10.11.61/255.255.255.0 //subnet为10.10.11.0

日志错误信息：
Alert*.log
2013-12-08 12:38:41.795
[ohasd(8388676)]CRS-2765:Resource 'ora.crsd' has failed on server 'host1'.
2013-12-08 12:38:43.189
[crsd(11141158)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error
[PROC-44: Error in network address and interface operations Network address and interface
operations error [7]]. Details at (:CRSD00111:) in
/u01/app/grid/product/11.2.0/grid/log/host1/crsd/crsd.log.
2013-12-08 12:38:43.878
[ohasd(8388676)]CRS-2765:Resource 'ora.crsd' has failed on server 'host1'.
2013-12-08 12:38:43.879
[ohasd(8388676)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not
restart.

Crsd.log
2013-12-08 12:38:42.985: [ OCRAPI][1]clsu_get_private_ip_addresses: no ip addresses found.
[ OCRAPI][1]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
2013-12-08 12:38:43.188: [ OCRAPI][1]a_init:13!: Clusterware init unsuccessful : [44]
2013-12-08 12:38:43.188: [ CRSOCR][1] OCR context init failure. Error: PROC-44: Error in
network address and interface oper
ations Network address and interface operations error [7]
2013-12-08 12:38:43.189: [ CRSMAIN][1] Created alert : (:CRSD00111:) : Could not init OCR,
error: PROC-44: Error in network address and interface operations Network address and interface operations error [7]
2013-12-08 12:38:43.189: [ CRSD][1][PANIC] CRSD exiting: Could not init OCR, code: 44
2013-12-08 12:38:43.189: [ CRSD][1] Done.

2. 处理方法
可以使用gpnptool命令处理以上问题。GPNPD（Greater Pittsburgh Nonprofit Partnership）
进程的主要的作用是在集群各个节点中同步GPnP profile文件，在Clusterware中，CSS、GPnP
等服务的启动都需要依赖于GPnP profile文件。而GPnP profile文件是一个xml文件，它存
储 的 位 置 是 在 ： $GRID_HOME/gpnp//profile/peer/profile.xml ，
$GRID_HOME/gpnp/profile/peer/profile.xml（全局备份，最原始的配置信息）。GPnPD进程写
的trace文件会存放在$GRID_HOME/log//gpnpd/gpnpd.log。
有几个重要的信息存储在GPnP profile文件:
? 网络接口和IP地址(公网和私网)
? ASM diskstring和spfile信息
GPnP profile保存的是RAC的配置信息，包括集群名称、网络类型信息（public/private）、
ASM和CSS的存储信息、安全的数字签名，以及ASM实例的SPFILE文件位置。当集群配置
发生变化时，所有节点的该文件会被自动更新。在安装、系统引导或者当使用标准的集群工
具更新期间，这些活动包括：oifcfg 改变网络信息、crsctl 改变css设备、ASM额外的存储
等，会通过gpdpd进程复制GPnP profile到所有的其他节点。
当集群启动的时候，Cluster Synchronization Services (CSS)将扫描所有的ASM disks，它利
用的是GPnP profile文件中的ASM discovery string。如下所示:

egistry.253.790450611"/>
这里可以看到spfile文件是存放在ASM卷组+DATA中的。但是有一个需要注意到事情是
我们启动ASM的时候，需要spfile文件，Oracle从GPnP profile中知道spfile的路径，然后
它就会从底层磁盘中直接读取spfile标识，启动asm实例。
如果GPnP出现问题，可以使用cluvfy comp gpnp组件验证命令检查在集群中所有节点
网格即插即用的完整性：cluvfy comp gpnp [-n node_list] [-verbose]
最后，注意Oracle不支持手动修改profile.xml文件，直接对它的修改可能导致无法修复
的问题，最终导致在所有节点重建Clusterware。

2.1 以排他模式和不启动crsd进程的方式启动crs
# crsctl start crs -excl –nocrs

2.2 备份crs配置信息
#mkdir /u01/gpnp
#gpnptool get -o=/u01/gpnp/profile.xml
# more /u01/gpnp/profile.xml

xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" x
mlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile"
xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:x
si="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.
xsd" ProfileSequence="22" ClusterUId="8c32638bd2727ff8bfba9eec919fb4e4"
ClusterName="host-scan" PALocation="">
rofile>
IP="192.168.101.0" Use="public"/>
etwork id="net3" Adapter="en1" IP="10.10.12.0"
Use="cluster_interconnect"/>
S-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/>
DiscoveryString="" SPFile="+DATA/ho
st-scan/asmparameterfile/registry.253.832710839"/>
xmlns:ds="http://www.w3.org/2000/09/xmldsig#">>

Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
Algorithm="http://www.w3.
org/2000/09/xmldsig#rsa-sha1"/>
Algorithm="http://www.w3.org/2000/09/xmldsig
#enveloped-signature"/>http://www.w3.org/2001/10/xml-exc-c14n#">
www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl
xsi"/>://www.w3.org/2000/09/xmldsig#sha1"/>h2DxPI2dIQbGBji8BAAdimWLkkA=
s:DigestValue>
nfo>O1haZSJ3N6gC7B5blNK/0PeljlfjBoWODejCLCOOsqmb46eJM2TvhIoD7gV
SVF2P0JKzXblsGlqkPpbfgzfasVpS/3nyowy/oMlfXI
KloO445gzImd/lL1FCsvsZpe+GdqtvKgKaSKNJENHF/Ht4w9YCz6Gz6hhXi8u5n3jGU5E=
eValue>

查看CRS的配置信息：
# gpnptool get

Warning: some command line parameters were defaulted. Resulting command line:
/u01/app/grid/product/11.2.0/grid/bin/gpnptool.bin get -o-

xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile"
xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile"
xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd"
ProfileSequence="22" ClusterUId="8c32638bd2727ff8bfba9eec919fb4e4"
ClusterName="host-scan" PALocation="">
HostName="*">
Use="public"/>
Use="cluster_interconnect"/>
id="css" DiscoveryString="+asm" LeaseDuration="400"/>
DiscoveryString=""
SPFile="+DATA/host-scan/asmparameterfile/registry.253.832710839"/>
xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
URI="">
Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl
xsi"/>
Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>h2DxPI2dIQbGBji8BA
AdimWLkkA=O1haZSJ3N
6gC7B5blNK/0PeljlfjBoWODejCLCOOsqmb46eJM2TvhIoD7gVSVF2P0JKzXblsGlqkPpbfgzfasVpS/3n
yowy/oMlfXIKloO445gzImd/lL1FCsvsZpe+GdqtvKgKaSKNJENHF/Ht4w9YCz6Gz6hhXi8u5n3jGU5E=

Success.

备注：备份的profile.xml文件配置信息与‘gpnptool get’显示配置信息完全相同，同时也
与$ORACLE_HOME /gpnp//profiles/peer下的profile.xml文件信息相同。

2.3 修改备份的CRS配置信息
2.3.1 备份配置文件：
# cp /u01/gpnp/profile.xml /u01/gpnp/p.xml

2.3.2 获取当前的序列号（每次修改并写回crs中会有一个序列号作为标识）：
# gpnptool getpval -p=/u01/gpnp/p.xml -prf_sq -o-
22

2.3.3 获取共有网络和私有网络标识id （与实际网卡名称不一致，可以在配置文件中找到）：
# gpnptool getpval -p=/u01/gpnp/p.xml -net -o-
net1 net3

2.3.4 修改配置文件中的序列号（原序列号值加1）和私网的实际网段（subnet）信息：
# gpnptool edit -p=/u01/gpnp/p.xml -o=/u01/gpnp/p.xml -ovr -prf_sq=23 -net3:net_ip=10.10.11.0
Resulting profile written to "/u01/gpnp/p.xml".
Success.
修改网卡名称
#gpnptool edit -p=/home/app/11.2.0.3/grid/gpnp/profiles/peer/profile.xml -o=/home/app/11.2.0.3/grid/gpnp/profiles/peer/profile.xml -ovr -prf_sq=5 net3:net_ada=em3

2.3.5 查看配置文件是否修改成功：
# more /u01/gpnp/p.xml

xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" x
mlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile"
xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:x
si="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.
xsd" ProfileSequence="23" ClusterUId="8c32638bd2727ff8bfba9eec919fb4e4"
ClusterName="host-scan" PALocation="">
rofile>
IP="192.168.101.0" Use="public"/>
etwork id="net3" Adapter="en1" IP="10.10.11.0"
Use="cluster_interconnect"/>
S-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/>
DiscoveryString="" SPFile="+DATA/ho
st-scan/asmparameterfile/registry.253.832710839"/>
xmlns:ds="http://www.w3.org/2000/09/xmldsig#">>

Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
Algorithm="http://www.w3.
org/2000/09/xmldsig#rsa-sha1"/>
Algorithm="http://www.w3.org/2000/09/xmldsig
#enveloped-signature"/>http://www.w3.org/2001/10/xml-exc-c14n#">
www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl
xsi"/>://www.w3.org/2000/09/xmldsig#sha1"/>h2DxPI2dIQbGBji8BAAdimWLkkA=
s:DigestValue>
nfo>O1haZSJ3N6gC7B5blNK/0PeljlfjBoWODejCLCOOsqmb46eJM2TvhIoD7gV
SVF2P0JKzXblsGlqkPpbfgzfasVpS/3nyowy/oMlfXI
KloO445gzImd/lL1FCsvsZpe+GdqtvKgKaSKNJENHF/Ht4w9YCz6Gz6hhXi8u5n3jGU5E=
eValue>
备注：红色标识改动信息。

2.3.6 验证序列号是否修改完成：
# gpnptool getpval -p=/u01/gpnp/p.xml -prf_sq -o-
23

2.3.7 用私钥重新标识配置文件：
# gpnptool sign -p=/u01/gpnp/p.xml -o=/u01/gpnp/p.xml -ovr -w=cw-fs:peer
Resulting profile written to "/u01/gpnp/p.xml".
Success.

2.3.8 查看配置文件改动：
# more /u01/gpnp/p.xml

xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" x
mlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile"
xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:x
si="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.
xsd" ProfileSequence="23" ClusterUId="8c32638bd2727ff8bfba9eec919fb4e4"
ClusterName="host-scan" PALocation="">
rofile>
IP="192.168.101.0" Use="public"/>
etwork id="net3" Adapter="en1" IP="10.10.11.0"
Use="cluster_interconnect"/>
S-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/>
DiscoveryString="" SPFile="+DATA/ho
st-scan/asmparameterfile/registry.253.832710839"/>
xmlns:ds="http://www.w3.org/2000/09/xmldsig#">>

Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
Algorithm="http://www.w3.
org/2000/09/xmldsig#rsa-sha1"/>
Algorithm="http://www.w3.org/2000/09/xmldsig
#enveloped-signature"/>http://www.w3.org/2001/10/xml-exc-c14n#">
www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl
xsi"/>://www.w3.org/2000/09/xmldsig#sha1"/>6p7GJFTV5ZoPtIqkUEudEvYnHB4=
s:DigestValue>
nfo>QNcNKDq1nbXDh1Htp8DYklSba6jzoYnSruNeJdLi6f9TALAQrLsrSAd6dYW
af4V7f2hGCb9qBGvr1pibl4JUeAnN7BBhgWOKyfwYKE
Bx2FdP2RX5tPkLZC+k2X/PO3SpFfIH7NKbvyqmz5xiso2i5C134ZG0RF9s752ZU0e2jVk=
eValue>
备注：红色标识改动信息。

2.3.9 回写配置文件信息到crs中：
# gpnptool put -p=/u01/gpnp/p.xml

2.3.10 查看crs中配置信息：
# gpnptool find -c=host-scan //host-scan为RAC的scan name;
Found 1 instances of service 'gpnp'.
mdns:service:gpnp._tcp.local.://host1:30391/agent=gpnpd,cname=host-scan,host=host1,pid=12
648872/gpnpd h:host1 c:host-scan

# gpnptool rget -h=host1 //host1为主机名
Warning: some command line parameters were defaulted. Resulting command line:
/u01/app/grid/product/11.2.0/grid/bin/gpnptool.bin rget -h=host1 -o-
Found 1 gpnp service instance(s) to rget profile from.
RGET from tcp://host1:30391
(mdns:service:gpnp._tcp.local.://host1:30391/agent=gpnpd,cname=host-scan,host=host1,pid=12
648872/gpnpd h:host1 c:host-scan):

xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile"
xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile"
xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd"
ProfileSequence="23" ClusterUId="8c32638bd2727ff8bfba9eec919fb4e4"
ClusterName="host-scan" PALocation="">
HostName="*">
Use="public"/>
Use="cluster_interconnect"/>
id="css" DiscoveryString="+asm" LeaseDuration="400"/>
DiscoveryString=""
SPFile="+DATA/host-scan/asmparameterfile/registry.253.832710839"/>
xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
URI="">
Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl
xsi"/>
Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>6p7GJFTV5ZoPtIqkUE
udEvYnHB4=QNcNKDq1n
bXDh1Htp8DYklSba6jzoYnSruNeJdLi6f9TALAQrLsrSAd6dYWaf4V7f2hGCb9qBGvr1pibl4JUeAnN7B
BhgWOKyfwYKEBx2FdP2RX5tPkLZC+k2X/PO3SpFfIH7NKbvyqmz5xiso2i5C134ZG0RF9s752ZU0e2j
Vk=
Success.
#

2.3.11 启动crsd进程：
# crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'host1'
CRS-2676: Start of 'ora.crsd' on 'host1' succeeded
# crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

2.3.12 查看和修改私有网络配置：
# oifcfg getif
en0 192.168.101.0 global public
en1 10.10.11.0 global cluster_interconnect
如不正确使用下面命令重新修改：
# oifcfg setif -global en1/10.10.11.0:cluster_interconnect
# oifcfg getif
en0 192.168.101.0 global public
en1 10.10.11.0 global cluster_interconnect

2.3.13 重启crs服务：
# crsctl stop crs -f
# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
# crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE host1
ora....ER.lsnr ora....er.type ONLINE ONLINE host1
ora....N1.lsnr ora....er.type ONLINE ONLINE host1
ora.asm ora.asm.type ONLINE ONLINE host1
ora.cvu ora.cvu.type ONLINE ONLINE host1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE host1
ora....T1.lsnr application ONLINE ONLINE host1
ora.host1.gsd application OFFLINE OFFLINE
ora.host1.ons application ONLINE ONLINE host1
ora.host1.vip ora....t1.type ONLINE ONLINE host1
ora.host2.vip ora....t1.type ONLINE ONLINE host1
ora....network ora....rk.type ONLINE ONLINE host1
ora.oc4j ora.oc4j.type ONLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE host1
ora....ry.acfs ora....fs.type ONLINE ONLINE host1
ora.scan1.vip ora....ip.type ONLINE ONLINE host1
ora.yxdb.db ora....se.type ONLINE ONLINE host1
#
# oifcfg getif
en0 192.168.101.0 global public
en1 10.10.11.0 global cluster_interconnect
2.4 使用ocr自动备份不能恢复以上问题
以下为使用ocrconfig –restore方法进行恢复过程记录：
# ocrconfig -showbackupp
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy

host1 2013/12/08 10:00:35
/u01/app/grid/product/11.2.0/grid/cdata/host-scan/backup00.ocr

host1 2013/12/08 06:00:34
/u01/app/grid/product/11.2.0/grid/cdata/host-scan/backup01.ocr

host1 2013/12/05 15:59:07
/u01/app/grid/product/11.2.0/grid/cdata/host-scan/backup02.ocr

host1 2013/12/08 06:00:34
/u01/app/grid/product/11.2.0/grid/cdata/host-scan/day.ocr

host1 2013/11/29 00:51:41
/u01/app/grid/product/11.2.0/grid/cdata/host-scan/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available
# ocrconfig -restore /u01/app/grid/product/11.2.0/grid/cdata/host-scan/backup00.ocr
PROT-35: The configured Oracle Cluster Registry locations are not accessible
# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
# ocrconfig -restore /u01/app/grid/product/11.2.0/grid/cdata/host-scan/backup00.ocr
# crsctl stop crs -f
# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
# crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
# crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.