测试：Oracle 19c RAC添加私网

最近有个客户需求是在某12.2版本的RAC环境上添加心跳网络，顺便考虑将之前的心跳网络改为asm专用。我目前只有19c的RAC的测试环境（19c是12c的最终稳定版本），直接测试验证下过程备忘。

• 1.测试环境概述
• 2.添加心跳网络
• 3.修改asm网络
• 4.重启has集群
• 5.检查心跳状态

1.测试环境概述

首先我的测试环境是这样的：

[grid@db195 ~]$ oifcfg getif
enp0s3  192.168.1.0  global  public
enp0s8  10.10.1.0  global  cluster_interconnect,asm

2.添加心跳网络

主机层面添加新的网卡enp0s9，配置对应的IP地址后，可以将enp0s9添加为心跳网络：

[grid@db195 ~]$ oifcfg setif -global enp0s9/10.10.1.0:cluster_interconnect
[grid@db195 ~]$ oifcfg getif
enp0s3  192.168.1.0  global  public
enp0s8  10.10.1.0  global  cluster_interconnect,asm
enp0s9  10.10.1.0  global  cluster_interconnect

3.修改asm网络

如果确认需要将之前的设置为asm专用，可以再修改：

[grid@db195 ~]$ oifcfg setif -global enp0s8/10.10.1.0:asm
[grid@db195 ~]$ oifcfg getif
enp0s3  192.168.1.0  global  public
enp0s8  10.10.1.0  global  asm
enp0s9  10.10.1.0  global  cluster_interconnect

此时查看RAC其他节点的配置确保一样。

4.重启has集群

使用ifconfig -a查看新增网卡enp0s9的信息，还是没有被HAIP接管的（没有对应169网段的地址信息）

enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.196  netmask 255.255.255.0  broadcast 10.10.1.255
        inet6 fe80::2549:d14d:7dfd:61c4  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:59:1a:a6  txqueuelen 1000  (Ethernet)
        RX packets 66  bytes 19094 (18.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1170 (1.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

需要使用crsctl stop has关闭各节点集群：

[root@db193 ~]# /u01/app/19.3.0/grid/bin/crsctl stop has
[root@db195 ~]# /u01/app/19.3.0/grid/bin/crsctl stop has

[root@db193 ~]# /u01/app/19.3.0/grid/bin/crsctl start has
[root@db195 ~]# /u01/app/19.3.0/grid/bin/crsctl start has

5.检查心跳状态

重启后，再次观察ifconfig -a看到新增网卡enp0s9有对应的HAIP地址了，但是enp0s8没有了，因为enp0s8只是用的asm：

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.195  netmask 255.255.255.0  broadcast 10.10.1.255
        inet6 fe80::5a1e:e7de:fe1a:a286  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:1d:85:d4  txqueuelen 1000  (Ethernet)
        RX packets 38565  bytes 27624748 (26.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 69723  bytes 72018893 (68.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.196  netmask 255.255.255.0  broadcast 10.10.1.255
        inet6 fe80::2549:d14d:7dfd:61c4  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:59:1a:a6  txqueuelen 1000  (Ethernet)
        RX packets 12510  bytes 7500503 (7.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14723  bytes 17562072 (16.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s9:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.3.55  netmask 255.255.224.0  broadcast 169.254.31.255
        ether 08:00:27:59:1a:a6  txqueuelen 1000  (Ethernet)

如果之前的enp0s8网卡没有修改为asm专用，也就是asm,cluster_interconnect的话，那么enp0s8网卡就会保持有HAIP地址：

--设置enp0s8为asm,cluster_interconnect
[grid@db195 ~]$ oifcfg setif -global enp0s8/10.10.1.0:asm,cluster_interconnect

--重启集群后，ifconfig -a就会发现enp0s8网卡保持有HAIP地址：
enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.195  netmask 255.255.255.0  broadcast 10.10.1.255
        inet6 fe80::5a1e:e7de:fe1a:a286  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:1d:85:d4  txqueuelen 1000  (Ethernet)
        RX packets 41817  bytes 28559675 (27.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 89166  bytes 88323795 (84.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.7.148  netmask 255.255.240.0  broadcast 169.254.15.255
        ether 08:00:27:1d:85:d4  txqueuelen 1000  (Ethernet)

enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.196  netmask 255.255.255.0  broadcast 10.10.1.255
        inet6 fe80::2549:d14d:7dfd:61c4  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:59:1a:a6  txqueuelen 1000  (Ethernet)
        RX packets 103509  bytes 77082525 (73.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 237436  bytes 290562629 (277.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s9:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.22.254  netmask 255.255.240.0  broadcast 169.254.31.255
        ether 08:00:27:59:1a:a6  txqueuelen 1000  (Ethernet)

注意：如果新增enp0s9心跳网卡和之前心跳网段不一样，比如10.10.2.195，还要注意和oifcfg setif enp0s9/10.10.2.0:cluster_interconnect，网段与新网卡的网路保持一致，相互匹配。

总结下这个事情就是先确认物理增加心跳网卡和心跳线，然后数据库层面直接使用oifcfg可增加或修改，需要重启has集群生效。