HA切换失败原因分析

1. 问题描述

redhat在进行HA切换时，需要先停止service，并释放调当前主机占有的资源，比如说IP Address和Filesystem，但今天我在验证HA切换时，发现service一直停止失败，导致HA切换失败，具体报错信息如下

Jun 6 11:10:50 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: Stopping service service:service_bill

Jun 6 11:10:50 SHQZ-PS-IOT-SV2-BILL01 rgmanager[7526]: [ip] Removing IPv4 address 172.17.131.222/26 from eth0

Jun 6 11:10:51 SHQZ-PS-IOT-SV2-BILL01 ntpd[2152]: Deleting interface #7 eth0, 172.17.131.222#123, interface stats: received=0, sent=0, dropped=0, active_time=1542 secs

Jun 6 11:11:01 SHQZ-PS-IOT-SV2-BILL01 rgmanager[7670]: [fs] unmounting /billdata

Jun 6 11:11:06 SHQZ-PS-IOT-SV2-BILL01 rgmanager[7804]: [fs] unmounting /billdata

Jun 6 11:11:11 SHQZ-PS-IOT-SV2-BILL01 rgmanager[7938]: [fs] unmounting /billdata

Jun 6 11:11:11 SHQZ-PS-IOT-SV2-BILL01 rgmanager[7983]: [fs] 'umount /billdata' failed, error=1

Jun 6 11:11:11 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: stop on fs "billdata" returned 1 (generic error)

Jun 6 11:11:12 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: #12: RG service:service_bill failed to stop; intervention required

Jun 6 11:11:12 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: Service service:service_bill is failed

我这里只配置了IP Address和Filesystem两个资源，从上面的日志信息可以看到，首先删除IP Address，这个是没有问题的，接下来释放Filesystem资源，这时候就出问题了，rgmanage服务尝试unmounting /billdata 三次都失败了，导致service停止失败，HA切换也就失败了。

2. 解决方法

登陆到HA管理界面，点击"Resource",选择我配置的Filesystem资源

注意：勾选Force Unmount选项，然后点击"Apply"。

接下来，再次测试HA切换，现在资源都在SHQZ-PS-IOT-SV2-BILL01机器上，执行命令clusvcadm -r service_bill -m SHQZ-PS-IOT-SV2-BILL02,将服务切换到SHQZ-PS-IOT-SV2-BILL01上

# clustat查看集群状态

[root@SHQZ-PS-IOT-SV2-BILL01 cluster]# clustat

Cluster Status for cl_bill @ Wed Jun  6 14:13:42 2018

Member Status: Quorate

 Member Name                                 ID   Status

 ------ ----                                 ---- ------

 SHQZ-PS-IOT-SV2-BILL02                      2 Online, rgmanager

 SHQZ-PS-IOT-SV2-BILL01                      3 Online, Local, rgmanager

 Service Name                                Owner (Last)                               State

 ------- ----                                ----- ------                               -----

 service:service_bill                        SHQZ-PS-IOT-SV2-BILL01                     started

# 切换服务到SHQZ-PS-IOT-SV2-BILL01上

[root@SHQZ-PS-IOT-SV2-BILL01 cluster]# clusvcadm -r service_bill -m SHQZ-PS-IOT-SV2-BILL02

Trying to relocate service:service_bill to SHQZ-PS-IOT-SV2-BILL02...Success

service:service_bill is now running on SHQZ-PS-IOT-SV2-BILL02

这次，我们总算是成功了，登陆SHQZ-PS-IOT-SV2-BILL02查看IP Address和Filesystem资源是否存在

[root@SHQZ-PS-IOT-SV2-BILL02 cluster]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000

    link/ether 00:50:56:9e:34:c1 brd ff:ff:ff:ff:ff:ff

    inet 172.17.131.224/26 brd 172.17.131.255 scope global eth0

    inet 172.17.131.222/26 scope global secondary eth0

    inet6 fe80::250:56ff:fe9e:34c1/64 scope link

       valid_lft forever preferred_lft forever

[root@SHQZ-PS-IOT-SV2-BILL02 cluster]# df -h /billdata

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg_bill-lvbilldata

                      985G   97G  838G  11% /billdata

可以看到资源都在SHQZ-PS-IOT-SV2-BILL02上了

3. 原因分析

勾选Force Unmount选项后，HA切换时，SHQZ-PS-IOT-SV2-BILL01日志如下：

Jun 6 12:41:05 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: Stopping service service:service_bill

Jun 6 12:41:05 SHQZ-PS-IOT-SV2-BILL01 rgmanager[5791]: [ip] Removing IPv4 address 172.17.131.222/26 from eth0

Jun 6 12:41:06 SHQZ-PS-IOT-SV2-BILL01 ntpd[2152]: Deleting interface #9 eth0, 172.17.131.222#123, interface stats: received=0, sent=0, dropped=0, active_time=1516 secs

Jun 6 12:41:15 SHQZ-PS-IOT-SV2-BILL01 rgmanager[5852]: [fs] unmounting /billdata

Jun 6 12:41:15 SHQZ-PS-IOT-SV2-BILL01 rgmanager[5918]: [fs] Sending SIGTERM to processes on /billdata

Jun 6 12:41:21 SHQZ-PS-IOT-SV2-BILL01 rgmanager[5953]: [fs] unmounting /billdata

Jun 6 12:41:24 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: Service service:service_bill is stopped

SHQZ-PS-IOT-SV2-BILL02日志如下：

Jun 6 12:41:24 SHQZ-PS-IOT-SV2-BILL02 rgmanager[2368]: Starting stopped service service:service_bill

Jun 6 12:41:25 SHQZ-PS-IOT-SV2-BILL02 rgmanager[12296]: [fs] mounting /dev/dm-2 on /billdata

Jun 6 12:41:25 SHQZ-PS-IOT-SV2-BILL02 rgmanager[12318]: [fs] mount -t ext4 /dev/dm-2 /billdata

Jun 6 12:41:25 SHQZ-PS-IOT-SV2-BILL02 kernel: EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts:

Jun 6 12:41:25 SHQZ-PS-IOT-SV2-BILL02 rgmanager[12404]: [ip] Adding IPv4 address 172.17.131.222/26 to eth0

Jun 6 12:41:29 SHQZ-PS-IOT-SV2-BILL02 rgmanager[2368]: Service service:service_bill started

Jun 6 12:41:29 SHQZ-PS-IOT-SV2-BILL02 ntpd[2123]: Listening on interface #6 eth0, 172.17.131.222#123 Enabled

通过分析上面的日志，我猜测原因可能是这样的：

如果/billdata这个目录一直有进程在访问（我的/billdata目录确实一直被进程访问），在勾选Force Unmount选项前，rgmanager服务在umount /billdata时，因为有进程在访问该目录，导致umount失败。

勾选Force Unmount后，rgmanger在umount /billdata时，如果有进程在占用/billdata目录，会发出SIGTERM信号给该进程，解除其对/billdata的占用,然后再umount，最后stop service。同时SHQZ-PS-IOT-SV2-BILL02上监测到service停止，就会重新启动service，接管IP Address和Filesystem资源