KingbaseES V8R6集群运维案例之---主备failover切换原因分析

案例说明：

生产环境，KingbaseES V8R6的集群发生failover切换，分析集群切换的原因。

适用版本：

KingbaseES V8R6

集群架构：

137.xx.xx.67主  原备库
137.xx.xx.94    原主库
137.xx.xx.68    vip地址

一、日志分析

1、分析原备库hamgr.log

如下所示，通过原备库hamgr.log日志获取到具体的故障时间：

2023-10-30 06:54:29] [WARNING] unable to ping "user=esrep dbname=esrep port=15555 host=137.xx.xx.94 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3"
[2023-10-30 06:54:29] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2023-10-30 06:54:29] [WARNING] unable to connect to upstream node "node1" (ID: 1)
[2023-10-30 06:54:29] [INFO] sleeping 6 seconds until next reconnection attempt
[2023-10-30 06:54:35] [INFO] checking state of node 1, 1 of 10 attempts
[2023-10-30 06:54:45] [WARNING] unable to ping "user=esrep connect_timeout=10 dbname=esrep host=137.xx..94 port=15555 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr"
[2023-10-30 06:54:45] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2023-10-30 06:54:45] [INFO] sleeping 6 seconds until next reconnection attempt
[2023-10-30 06:54:51] [INFO] checking state of node 1, 2 of 10 attempts
.......

2、查看原主库sys_log日志

如下所示，原主库缺失了“2023-10-30 06:53:32” 至 “2023-10-30 08:40:38”的日志，至少可以判断在这段时间数据库服务没有被启动。

2023-10-30 06:53:32 CST[3063476]:[1-1] user=cocall,db=cocall,app=Kingbase8 JDBC Driver,client=137.17.17.90LOG:  duration: 166.938 ms  execute <unnamed>: SELECT distinct cd.remoteId, cd.remoteDid from ccUserDomain cud inner join ccDomain cd on cud.did = cd.did inner join ccUser cu on cud."uid" = cu."uid" where cd.remoteDid is not null and cu.remoteUid is null 

----------缺失日志-------
2023-10-30 08:40:38 CST[96015]:[9-1] user=,db=,app=,client=WARNING:  empty user key file.
2023-10-30 08:40:38 CST[96068]:[1-1] user=,db=,app=,client=LOG:  database system was interrupted; last known up at 2023-10-30 06:30:05 CST
2023-10-30 08:40:39 CST[96071]:[1-1] user=esrep,db=esrep,app=[unknown],client=137.17.17.94FATAL:  the database system is starting up
2023-10-30 08:40:39 CST[96072]:[1-1] user=esrep,db=esrep,app=[unknown],client=137.17.17.94FATAL:  the database system is starting up
2023-10-30 08:40:39 CST[96068]:[2-1] user=,db=,app=,client=LOG:  database system was not properly shut down; automatic recovery in progress
2023-10-30 08:40:39 CST[96068]:[3-1] user=,db=,app=,client=LOG:  redo starts at 9/62075558
2023-10-30 08:40:39 CST[96068]:[4-1] user=,db=,app=,client=LOG:  redo wal segment count 29
2023-10-30 08:40:39 CST[96068]:[5-1] user=,db=,app=,client=LOG:  invalid record length at 9/620816D8: wanted 24, got 0
2023-10-30 08:40:39 CST[96068]:[6-1] user=,db=,app=,client=LOG:  redo done at 9/620816A8
2023-10-30 08:40:39 CST[96068]:[7-1] user=,db=,app=,client=LOG:  checkpoint starting: end-of-recovery immediate
2023-10-30 08:40:39 CST[96068]:[8-1] user=,db=,app=,client=LOG:  checkpoint complete: wrote 16 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.472 s, sync=0.003 s, total=0.499 s; sync files=15, longest=0.002 s, average=0.000 s; distance=48 kB, estimate=48 kB
2023-10-30 08:40:39 CST[96015]:[10-1] user=,db=,app=,client=LOG:  database system is ready to accept connections

3、查看原主库kbha.log日志

通过原主库kbha.log日志，没有发现磁盘读写异常、网关连接失败的故障；但是也缺失了“2023-10-30 06:53:39” 至 “2023-10-30 06:57:03”的日志，这段时间主库主机应该出现异常。

[2023-10-30 06:53:39] [NOTICE] PING 137.17.17.65 (137.17.17.65) 56(84) bytes of data.

--- 137.17.17.65 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 0.191/0.214/0.238/0.023 ms

---------- 缺失日志--------------

[2023-10-30 06:57:03] [NOTICE] the kbha is starting ...
[2023-10-30 06:57:03] [NOTICE] [thread 227340768] the values of 'mount_point_dir_list' is NULL, set it as 'data_directory'
[2023-10-30 06:57:05] [NOTICE] PING 137.17.17.65 (137.17.17.65) 56(84) bytes of data.

4、分析主库节点系统message日志

如下所示，在“Oct 30 06:54:07” 至 “Oct 30 06:56:11" ，message缺失一分钟的日志，判断系统出现系统hang住的异常。并且在”“Oct 30 06:56:11"系统出现重启的日志。

Oct 30 06:54:07 xc-v-ct-shgy-spba-db-31 audit: PROCTITLE proctitle=62617368002D63004C414E473D656E5F55532E757466383B504154483D2F7573722F6C6F63616C2F7362696E3A2F7573722F6C6F63616C2F62696E3A2F7362696E3A2F62696E3A2F7573722F7362696E3A2F7573722F62696E3B6D7073746174202D50203120312031207C61776B2027424547494E7B69646C653D303B696E64

-------缺失1分钟日志后，系统重启，系统是否被hang主----------

Oct 30 06:56:11 xc-v-ct-shgy-spba-db-31 kernel: [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x481fd010]
Oct 30 06:56:11 xc-v-ct-shgy-spba-db-31 kernel: [    0.000000] Linux version 4.19.90-17.5.ky10.aarch64 NeoKylin Linux Adavanced Server @ CS2C (YHKYLIN-OS@localhost.localdomain) (gcc version 7.3.0 (GCC)) #1 SMP Fri Aug 7 13:35:33 CST 2020
Oct 30 06:56:11 xc-v-ct-shgy-spba-db-31 kernel: [    0.000000] efi: Getting EFI parameters from FDT:
Oct 30 06:56:11 xc-v-ct-shgy-spba-db-31 kernel: [    0.000000] efi: EFI v2.70 by EDK II
Oct 30 06:56:11 xc-v-ct-shgy-spba-db-31 kernel: [    0.000000] efi:  SMBIOS 3.0=0x83bef0000  MEMATTR=0x83a542018  ACPI 2.0=0x8383e0000  MEMRESERVE=0x8383bf018
Oct 30 06:56:11 xc-v-ct-shgy-spba-db-31 kernel: [    0.000000] crashkernel reserved: 0x00000000dfe00000 - 0x00000000ffe00000 (512 MB)
Oct 30 06:56:11 xc-v-ct-shgy-spba-db-31 kernel: [    0.000000] cma: Reserved 512 MiB at 0x00000000a0000000
Oct 30 06:56:11 xc-v-ct-shgy-spba-db-31 kernel: [    0.000000] ACPI: Early table checksum verification disabled
........

如下图所示，主库主机系统重启：

系统重启完成时间：

5、 日志信息对比分析

系统启动完成时间大约在“Oct 30 06:56:58 ”，从node94的kbha.log日志获悉，kbha进程重启在06:57后，和系统重启完的时间对应。及数据库sys_log日志缺失的时间对应，应该是系统重启后，数据库服务未启动，导致日志缺失。

二、分析总结

从以上综合分析获悉，此次集群主备failover切换，应该和主库主机系统重启导致，主库系统重启后，触发主备failover切换。