KingbaseES V8R3集群管理和维护案例之---failover切换wal日志变化分析

​

案例说明：

本案例通过对KingbaseES V8R3集群failover切换过程进行观察，分析了主备库切换后wal日志的变化，对应用者了解KingbaseES V8R3(R6) failover切换过程有一定的帮助。

以下为现场案例：

failover切换后主备库的wal日志信息：

新主库数据库服务启动故障：（sys_log）

=如下所示，在sys_log中，新主库启动startup后，建立流复制，流复制的起始wal日志是：“ 00000004000000050000002A”，导致复制失败。=

适用版本：

KingbaseES V8R3/R6

节点信息：

集群节点状态信息：

[kingbase@node102 bin]$ ./ksql -U SYSTEM -W 123456 TEST -p 9999
ksql (V008R003C002B0290)
Type "help" for help.

TEST=# show pool_nodes;
 node_id |   hostname    | port  | status | lb_weight |  role   | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
 0       | 192.168.1.101 | 54321 | up     | 0.500000  | standby | 0          | true              | 0
 1       | 192.168.1.102 | 54321 | up     | 0.500000  | primary | 0          | false             | 0
(2 rows)

一、查看failover切换前节点信息

1、原主库wal日志

[kingbase@node102 sys_xlog]$ ls -lh
.......
-rw------- 1 kingbase kingbase  16M Jul  7 15:25 00000009000000000000002D.partial
-rw------- 1 kingbase kingbase  339 Jul  7 15:09 00000009.history
-rw------- 1 kingbase kingbase  16M Jul 29 10:56 0000000A000000000000002D
-rw------- 1 kingbase kingbase  16M Jul 29 16:32 0000000A000000000000002E
-rw------- 1 kingbase kingbase  16M Aug  3 10:22 0000000A000000000000002F
-rw------- 1 kingbase kingbase  382 Jul  7 15:25 0000000A.history

2、原主库控制文件信息

[kingbase@node102 bin]$ ./sys_controldata -D ../data
sys_control version number:            830
Catalog version number:               201608131
Database system identifier:           7080416207291699599
Database cluster state:               in production
sys_control last modified:             Wed 03 Aug 2022 10:26:57 AM CST
Latest checkpoint location:           0/2F000108
Prior checkpoint location:            0/2F000028
Latest checkpoint's REDO location:    0/2F0000D0
Latest checkpoint's REDO WAL file:    0000000A000000000000002F
Latest checkpoint's TimeLineID:       10
Latest checkpoint's PrevTimeLineID:   10

3、备库wal日志

[kingbase@node101 bin]$ ls -lh ../data/sys_xlog
.......
-rw------- 1 kingbase kingbase  16M Jul  7 15:25 00000009000000000000002D
-rw------- 1 kingbase kingbase  339 Jun 22 16:15 00000009.history
-rw------- 1 kingbase kingbase  16M Jul 29 16:14 0000000A000000000000002D
-rw------- 1 kingbase kingbase  16M Aug  3 10:22 0000000A000000000000002E
-rw------- 1 kingbase kingbase  16M Aug  3 10:27 0000000A000000000000002F
-rw------- 1 kingbase kingbase  382 Jul 29 10:33 0000000A.history

4、备库控制文件信息

[kingbase@node101 bin]$ ./sys_controldata -D ../data
sys_control version number:            830
Catalog version number:               201608131
Database system identifier:           7080416207291699599
Database cluster state:               in archive recovery
sys_control last modified:             Wed 03 Aug 2022 10:26:55 AM CST
Latest checkpoint location:           0/2F000028
Prior checkpoint location:            0/2E0002C8
Latest checkpoint's REDO location:    0/2F000028
Latest checkpoint's REDO WAL file:    0000000A000000000000002F
Latest checkpoint's TimeLineID:       10
Latest checkpoint's PrevTimeLineID:   10

二、执行failover切换（关闭主库数据库服务）

1、关闭主库数据库服务

[kingbase@node102 bin]$ ./sys_ctl stop -D ../data
waiting for server to shut down....... done
server stopped

三、failover切换完成主备状态信息

1、新主库wal日志

[kingbase@node101 bin]$ ls -lh ../data/sys_xlog
.......
-rw------- 1 kingbase kingbase  339 Jun 22 16:15 00000009.history
-rw------- 1 kingbase kingbase  16M Jul 29 16:14 0000000A000000000000002D
-rw------- 1 kingbase kingbase  16M Aug  3 10:22 0000000A000000000000002E
-rw------- 1 kingbase kingbase  16M Aug  3 10:30 0000000A000000000000002F
-rw------- 1 kingbase kingbase  16M Aug  3 10:30 0000000A0000000000000030.partial
-rw------- 1 kingbase kingbase  382 Jul 29 10:33 0000000A.history
-rw------- 1 kingbase kingbase  16M Aug  3 10:31 0000000B0000000000000030
-rw------- 1 kingbase kingbase  426 Aug  3 10:30 0000000B.history

切换完成后timeline发生切换：

查看timeline history文件信息：

2、新主库控制文件信息

[kingbase@node101 bin]$  ./sys_controldata -D ../data
sys_control version number:            830
Catalog version number:               201608131
Database system identifier:           7080416207291699599
Database cluster state:               in production
sys_control last modified:             Wed 03 Aug 2022 10:35:48 AM CST
Latest checkpoint location:           0/3005E110
Prior checkpoint location:            0/30004BD8
Latest checkpoint's REDO location:    0/3005B370
Latest checkpoint's REDO WAL file:    0000000B0000000000000030
Latest checkpoint's TimeLineID:       11
Latest checkpoint's PrevTimeLineID:   11

3、新备库wal日志

[kingbase@node102 bin]$ ls -lh ../data/sys_xlog
.......
-rw------- 1 kingbase kingbase  16M Jul 29 10:56 0000000A000000000000002D
-rw------- 1 kingbase kingbase  16M Jul 29 16:32 0000000A000000000000002E
-rw------- 1 kingbase kingbase  16M Aug  3 10:34 0000000A000000000000002F
-rw------- 1 kingbase kingbase  16M Aug  3 10:34 0000000A0000000000000030.partial
-rw------- 1 kingbase kingbase  382 Aug  3 10:34 0000000A.history
-rw------- 1 kingbase kingbase  16M Aug  3 10:34 0000000B0000000000000030
-rw------- 1 kingbase kingbase  426 Aug  3 10:34 0000000B.history

4、新备库控制文件信息

[kingbase@node102 bin]$ ./sys_controldata -D ../data
sys_control version number:            830
Catalog version number:               201608131
Database system identifier:           7080416207291699599
Database cluster state:               in archive recovery
sys_control last modified:             Wed 03 Aug 2022 10:35:42 AM CST
Latest checkpoint location:           0/30004BD8
Prior checkpoint location:            0/30004BD8
Latest checkpoint's REDO location:    0/30004BA0
Latest checkpoint's REDO WAL file:    0000000B0000000000000030
Latest checkpoint's TimeLineID:       11
Latest checkpoint's PrevTimeLineID:   11

四、将原主库作为备库恢复到集群

1、在原主库data下创建recovery.conf

[kingbase@node102 data]$ cp ../etc/recovery.done ./recovery.conf

2、查看recovery.log信息

primary node/Im node status is changed, primary ip[192.168.1.101], recovery.conf NEED_CHANGE [0] (0 is need ), I,m status is [1] (1 is down), I will be in recovery.
 node_id |   hostname    | port  | status | lb_weight |  role   | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
 0       | 192.168.1.101 | 54321 | up     | 0.500000  | primary | 0          | true              | 0
 1       | 192.168.1.102 | 54321 | down   | 0.500000  | standby | 0          | false             | 0
(2 rows)

if recover node up, let it down , for rewind
2022-08-03 10:34:35 sys_rewind...
sys_rewind  --target-data=/home/kingbase/cluster/R3HA/db/data --source-server="host=192.168.1.101 port=54321 user=SUPERMANAGER_V8ADMIN dbname=TEST"
datadir_source = /home/kingbase/cluster/R3HA/db/data
rewinding from last common checkpoint at 0/2F000108 on timeline 10
find last common checkpoint start time from 2022-08-03 10:34:35.349563 CST to 2022-08-03 10:34:35.405349 CST, in "0.055786" seconds.
reading source file list
reading target file list
reading WAL in target
Rewind datadir file from source
update the control file: minRecoveryPoint is '0/3004D0C8', minRecoveryPointTLI is '11', and database state is 'in archive recovery'
rewind start wal location 0/2F0000D0 (file 0000000A000000000000002F), end wal location 0/3004D0C8 (file 0000000B0000000000000030). time from 2022-08-03 10:34:37.349563 CST to 2022-08-03 10:34:37.872586 CST, in "2.523023" seconds.
Done!
 sed conf change #synchronous_standby_names

2022-08-03 10:34:39 file operate
cp recovery.conf...
 change recovery.conf ip -> primary.ip
2022-08-03 10:34:39 change recovery.conf
delete pid file if exist
del the replication_slots if exis
drop the slot [slot_node101].
drop the slot [slot_node102].
2022-08-03 10:34:40 start up the kingbase...
waiting for server to start....LOG:  redirecting log output to logging collector process
HINT:  Future log output will appear in directory "/home/kingbase/cluster/R3HA/db/data/sys_log".
 done
server started
ksql "port=54321 user=SUPERMANAGER_V8ADMIN dbname=TEST connect_timeout=10"   -c "select 33333;"
 SYS_CREATE_PHYSICAL_REPLICATION_SLOT
--------------------------------------
 (slot_node101,)
(1 row)

2022-08-03 10:34:42 create the slot [slot_node101] success.
 SYS_CREATE_PHYSICAL_REPLICATION_SLOT
--------------------------------------
 (slot_node102,)
(1 row)

2022-08-03 10:34:42 create the slot [slot_node102] success.
2022-08-03 10:34:42 start up standby successful!
can not get the replication of myself

如下所示：recovery过程：

五、总结

在集群执行failover切换时，可以结合wal日志和recovery.log和控制文件的变化，可以详细了解failover切换中wal日志的变化，及通过sys_rewind工具对原主库的恢复过程。