KingbaseES V8R3集群维护案例之---pcp_node_refresh应用

案例说明：

在一次KingbaseES V8R3集群切换分析中，运维人员执行了pcp_node_refresh，导致集群发生了failover的切换。此文档对pcp_node_refresh工具做了应用说明及测试，不建议在生产环境使用此工具。

适用版本：

KingbaseES V8R3

一、pcp_node_refresh应用说明

1.  pcp_node_refresh 必须要指定节点ID（就是show pool_nodes中查出来的id），刷新这个节点的状态。动作是将数据库踢出集群，然后再加入进来。
2.  pcp_node_refresh 指向主库，将主库踢出集群，这个时候会认为主库故障触发会切换  ----------- 集群failover切换。
3.  pcp_node_refresh 指向备库，将备库踢出集群再加入，集群无影响。

二、测试案例

1、集群节点状态

TEST=# show pool_nodes;
 node_id |   hostname    | port  | status | lb_weight |  role   | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
 0       | 192.168.1.101 | 54321 | up     | 0.500000  | primary | 0          | true              | 0
 1       | 192.168.1.102 | 54321 | down   | 0.500000  | standby | 0          | false             | 0
(2 rows)

---如上所示，备库节点状态是down，执行pcp_node_refresh刷新节点。

2、执行pcp_node_refresh刷新备库节点

1）执行pcp_node_refresh

# 执行pcp_node_refresh
[kingbase@node101 bin]$ ./pcp_node_refresh -U kingbase -h 192.168.1.101 1
Password:
pcp_node_refresh -- Command Successful

# 查看节点状态
[kingbase@node101 bin]$ ./ksql -U SYSTEM -W 123456 TEST -p 9999
ksql (V008R003C002B0290)
Type "help" for help.

TEST=# show pool_nodes;
 node_id |   hostname    | port  | status | lb_weight |  role   | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
 0       | 192.168.1.101 | 54321 | up     | 0.500000  | primary | 0          | false             | 0
 1       | 192.168.1.102 | 54321 | up     | 0.500000  | standby | 0          | true              | 0
(2 rows)

TEST=# select * from sys_stat_replication;
  PID  | USESYSID | USENAME | APPLICATION_NAME |  CLIENT_ADDR  | CLIENT_HOSTNAME | CLIENT_PORT |         BACKEND_START         | B
ACKEND_XMIN |   STATE   | SENT_LOCATION | WRITE_LOCATION | FLUSH_LOCATION | REPLAY_LOCATION | SYNC_PRIORITY | SYNC_STATE
-------+----------+---------+------------------+---------------+-----------------+-------------+-------------------------------+--
------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
 28587 |       10 | SYSTEM  | node101          | 192.168.1.102 |                 |       47629 | 2022-09-05 13:42:43.870560+08 |
            | streaming | 1/E92607D0    | 1/E92607D0     | 1/E92607D0     | 1/E9260780      |             0 | async
(1 row)

---如上所示，备库节点状态恢复正常。

2）查看日志信息

如下cluster.log所示：执行pcp_node_refresh后，对备库节点作为故障节点进行了recovery的恢复。

如下recovery.log所示，执行pcp_node_refresh后，对备库节点调用sys_rewind恢复备库节点。

3、执行pcp_node_refresh刷新主库节点

1）执行pcp_node_refresh

[kingbase@node101 bin]$ ./pcp_node_refresh -U kingbase -h 192.168.1.101 0
Password:
pcp_node_refresh -- Command Successful

TEST=# show pool_nodes;
 node_id |   hostname    | port  | status | lb_weight |  role   | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
 0       | 192.168.1.101 | 54321 | up     | 0.500000  | standby | 0          | true              | 0
 1       | 192.168.1.102 | 54321 | up     | 0.500000  | primary | 0          | false             | 0
(2 rows)

---如上所示，主备节点发生了切换。

2）查看日志信息

如下cluster.log所示：执行pcp_node_refresh后，对主库节点作为故障调用failover_stream.sh执行了failover切换。

如下failover.log所示：执行pcp_node_refresh后，failover切换过程。

-----------------2022-09-05 13:45:42 failover beging---------------------------------------
----failover-stats is %H = hostname of the new master node [192.168.1.102], %P = old primary node id [1], %d = node id[0], %h = host name [192.168.1.101], %O = old primary host[192.168.1.102] %m = new master node id [1], %M = old master node id [1], %D = database cluster path [/home/kingbase/cluster/R3HA/db/data].
----ping trust ip
ping trust ip 192.168.1.1 success
----determine whether the faulty db is master or standby
standby down, master still 192.168.1.102
The sys_stat_replication view result is : []
2022-09-05 13:45:44 sync to async
ALTER SYSTEM
 SYS_RELOAD_CONF
-----------------
 t
(1 row)

-----------------2022-09-05 13:45:44 failover end---------------------------------------

三、总结

pcp_node_refresh不建议在生产环境使用，对于备库节点在数据库服务和流复制状态都正常的情况下，如果用show pool_nodes显示的是‘down’状态，可以使用pcp_attach_node重新注册备库节点，使节点状态恢复正常。