KingbaseES R6 集群物理copy方式手工添加新备库节点

案例说明： 对于主库数据量比较大的环境，在添加新节点是可以采用在线clone方式创建新的备库节点，也可以在离线的状态下，直接拷贝其中一个备库的所有集群相关目录来创建新的备库节点。本案例介绍了通过离线物理copy目录的方式创建新的备库节点，包括详细的操作步骤。

一、案例环境

​操作系统：

[root@node1 ~]# cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)
  数据库：
test=# select version();
                                                       version
------------------------------------------------------------------------------
 KingbaseES V008R006C003B0010 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46), 64-bit
(1 row)

集群架构：

一、配置新节点系统环境（和集群其他节点相同）

1、配置资源访问限制策略

cat /etc/security/limits.conf
​
# End of file
​
  * soft nofile 655360
    root soft nofile 655360
    * hard nofile 655360
    root hard nofile 655360
    * soft nproc 655360
    root soft nproc 655360
    * hard nproc 655360
    root hard nproc 655360
    * soft core unlimited
    root soft core unlimited
    * hard core unlimited
    root hard core unlimited
    * soft memlock 50000000
    root soft memlock 50000000
    * hard memlock 50000000
    root hard memlock 50000000

2、配置sysctl.conf文件

[root@node2 ~]# cat /etc/sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
​
    kernel.sem= 5010 641280 5010 256
    fs.file-max=7672460
    fs.aio-max-nr=1048576
    net.core.rmem_default=262144
    net.core.rmem_max=4194304
    net.core.wmem_default=262144
    net.core.wmem_max=4194304
    net.ipv4.ip_local_port_range=9000 65500
    net.ipv4.tcp_wmem=8192 65536 16777216
    net.ipv4.tcp_rmem=8192 87380 16777216
    vm.min_free_kbytes=512000
    vm.vfs_cache_pressure=200
    vm.swappiness=20
    net.ipv4.tcp_max_syn_backlog=4096
    net.core.somaxconn=4096

3、关闭新节点防火墙

[root@node3 .ssh]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-06-24 13:59:09 CST; 23min ago
 Main PID: 798 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─798 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
​
Jun 24 13:59:06 localhost.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Jun 24 13:59:09 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
[root@node3 .ssh]# systemctl stop firewalld
[root@node3 .ssh]# systemctl disable firewalld
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
​
[root@node3 .ssh]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
​
Jun 24 13:59:06 localhost.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Jun 24 13:59:09 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
Jun 24 14:22:46 node3 systemd[1]: Stopping firewalld - dynamic firewall daemon...
Jun 24 14:22:48 node3 systemd[1]: Stopped firewalld - dynamic firewall daemon. 

4、验证新节点和其他节点的ssh信任关系

kingbase用户：kingbase 用户能以 root 用户免密ssh 到 所有节点。

root用户：

5、关闭selinux

三、查看当前集群状态现象

1、集群节点状态信息

[kingbase@node1 bin]$ ./repmgr cluster show
 ID | Name    | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+-----------+----------+----------+----------+----------+
 1  | node248 | primary | * running |          | default  | 100      | 5        | host=192.168.7.248 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node249 | standby |   running | node248  | default  | 100      | 5        | host=192.168.7.249 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

2、主备流复制状态信息

test=# select * from sys_stat_replication;
  pid  | usesysid | usename | application_name |  client_addr  | client_hostname | client_port |         backend_start         | backend_xmin |   state   |  sent_lsn  | write_lsn  | flush_lsn  | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state |          reply_time
-------+----------+---------+------------------+---------------+-----------------+
 13347 |    16384 | esrep   | node249          | 192.168.7.249 |                 |       23228 | 2021-03-01 14:45:03.723296+08 |              | streaming | 1/F205BC90 | 1/F205BC90 | 1/F205BC90 | 1/F205BC90 |           |           |            |             1 | quorum     | 2021-03-01 14:54:58.127023+08
(1 row)

四、物理copy创建新备库节点

1、新备库创建目录（和集群其他节点一致）

[kingbase@node3 .ssh]$ mkdir -p /home/kingbase/cluster/R6HA/KHA/kingbase/data

2、从已有的主库节点拷贝集群相关目录和文件

[kingbase@node2 KHA]$ scp -r * node3:/home/kingbase/cluster/R6HA/KHA/

源库必须是正常关闭状态。

3、配置ip和arping可执行文件权限

===如果集群使用vip，需要对ip和arping可执行文件配置setuid权限。===

[root@node3 soft]# chmod 4755 /sbin/ip
[root@node3 soft]# chmod 4755 /sbin/arping
[root@node3 soft]# ls -lh /sbin/ip
-rwsr-xr-x. 1 root root 319K Nov 20  2015 /sbin/ip
[root@node3 soft]# ls -lh /sbin/arping
-rwsr-xr-x. 1 root root 24K Nov 21  2015 /sbin/arping

四、将新备库节点加入集群

1、编辑repmgr.conf文件

[kingbase@node3 etc]$ cat repmgr.conf
on_bmj=off
node_id=3
node_name=node243
promote_command='/home/kingbase/cluster/R6HA/KHA/kingbase/bin/repmgr  standby promote -f /home/kingbase/cluster/R6HA/KHA/kingbase/etc/repmgr.conf'
follow_command='/home/kingbase/cluster/R6HA/KHA/kingbase/bin/repmgr  standby follow  -f /home/kingbase/cluster/R6HA/KHA/kingbase/etc/repmgr.conf -W --upstream-node-id=%n'
conninfo='host=192.168.7.243 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3'
log_file='/home/kingbase/cluster/R6HA/KHA/kingbase/hamgr.log'
data_directory='/home/kingbase/cluster/R6HA/KHA/kingbase/data'
sys_bindir='/home/kingbase/cluster/R6HA/KHA/kingbase/bin'
ssh_options='-q -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ServerAliveInterval=2 -o ServerAliveCountMax=5 -p 22'
reconnect_attempts=3
reconnect_interval=5
failover='automatic'
recovery='manual'
monitoring_history='no'
trusted_servers='192.168.7.1'
virtual_ip='192.168.7.240/24'
net_device='enp0s3'
ipaddr_path='/sbin'
arping_path='/sbin'
synchronous='quorum'
repmgrd_pid_file='/home/kingbase/cluster/R6HA/KHA/kingbase/hamgrd.pid'
ping_path='/usr/bin'

[kingbase@node3 bin]$ ./repmgr node rejoin -h 192.168.7.248 -U esrep -d esrep --force

ERROR: database is still running in state "shut down in recovery"

HINT: "repmgr node rejoin" cannot be executed on a running node

2、启动备库数据库服务

启动数据库：

[kingbase@node3 bin]$ chmod 700 ../data
[kingbase@node3 bin]$ ./sys_ctl start -D ../data
waiting for server to start....2021-03-01 13:59:02.770 CST [20835] LOG:  sepapower extension initialized
2021-03-01 13:59:02.813 CST [20835] LOG:  starting KingbaseES V008R006C003B0010 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46), 64-bit
2021-03-01 13:59:02.814 CST [20835] LOG:  listening on IPv4 address "0.0.0.0", port 54321
2021-03-01 13:59:02.814 CST [20835] LOG:  listening on IPv6 address "::", port 54321
2021-03-01 13:59:02.859 CST [20835] LOG:  listening on Unix socket "/tmp/.s.KINGBASE.54321"
2021-03-01 13:59:02.902 CST [20835] LOG:  redirecting log output to logging collector process
2021-03-01 13:59:02.902 CST [20835] HINT:  Future log output will appear in directory "sys_log".
 done
server started

3、将节点作为standby注册到集群

[kingbase@node3 bin]$ ./repmgr standby register --force
INFO: connecting to local node "node243" (ID: 3)
INFO: connecting to primary database
WARNING: --upstream-node-id not supplied, assuming upstream node is primary (node ID 1)
WARNING: local node not attached to primary node 1
NOTICE: -F/--force supplied, continuing anyway
INFO: standby registration complete
NOTICE: standby node "node243" (ID: 3) successfully registered

4、将节点加入到集群

[kingbase@node3 bin]$ ./repmgr node rejoin -h 192.168.7.248 -U esrep -d esrep
ERROR: connection to database failed
DETAIL:
fe_sendauth: no password supplied

 === 由以上获知，加入节点无法访问主库数据库，认证失败。===

5、将集群其他节点的认证文件拷贝到新节点的宿主目录下

[kingbase@node3 ~]$ ls -lha .encpwd
-rw-------. 1 kingbase kingbase 55 Mar  1 14:33 .encpwd
[kingbase@node3 ~]$ cat .encpwd
*:*:*:system:MTIzNDU2
*:*:*:esrep:S2luZ2Jhc2VoYTExMA==

=== 对于R6集群使用了.encpwd的隐藏文件，用于系统用户免密登录数据库===

6、新节点加入到集群

[kingbase@node3 bin]$ ./repmgr node rejoin -h 192.168.7.248 -U esrep -d esrep
INFO: timelines are same, this server is not ahead
DETAIL: local node lsn is 1/F2055920, rejoin target lsn is 1/F2062AB0
NOTICE: setting node 3's upstream to node 1
WARNING: unable to ping "host=192.168.7.243 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3"
DETAIL: PQping() returned "PQPING_NO_RESPONSE"
NOTICE: begin to start server at 2021-03-01 14:34:19.973116
NOTICE: starting server using "/home/kingbase/cluster/R6HA/KHA/kingbase/bin/sys_ctl  -w -t 90 -D '/home/kingbase/cluster/R6HA/KHA/kingbase/data' -l /home/kingbase/cluster/R6HA/KHA/kingbase/bin/logfile start"
NOTICE: start server finish at 2021-03-01 14:34:20.187969
NOTICE: NODE REJOIN successful
DETAIL: node 3 is now attached to node 1

  === 从以上获知，新节点node243作为备库加入到集群中===

7、查看集群节点状态

[kingbase@node3 bin]$ ./repmgr cluster show
 ID | Name    | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+-----------+----------+----------+----------+----------+--------
 1  | node248 | primary | * running |          | default  | 100      | 5        | host=192.168.7.248 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node249 | standby |   running | node248  | default  | 100      | 5        | host=192.168.7.249 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 3  | node243 | standby |   running | node248  | default  | 100      | 5        | host=192.168.7.243 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

8、查看主备流复制状态

test=# select * from sys_stat_replication;
  pid  | usesysid | usename | application_name |  client_addr  | client_hostname | client_port |         backend_start         | backend_xmin |   state   |  sent_lsn  | write_lsn  | flush_lsn  | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state |          reply_time
-------+----------+---------+------------------+---------------+-----------------+-------------+
 13347 |    16384 | esrep   | node249          | 192.168.7.249 |                 |       23228 | 2021-03-01 14:45:03.723296+08 |              | streaming | 1/F20659C8 | 1/F20659C8 | 1/F20659C8 | 1/F20659C8 |           |           |            |             1 | quorum     | 2021-03-01 15:35:31.027066+08
 25123 |    16384 | esrep   | node243          | 192.168.7.243 |                 |       49130 | 2021-03-01 15:33:59.607489+08 |              | streaming | 1/F20659C8 | 1/F20659C8 | 1/F20659C8 | 1/F20659C8 |           |           |            |             1 | quorum     | 2021-03-01 14:36:01.384836+08

9、数据同步测试

1） Primary DML操作

test=# \c prod
You are now connected to database "prod" as user "system".
​
prod=# create table t8 (like t7);
CREATE TABLE
prod=# \d
                     List of relations
 Schema |        Name         |       Type        | Owner
--------+---------------------+-------------------+--------
 ......
 public | t8                  | table             | system
(16 rows)

2） standby 查看同步数据

You are now connected to database "prod" as user "system".
prod=# \d
                     List of relations
 Schema |        Name         |       Type        | Owner
--------+---------------------+-------------------+--------
 .......
 public | t8                  | table             | system
(16 rows)

五、重新启动集群测试

1、重新启动集群

[kingbase@node1 bin]$ ./sys_monitor.sh restart
2021-03-01 15:37:28 Ready to stop all DB ...
Service process "node_export" was killed at process 14434
Service process "postgres_ex" was killed at process 14435
Service process "node_export" was killed at process 14008
Service process "postgres_ex" was killed at process 14009
There is no service "node_export" running currently.
There is no service "postgres_ex" running currently.
2021-03-01 15:37:37 begin to stop repmgrd on "[192.168.7.248]".
2021-03-01 15:37:38 repmgrd on "[192.168.7.248]" stop success.
2021-03-01 15:37:38 begin to stop repmgrd on "[192.168.7.249]".
2021-03-01 15:37:39 repmgrd on "[192.168.7.249]" stop success.
2021-03-01 15:37:39 begin to stop repmgrd on "[192.168.7.243]".
2021-03-01 15:37:40 repmgrd on "[192.168.7.243]" already stopped.
2021-03-01 15:37:40 begin to stop DB on "[192.168.7.249]".
waiting for server to shut down.... done
server stopped
2021-03-01 15:37:41 DB on "[192.168.7.249]" stop success.
2021-03-01 15:37:41 begin to stop DB on "[192.168.7.243]".
waiting for server to shut down.... done
server stopped
2021-03-01 15:37:43 DB on "[192.168.7.243]" stop success.
2021-03-01 15:37:43 begin to stop DB on "[192.168.7.248]".
waiting for server to shut down...... done
server stopped
2021-03-01 15:37:46 DB on "[192.168.7.248]" stop success.
2021-03-01 15:37:46 Done.
2021-03-01 15:37:46 Ready to start all DB ...
2021-03-01 15:37:46 begin to start DB on "[192.168.7.248]".
waiting for server to start.... done
server started
2021-03-01 15:37:48 execute to start DB on "[192.168.7.248]" success, connect to check it.
2021-03-01 15:37:49 DB on "[192.168.7.248]" start success.
2021-03-01 15:37:49 Try to ping trusted_servers on host 192.168.7.248 ...
2021-03-01 15:37:52 Try to ping trusted_servers on host 192.168.7.249 ...
2021-03-01 15:37:54 Try to ping trusted_servers on host 192.168.7.243 ...
2021-03-01 15:37:57 begin to start DB on "[192.168.7.249]".
waiting for server to start.... done
server started
2021-03-01 15:37:59 execute to start DB on "[192.168.7.249]" success, connect to check it.
2021-03-01 15:38:00 DB on "[192.168.7.249]" start success.
2021-03-01 15:38:00 begin to start DB on "[192.168.7.243]".
waiting for server to start.... done
server started
2021-03-01 15:38:02 execute to start DB on "[192.168.7.243]" success, connect to check it.
2021-03-01 15:38:03 DB on "[192.168.7.243]" start success.
 ID | Name    | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+-----------+----------+----------+----------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------
 1  | node248 | primary | * running |          | default  | 100      | 5        | host=192.168.7.248 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node249 | standby |   running | node248  | default  | 100      | 5        | host=192.168.7.249 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 3  | node243 | standby |   running | node248  | default  | 100      | 5        | host=192.168.7.243 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
2021-03-01 15:38:03 The primary DB is started.
2021-03-01 15:38:07 Success to load virtual ip [192.168.7.240/24] on primary host [192.168.7.248].
2021-03-01 15:38:07 Try to ping vip on host 192.168.7.248 ...
2021-03-01 15:38:10 Try to ping vip on host 192.168.7.249 ...
2021-03-01 15:38:13 Try to ping vip on host 192.168.7.243 ...
2021-03-01 15:38:16 begin to start repmgrd on "[192.168.7.248]".
[2021-03-01 15:38:17] [NOTICE] using provided configuration file "/home/kingbase/cluster/R6HA/KHA/kingbase/bin/../etc/repmgr.conf"
[2021-03-01 15:38:17] [NOTICE] redirecting logging output to "/home/kingbase/cluster/R6HA/KHA/kingbase/hamgr.log"
​
2021-03-01 15:38:17 repmgrd on "[192.168.7.248]" start success.
2021-03-01 15:38:17 begin to start repmgrd on "[192.168.7.249]".
[2021-03-01 15:38:08] [NOTICE] using provided configuration file "/home/kingbase/cluster/R6HA/KHA/kingbase/bin/../etc/repmgr.conf"
[2021-03-01 15:38:08] [NOTICE] redirecting logging output to "/home/kingbase/cluster/R6HA/KHA/kingbase/hamgr.log"
​
2021-03-01 15:38:18 repmgrd on "[192.168.7.249]" start success.
2021-03-01 15:38:18 begin to start repmgrd on "[192.168.7.243]".
[2021-03-01 14:38:40] [NOTICE] using provided configuration file "/home/kingbase/cluster/R6HA/KHA/kingbase/bin/../etc/repmgr.conf"
[2021-03-01 14:38:40] [NOTICE] redirecting logging output to "/home/kingbase/cluster/R6HA/KHA/kingbase/hamgr.log"
​
2021-03-01 15:38:19 repmgrd on "[192.168.7.243]" start success.
 ID | Name    | Role    | Status    | Upstream | repmgrd | PID   | Paused? | Upstream last seen
----+---------+---------+-----------+----------+---------+-------+---------+--------------------
 1  | node248 | primary | * running |          | running | 27678 | no      | n/a
 2  | node249 | standby |   running | node248  | running | 25551 | no      | 1 second(s) ago
 3  | node243 | standby |   running | node248  | running | 20067 | no      | n/a
2021-03-01 15:38:31 Done.

=== 从以上获知，新的节点已经可以通过sys_monitor.sh进行管理。===

2、查看集群节点状态

[kingbase@node1 bin]$ ./repmgr cluster show
 ID | Name    | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+-----------+----------+----------+----------+----------+--------
 1  | node248 | primary | * running |          | default  | 100      | 5        | host=192.168.7.248 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node249 | standby |   running | node248  | default  | 100      | 5        | host=192.168.7.249 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 3  | node243 | standby |   running | node248  | default  | 100      | 5        | host=192.168.7.243 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

3、在通过sys_monitor.sh启动集群时自动在新节点创建crond服务

[root@node3 cron.d]# cat KINGBASECRON
*/1 * * * * kingbase . /etc/profile;/home/kingbase/cluster/R6HA/KHA/kingbase/bin/kbha -A daemon -f /home/kingbase/cluster/R6HA/KHA/kingbase/bin/../etc/repmgr.conf >> /home/kingbase/cluster/R6HA/KHA/kingbase/bin/../kbha.log 2>&
​

六、操作步骤总结：

1）配置新节点系统环境和集群其他节点保持一致。
    2）手工配置新节点和集群其他节点：root----root，kingbase----kingbase，kingbase----root用户之间的信任关系。
    3）关闭新节点防火墙和selinux。
    4）关闭集群，从主库拷贝集群目录和相关文件到新节点（包括数据库）。
    5）配置ip和arping可执行文件的setuid权限。
    6）配置新备库repmgr.conf文件。
    7）启动集群，启动新备库数据库服务，将新备库注册到集群。
    8）拷贝.encpw文件到新备库，关闭新备库数据库服务，将新备库节点加入到集群。
    9）验证集群所有节点状态信息和流复制信息。
  10）重新启动集群验证。