Kingbase V8R6集群安装部署案例---脚本在线一键扩容

案例说明：

KingbaseES V8R6支持图形化方式在线扩容，但是在一些生产环境，在服务器不支持图形化界面的情况下 ，只能通过脚本命令行的方式执行集群的部署或在线扩容。

Tips：

KingbaseES V8R6C5默认情况下部署脚本（V8R6_cluster_install.sh）和配置文件（install.conf）不支持在线扩容，需要从KingbaseES V8R6C6版本下拷贝脚本和配置文件到 KingbaseES V8R6C5环境下使用。

适用版本：

KingbaseES V8R6

一、集群节点状态信息

1、主机节点信息

2、扩容前集群节点状态

[kingbase@node101 bin]$ ./repmgr cluster show
 ID | Name    | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+-----------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------
 1  | node101 | primary | * running |          | default  | 100      | 3        | host=192.168.1.101 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node102 | standby |   running | node101  | default  | 100      | 3        | host=192.168.1.102 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

[kingbase@node101 bin]$ ./repmgr service status
 ID | Name    | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+---------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node101 | primary | * running |          | running | 2997 | no      | n/a
 2  | node102 | standby |   running | node101  | running | 2777 | no      | 0 second(s) ago

二、新增节点系统配置

扩容说明参考官方文档：

https://help.kingbase.com.cn/stage-api/profile/document/kes/v8r6/html/highly/availability/cluster-use/cluster-use-6.html#sshd

1、配置新增节点系统环境

1）扩容节点需要和原集群节点具有相同的操作系统环境，并且版本一致；

2）配置新增节点系统内核参数、防火墙、selinux、进程资源限制等，具体配置可见《KingbaseES官方文档》。

3）需在新增节点建立数据库用户及root用户到集群其他节点的ssh互信。

2、在新增节点创建目录和相关文件

=以下文件，可以从主库的数据库软件安装目录下获取。=

Tips：

V8R6_cluster_install.sh和install.conf文件从KingbaseES V8R6C6版本下拷贝，KingbaseES V8R6C5版本目前不支持。

[kingbase@node103 r6_install]$ ls -lh
total 264M
-rw-rw-r-- 1 kingbase kingbase 261M Apr  7 10:33 db.zip
-rw-rw-r-- 1 kingbase kingbase 8.2K Apr  7 10:54 install.conf
-rwxr-x--- 1 kingbase kingbase 3.2K Apr  7 10:42 license.dat
-rw-rw-r-- 1 kingbase kingbase 2.1M Apr  7 10:33 securecmdd.zip
-rwxrwxr-x 1 kingbase kingbase 3.3K Apr  7 10:33 trust_cluster.sh
-rwxrwxr-x 1 kingbase kingbase  87K Apr  7 10:33 V8R6_cluster_install.sh

三、在线脚本扩容（新增节点执行）

1、编辑install.conf

[kingbase@node103 r6_install]$ cat install.conf |grep -v ^$|grep -v ^#

[install]
on_bmj=0
all_ip=(192.168.1.101 192.168.1.102)
witness_ip=""
production_ip=()
local_disaster_recovery_ip=()
remote_disaster_recovery_ip=()
install_dir="/home/kingbase/cluster/HA_R6/kha/kingbase"
zip_package="/home/kingbase/r6_install/db.zip"
license_file=(license.dat)
db_user="system"                 # the user name of database
db_port="54321"                  # the port of database, defaults is 54321
db_mode="oracle"                 # database mode: pg, oracle
db_auth="scram-sha-256"          # database authority: scram-sha-256, md5, default is scram-sha-256
db_case_sensitive="yes"          # database case sensitive settings: yes, no. default is yes - case sensitive; no - case insensitive (NOTE. cannot set to 'no' when db_mode="pg").
trusted_servers="192.168.1.1"
data_directory="/home/kingbase/cluster/HA_R6/kha/kingbase/data"
virtual_ip=""
net_device=(enp0s3 enp0s3)
net_device_ip=(192.168.1.101 192.168.1.102)
ipaddr_path="/usr/sbin"
arping_path="/opt/Kingbase/ES/V8R6_041/Server/bin/"
ping_path="/bin"
super_user="root"
execute_user="kingbase"
deploy_by_sshd=1                 # choose whether to use sshd when deploy, 0 means not to use (deploy by sys_securecmdd), 1 means to use (deploy by sshd), default value is 1; when on_bmj=1, it will auto set to no(deploy_by_sshd=0)
use_scmd=1                       # Is the cluster running on sys_securecmdd or sshd? 1 means yes (on sys_securecmdd), 0 means no (on sshd), default value is 1; when on_bmj=1, it will auto set to yes(use_scmd=1)
reconnect_attempts="10"          # the number of retries in the event of an error
reconnect_interval="6"           # retry interval
recovery="standby"               # the way of cluster recovery: standby/automatic/manual
ssh_port="22"                    # the port of ssh, default is 22
scmd_port="8890"                 # the port of sys_securecmdd, default is 8890
auto_cluster_recovery_level='1'
use_check_disk='off'
synchronous='quorum'

###### 以下为expand配置信息#######
[expand]
expand_type="0"                   # The node type of standby/witness node, which would be add to cluster. 0:standby  1:witness
primary_ip="192.168.1101"                    # The ip addr of cluster primary node, which need to expand a standby/witness node.
expand_ip="192.168.1.103"                     # The ip addr of standby/witness node, which would be add to cluster.
node_id="3"                       # The node_id of standby/witness node, which would be add to cluster. It does not the same with any one in  cluster node
                                 # for example: node_id="3"
install_dir="/home/kingbase/cluster/HA_R6/kha"
zip_package="/home/kingbase/r6_install/db.zip"
net_device=(enp0s3)                    # if virtual_ip set,it must be set
net_device_ip=(192.168.1.103)                 # if virtual_ip set,it must be set
license_file=(license.dat)
deploy_by_sshd="1"
ssh_port="22"
scmd_port="8890"
############################################
[shrink]
shrink_type=""                   # The node type of standby/witness node, which would be delete from cluster. 0:standby  1:witness
primary_ip=""                    # The ip addr of cluster primary node, which need to shrink a standby/witness node.
shrink_ip=""                     # The ip addr of standby/witness node, which would be delete from cluster.
node_id=""                       # The node_id of standby/witness node, which would be delete from cluster. It does not the same with any one in  cluster node
                                 # for example: node_id="3"
install_dir=""
ssh_port="22"                    # the port of ssh, default is 22
scmd_port="8890"                 # the port of sys_securecmd, default is 8890

=以下为扩容（expand）节点配置信息，详细说明参考官方文档。=

2、执行扩容（expand）

[kingbase@node103 r6_install]$ sh V8R6_cluster_install.sh expand

[CONFIG_CHECK] will deploy the cluster of
[RUNNING] success connect to the target "192.168.1.103" ..... OK
.......
INFO: connecting to local node "node3" (ID: 3)
INFO: connecting to primary database
WARNING: --upstream-node-id not supplied, assuming upstream node is primary (node ID 1)
INFO: standby registration complete
NOTICE: standby node "node3" (ID: 3) successfully registered
Usage: /home/kingbase/cluster/R6HA/kha//kingbase/bin/sys_monitor.sh {start|stop|restart|stoplocal|set [--restart]|change_password user password}
 ID | Name    | Role    | Status        | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+---------------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------
 1  | node101 | primary | * running     |          | default  | 100      | 3        | host=192.168.1.101 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node102 | standby |   running     | node101  | default  | 100      | 3        | host=192.168.1.102 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 3  | node3   | standby | ? unreachable | node101  | default  | 100      | ?        | host=192.168.1.103 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

WARNING: following issues were detected
  - unable to connect to node "node3" (ID: 3)
  - node "node3" (ID: 3) is registered as an active standby but is unreachable

=如上所示，扩容成功，但是新增节点node3的状态为“unreachable”。=

三、扩容故障问题解决

1、查看集群节点状态信息

=如下所示，在主库上查询，node3状态信息为“unreachable”。=

[kingbase@node101 bin]$ ./repmgr cluster show
 ID | Name    | Role    | Status        | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+---------------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------
 1  | node101 | primary | * running     |          | default  | 100      | 3        | host=192.168.1.101 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node102 | standby |   running     | node101  | default  | 100      | 3        | host=192.168.1.102 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 3  | node3   | standby | ? unreachable | node101  | default  | 100      | ?        | host=192.168.1.103 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

WARNING: following issues were detected
  - unable to connect to node "node3" (ID: 3)

2、查看新增节点防火墙信息

=如下所示，新增节点防火墙未关闭，将防火墙关闭。=

[root@node103 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-07-20 15:09:55 CST; 47min ago
 Main PID: 815 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─815 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Jul 20 15:09:52 node103 systemd[1]: Starting firewalld - dynamic firewall daemon...
Jul 20 15:09:55 node103 systemd[1]: Started firewalld - dynamic firewall daemon.

[root@node103 ~]# systemctl stop firewalld
[root@node103 ~]# systemctl disable firewalld

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
[root@node103 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

Jul 20 15:09:52 node103 systemd[1]: Starting firewalld - dynamic firewall daemon...
Jul 20 15:09:55 node103 systemd[1]: Started firewalld - dynamic firewall daemon.
Jul 20 15:57:05 node103 systemd[1]: Stopping firewalld - dynamic firewall daemon...
Jul 20 15:57:08 node103 systemd[1]: Stopped firewalld - dynamic firewall daemon.

[root@node103 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

3、重新注册新增节点

[kingbase@node103 bin]$ ./repmgr standby register --force
INFO: connecting to local node "node3" (ID: 3)
INFO: connecting to primary database
INFO: standby registration complete
NOTICE: standby node "node3" (ID: 3) successfully registered

4、查看集群节点状态信息

[kingbase@node101 bin]$ ./repmgr cluster show
 ID | Name    | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+-----------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------
 1  | node101 | primary | * running |          | default  | 100      | 3        | host=192.168.1.101 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node102 | standby |   running | node101  | default  | 100      | 3        | host=192.168.1.102 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 3  | node3   | standby |   running | node101  | default  | 100      | 3        | host=192.168.1.103 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

=如上所示，集群所有节点状态正常，集群扩容完成。=

四、总结

1、注意配置新增节点的系统环境，如防火墙、网络、内核参数、Selinux等。

2、需要建立新增节点数据库用户和root用户与集群原节点之间的ssh互信。

3、install.conf中【install】部分为原脚本部署集群时的配置，扩容需要配置【expand】部分。