postgresql双机热备、高可用方案（采用pacemaker+corosync实现）

http://blog.csdn.net/qguanri/article/details/51151974

需求描述

我们有两台centos7的数据库主机A、B。要对A、B实现双机热备，A作为数据库master主机，对外提供读写服务，B作为slave主机能实时同步A的数据。当A发生故障时，B从slave状态切换到master状态，以保证pg数据库能正常对外提供服务。

经过调研，最终我们给出的解决方案是采用pacemaker和corosync组件来实现postgresql的高可用。

所有部署脚本可以下github中找到：https://github.com/qinguanri/pg-ha

环境说明

操作系统centos7：

[root@10 ~]# cat /etc/redhat-release

CentOS Linux release 7.2.1511 (Core)

[root@10 ~]# uname -a

Linux 10.16.93.179 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

postgresql版本：9.2.15

2台主机：

角色	IP
node1	10.16.93.179
node2	10.16.93.182

- 分配2个虚IP。我们的数据库对外提供服务时，暴露的是虚IP：

vip-master: 10.16.93.180

vip-slave: 10.16.93.181

配置Linux集群环境

注意：请严格按照下文所给的顺序进行配置

1.安装Pacemaker和Corosync包

需要在所有节点执行：

$  yum install -y pacemaker pcs psmisc policycoreutils-python postgresql-server

2.禁用防火墙

在所有节点执行：

$  systemctl disable firewalld.service

$  systemctl stop firewalld.service

3.启用pcs

在所有节点执行：

$  systemctl start pcsd.service

$  systemctl enable pcsd.service

ln -s '/usr/lib/systemd/system/pcsd.service' '/etc/systemd/system/multi-user.target.wants/pcsd.service'

$ echo hacluster |  passwd hacluster --stdin

Changing password for user hacluster.

Changing password for user hacluster.

passwd: all authentication tokens updated successfully.

4.集群认证

在任何一个节点上执行，这里选择node1：

$  pcs cluster auth -u hacluster -p hacluster 10.16.93.179 10.16.93.182

10.16.93.179: Authorized

10.16.93.182: Authorized

5.同步配置

在node1上执行：

$  pcs cluster setup --last_man_standing=1 --name pgcluster 10.16.93.179 10.16.93.182

Shutting down pacemaker/corosync services...

Redirecting to /bin/systemctl stop  pacemaker.service

Redirecting to /bin/systemctl stop  corosync.service

Killing any remaining services...

Removing all cluster configuration files...

10.16.93.179: Succeeded

10.16.93.182: Succeeded

6.启动集群

在node1上执行。（有时需要稍等十几秒，才能看到命令的输出结果）：

$ pcs cluster start --all

172.17.5.90: Starting Cluster...

172.17.5.91: Starting Cluster...

172.17.5.92: Starting Cluster...

7.检验

1）检验corosync

在node1上执行：

$ pcs status corosync

Membership information

----------------------

    Nodeid      Votes Name

         1          1 10.16.93.179 (local)

         2          1 10.16.93.182

2）检验pacemaker

root@10 data]# pcs status

Cluster name: pgcluster

WARNING: corosync and pacemaker node names do not match (IPs used in setup?)

Last updated: Mon Apr  4 23:45:48 2016          Last change: Fri Apr  1 09:17:03 2016 by root via crm_attribute on tsung_test

Stack: corosync

Current DC: tsung_test (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum

2 nodes and 5 resources configured

Online: [ 10.16.93.179 tsung_test ]

Full list of resources:

 Master/Slave Set: pgsql-cluster [pgsql]

     Masters: [ tsung_test ]

     Slaves: [ 10.16.93.179 ]

 Resource Group: master-group

     vip-master (ocf::heartbeat:IPaddr2):       Started tsung_test

 Resource Group: slave-group

     vip-slave  (ocf::heartbeat:IPaddr2):       Started 10.16.93.179

PCSD Status:

  10.16.93.179: Online

  tsung_test (10.16.93.182): Online

Daemon Status:

  corosync: active/disabled

  pacemaker: active/disabled

  pcsd: active/enabled

注意：当vip-master和vip-slave都是Started状态时，集群节点配置正确。否则，请检查你的配置是否有误

安装和配置PostgreSQL

1.创建数据库目录

需要在所有节点上执行：

$  mkdir -p /data/postgresql/{data,xlog_archive}

$  chown -R postgres:postgres /data/postgresql/

$  chmod 0700 /data/postgresql/data

2.初始化db

在node1上执行：

$  su - postgres

$ initdb -D /data/postgresql/data/

The files belonging to this database system will be owned by user "postgres".

This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".

The default database encoding has accordingly been set to "UTF8".

The default text search configuration will be set to "english".

fixing permissions on existing directory /data/postgresql/data ... ok

creating subdirectories ... ok

selecting default max_connections ... 100

selecting default shared_buffers ... 32MB

creating configuration files ... ok

creating template1 database in /data/postgresql/data/base/1 ... ok

initializing pg_authid ... ok

initializing dependencies ... ok

creating system views ... ok

loading system objects' descriptions ... ok

creating collations ... ok

creating conversions ... ok

creating dictionaries ... ok

setting privileges on built-in objects ... ok

creating information schema ... ok

loading PL/pgSQL server-side language ... ok

vacuuming database template1 ... ok

copying template1 to template0 ... ok

copying template1 to postgres ... ok

WARNING: enabling "trust" authentication for local connections

You can change this by editing pg_hba.conf or using the option -A, or

--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    postgres -D /data/postgresql/data

or

    pg_ctl -D /data/postgresql/data -l logfile start

3.修改配置文件

在node1上执行：

$ vim /data/postgresql/data/postgresql.conf

listen_addresses = '*'

wal_level = hot_standby

synchronous_commit = on

archive_mode = on

archive_command = 'cp %p /data/postgresql/xlog_archive/%f'

max_wal_senders=5

wal_keep_segments = 32

hot_standby = on

restart_after_crash = off

replication_timeout = 5000

wal_receiver_status_interval = 2

max_standby_streaming_delay = -1

max_standby_archive_delay = -1

synchronous_commit = on

restart_after_crash = off

hot_standby_feedback = on

$ vim /data/postgresql/data/pg_hba.conf

local   all             all                      trust

host    all             all    127.0.0.1/32      trust

host    all             all    ::1/128           trust

host    all             all    10.16.0.0/16      md5

host    all             all    10.18.0.0/16      md5

host    replication     all    10.16.0.0/16      md5

注意：pg_hba.conf文件用于配置postgresql的访问控制，例如：host all all 10.16.0.0/16 md5,表示允许10.16.0.0/16网段的主机访问所有用户、所有数据库，密码使用md5密文的方式。你需要根据你的实际网段进行设置。当出现访问权限错误时，请检查你的pg_hba.conf文件

4.启动

启动数据库，创建replicator用户。将用户postgres的密码修改为postgres（pg初始安装时，postgres用户的登录密码默认为空）

在node1上执行：

$ pg_ctl -D /data/postgresql/data/ start

server starting

[    2015-10-16 08:51:31.451 UTC 53158 5620ba93.cfa6 1 0]LOG:  redirecting log output to logging collector process

[    2015-10-16 08:51:31.451 UTC 53158 5620ba93.cfa6 2 0]HINT:  Future log output will appear in directory "pg_log".

$ psql -U postgres

psql (9.2.13)

Type "help" for help.

postgres=# create role replicator with login replication password '8d5e9531-3817-460d-a851-659d2e51ca99';

CREATE ROLE

postgres=# alter user postgres with password 'postgres';

postgres=# \q

5.制作slave

在node2上执行：

$  su - postgres

$ pg_basebackup -h 10.16.93.179 -U postgres -D /data/postgresql/data/ -X stream -P

could not change directory to "/home/wenhang.pan"

20127/20127 kB (100%), 1/1 tablespace

$ vim /data/postgresql/data/recovery.conf

standby_mode = 'on'

primary_conninfo = 'host=10.16.93.179 port=5432 user=replicator application_name=zhaopin-5-92 password=8d5e9531-3817-460d-a851-659d2e51ca99 keepalives_idle=60 keepalives_in

terval=5 keepalives_count=5'

restore_command = 'cp /data/postgresql/xlog_archive/%f %p'

recovery_target_timeline = 'latest'

6.启动slave

在node2上执行：

$ pg_ctl -D /data/postgresql/data/ start

pg_ctl: another server might be running; trying to start server anyway

server starting

-bash-4.2$ LOG:  database system was interrupted while in recovery at log time 2015-10-16 08:19:07 GMT

HINT:  If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.

LOG:  entering standby mode

LOG:  redo starts at 0/3000020

LOG:  consistent recovery state reached at 0/30000E0

LOG:  database system is ready to accept read only connections

LOG:  streaming replication successfully connected to primary

7.查看集群状态

在node1上执行一下命令。这个时候，可看到node2是一个replication。同步状态为sync

[root@10 ~]# psql -U postgres

psql (9.2.15)

Type "help" for help.

postgres=# select * from pg_stat_replication ;

pid  | usesysid |  usename   | application_name | client_addr  | client_hostname | client_port |         backend_start

        |   state   | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state

-------+----------+------------+------------------+--------------+-----------------+-------------+-----------------------

--------+-----------+---------------+----------------+----------------+-----------------+---------------+------------

 17349 |    16384 | replicator | 10.16.93.182     | 10.16.93.182 |                 |       22690 | 2016-04-14 14:07:25.18

1297+08 | streaming | 0/9014508     | 0/9014508      | 0/9014508      | 0/9014508       |             1 | sync

(1 row)

8.停止PostgreSQL服务

在node1、node2上执行：

$ pg_ctl -D /data/postgresql/data/ -mi stop

waiting for server to shut down.... done

server stopped

注意：在配置自动切换前，一定要先把node1、node2上的PostgreSQL服务停掉

配置自动切换

上面的配置完成了数据库的初始化。下面进行自动切换相关的配置。当发生故障时，master/slave身份实现自动切换。

1.配置

在node1执行：

1）配置项较多，我们先将配置步骤写到脚本里

[root@10 ~]# vim cluster_setup.sh

# 将cib配置保存到文件

pcs cluster cib pgsql_cfg

# 在pacemaker级别忽略quorum

pcs -f pgsql_cfg property set no-quorum-policy="ignore"

# 禁用STONITH

pcs -f pgsql_cfg property set stonith-enabled="false"

# 设置资源粘性，防止节点在故障恢复后发生迁移

pcs -f pgsql_cfg resource defaults resource-stickiness="INFINITY"

# 设置多少次失败后迁移

pcs -f pgsql_cfg resource defaults migration-threshold="3"

# 设置master节点虚ip

pcs -f pgsql_cfg resource create vip-master IPaddr2 ip="10.16.93.180" cidr_netmask="24"    op start   timeout="60s" interval="0s"  on-fail="restart"    op monitor timeout="60s" interval="10s" on-fail="restart"    op stop    timeout="60s" interval="0s"  on-fail="block"

# 设置slave节点虚ip

pcs -f pgsql_cfg resource create vip-slave IPaddr2 ip="10.16.93.181" cidr_netmask="24"    op start   timeout="60s" interval="0s"  on-fail="restart"    op monitor timeout="60s" interval="10s" on-fail="restart"    op stop    timeout="60s" interval="0s"  on-fail="block"

# 设置pgsql集群资源

# pgctl、psql、pgdata和config等配置根据自己的环境修改,node list填写节点的hostname，master_ip填写虚master_ip

pcs -f pgsql_cfg resource create pgsql pgsql pgctl="/usr/bin/pg_ctl" psql="/usr/bin/psql" pgdata="/data/postgresql/data/" config="/data/postgresql/data/postgresql.conf" rep_mode="sync" node_list="tsung_test 10.16.93.179" master_ip="10.16.93.180"  repuser="replicator" primary_conninfo_opt="password=8d5e9531-3817-460d-a851-659d2e51ca99 keepalives_idle=60 keepalives_interval=5 keepalives_count=5" restore_command="cp /data/postgresql/xlog_archive/%f %p" restart_on_promote='true' op start   timeout="60s" interval="0s"  on-fail="restart" op monitor timeout="60s" interval="4s" on-fail="restart" op monitor timeout="60s" interval="3s"  on-fail="restart" role="Master" op promote timeout="60s" interval="0s"  on-fail="restart" op demote  timeout="60s" interval="0s"  on-fail="stop" op stop    timeout="60s" interval="0s"  on-fail="block"

 # 设置master/slave模式，clone-max=2，两个节点

pcs -f pgsql_cfg resource master pgsql-cluster pgsql master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

# 配置master ip组

pcs -f pgsql_cfg resource group add master-group vip-master

# 配置slave ip组

pcs -f pgsql_cfg resource group add slave-group vip-slave

# 配置master ip组绑定master节点

pcs -f pgsql_cfg constraint colocation add master-group with master pgsql-cluster INFINITY

# 配置启动master节点

pcs -f pgsql_cfg constraint order promote pgsql-cluster then start master-group symmetrical=false score=INFINITY

# 配置停止master节点

pcs -f pgsql_cfg constraint order demote  pgsql-cluster then stop  master-group symmetrical=false score=0

# 配置slave ip组绑定slave节点

pcs -f pgsql_cfg constraint colocation add slave-group with slave pgsql-cluster INFINITY

# 配置启动slave节点

pcs -f pgsql_cfg constraint order promote pgsql-cluster then start slave-group symmetrical=false score=INFINITY

# 配置停止slave节点

pcs -f pgsql_cfg constraint order demote  pgsql-cluster then stop  slave-group symmetrical=false score=0

# 把配置文件push到cib

pcs cluster cib-push pgsql_cfg

2）执行操作文件

$  sh cluster_setup.sh

2.查看状态

1）查看cluster状态

在node1上执行：

[root@10 ~]# pcs status

Cluster name: pgcluster

WARNING: corosync and pacemaker node names do not match (IPs used in setup?)

Last updated: Mon Apr  4 23:15:56 2016          Last change: Fri Apr  1 09:17:03 2016 by root via crm_attribute on tsung_test

Stack: corosync

Current DC: tsung_test (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum

2 nodes and 5 resources configured

Online: [ 10.16.93.179 tsung_test ]

Full list of resources:

 Master/Slave Set: pgsql-cluster [pgsql]

     Masters: [ tsung_test ]

     Slaves: [ 10.16.93.179 ]

 Resource Group: master-group

     vip-master (ocf::heartbeat:IPaddr2):       Started tsung_test

 Resource Group: slave-group

     vip-slave  (ocf::heartbeat:IPaddr2):       Started 10.16.93.179

PCSD Status:

  10.16.93.179: Online

  tsung_test (10.16.93.182): Online

Daemon Status:

  corosync: active/disabled

  pacemaker: active/disabled

  pcsd: active/enabled

2）查看PostgreSQL集群状态

在node1上执行：

[root@tsung_test data]# psql -U postgres

psql (9.2.15)

Type "help" for help.

postgres=# select * from pg_stat_replication ;

  pid  | usesysid |  usename   | application_name | client_addr  | client_hostname | client_port |         backend_start         |   state   | sent_location | write_locatio

n | flush_location | replay_location | sync_priority | sync_state

-------+----------+------------+------------------+--------------+-----------------+-------------+-------------------------------+-----------+---------------+--------------

--+----------------+-----------------+---------------+------------

 19415 |    16384 | replicator | 10.16.93.182     | 10.16.93.182 |                 |       18289 | 2016-04-01 09:17:29.037197-04 | streaming | 0/40F0ED8     | 0/40F0ED8

  | 0/40F0ED8      | 0/40F0ED8       |             0 | async

(1 row)

验证自动切换

参考文献