MHA快速搭建

很早之前写过MHA的文章，但是常常在技术群看到有同学问MHA搭建的问题，不是权限问题就是配置问题，我在这里就再次一写下配置过程以及快速的搭建。如果想知道更多的细节与原理，请参考：MySQL高可用架构之MHA

环境：

1主1从，manager放在从库。

主库：192.168.0.10

从库：192.168.0.20

两台机器的mysql安装完成初始化以后进行复制搭建，首先登录主库（192.168.0.10），查看pos点：

mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000001 |      154 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

然后在主库（192.168.0.10）
添加复制账号以及mha用的账号

mysql> grant replication slave on *.* to 'repl'@'192.168.0.10'  identified by 'repl';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> grant replication slave on *.* to 'repl'@'192.168.0.20'  identified by 'repl';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> grant all on *.* to 'root'@'192.168.0.20'  identified by '';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> grant all on *.* to 'root'@'192.168.0.10'  identified by '';
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> 

从库（192.168.0.10 ）change到mysql-bin.000001，pos点154

 CHANGE MASTER TO  MASTER_HOST='192.168.0.10',MASTER_USER='repl',MASTER_PASSWORD='repl',MASTER_LOG_FILE='mysql-bin.000001',MASTER_LOG_POS=154; 

mysql> start slave;
Query OK, 0 rows affected (0.15 sec)

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.0.10
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 1344
               Relay_Log_File: relaylog.000002
                Relay_Log_Pos: 1510
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table: 

复制到这来就搭建完成了，然后配置192.168.0.10和192.168.0.20 ssh互信。（两台机器都执行）

ssh-keygen -t rsa
ssh-copy-id -i  /root/.ssh/id_rsa.pub  '-p 22 192.168.0.10'
ssh-copy-id -i  /root/.ssh/id_rsa.pub  '-p 22 192.168.0.20'

安装MHA软件，首先安装epel源。（2台机器）

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

安装依赖包

yum install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes -y

安装MHA软件（两台机器）

tar xf mha4mysql-node-0.56.tar.gz
cd mha4mysql-node-0.56
perl Makefile.PL
make && make install

tar xf mha4mysql-manager-0.56.tar.gz
cd mha4mysql-manager-0.56
perl Makefile.PL
make && make install

在从库（192.168.0.20）创建目录：

mkdir /data/mha//log
cd /data/mha//touch mha.cnf

mha.cnf配置文件内容如下：

[server default]
client_bindir=/usr/local/mysql/bin/
manager_log=/data/mha//log/manager.log
manager_workdir=/data/mha//log
master_binlog_dir=/data/mysql//binlog/
master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script= /usr/local/bin/master_ip_online_change
report_script=/usr/local/bin/send_report
init_conf_load_script=/usr/local/bin/load_cnf
remote_workdir=/data/mysql//mysqltmp
#secondary_check_script= /usr/local/bin/masterha_secondary_check -s 192.168.0.30 -s 192.168.0.40
user=root
ping_interval=
repl_user=repl
ssh_port=
ssh_user=root
max_ping_errors=

[server1]
hostname=192.168.0.10
port=

[server2]
candidate_master=
check_repl_delay=
hostname=192.168.0.20
port=

编辑文件 /usr/local/bin/load_cnf 里面的密码修改成对应的密码

#!/usr/bin/perl

  print "password=123\n";
  print "repl_password=repl\n";

执行chek命令查看复制是否正常：

[root@dbserver-yayun- ]# masterha_check_repl --conf=/data/mha//mha.cnf
Mon Mar  ::  - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Mar  ::  - [info] Reading application default configuration from /data/mha//mha.cnf..
Mon Mar  ::  - [info] Updating application default configuration from /usr/local/bin/load_cnf..
Mon Mar  ::  - [info] Reading server configuration from /data/mha//mha.cnf..
Mon Mar  ::  - [info] Setting max_ping_errors to , ping_interval to .
Mon Mar  ::  - [info] MHA::MasterMonitor version 0.56.
Mon Mar  ::  - [info] GTID failover mode =
Mon Mar  ::  - [info] Dead Servers:
Mon Mar  ::  - [info] Alive Servers:
Mon Mar  ::  - [info]   192.168.0.10(192.168.0.10:)
Mon Mar  ::  - [info]   192.168.0.20(192.168.0.20:)
Mon Mar  ::  - [info] Alive Slaves:
Mon Mar  ::  - [info]   192.168.0.20(192.168.0.20:)  Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Mon Mar  ::  - [info]     Replicating from 192.168.0.10(192.168.0.10:)
Mon Mar  ::  - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Mar  ::  - [info] Current Alive Master: 192.168.0.10(192.168.0.10:)
Mon Mar  ::  - [info] Checking slave configurations..
Mon Mar  ::  - [info]  read_only= is not set on slave 192.168.0.20(192.168.0.20:).
Mon Mar  ::  - [warning]  relay_log_purge= is not set on slave 192.168.0.20(192.168.0.20:).
Mon Mar  ::  - [info] Checking replication filtering settings..
Mon Mar  ::  - [info]  binlog_do_db= , binlog_ignore_db=
Mon Mar  ::  - [info]  Replication filtering check ok.
Mon Mar  ::  - [info] GTID (with auto-pos) is not supported
Mon Mar  ::  - [info] Starting SSH connection tests..
Mon Mar  ::  - [info] All SSH connection tests passed successfully.
Mon Mar  ::  - [info] Checking MHA Node version..
Mon Mar  ::  - [info]  Version check ok.
Mon Mar  ::  - [info] Checking SSH publickey authentication settings on the current master..
Mon Mar  ::  - [info] HealthCheck: SSH to 192.168.0.10 is reachable.
Mon Mar  ::  - [info] Master MHA Node version is 0.56.
Mon Mar  ::  - [info] Checking recovery script configurations on 192.168.0.10(192.168.0.10:)..
Mon Mar  ::  - [info]   Executing command: save_binary_logs --command=test --start_pos= --binlog_dir=/data/mysql//binlog/ --output_file=/data/mysql//mysqltmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.
Mon Mar  ::  - [info]   Connecting to root@192.168.0.10(192.168.0.10:)..
  Creating /data/mysql//mysqltmp if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /data/mysql//binlog/, up to mysql-bin.
Mon Mar  ::  - [info] Binlog setting check done.
Mon Mar  ::  - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Mon Mar  ::  - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.0.20 --slave_ip=192.168.0.20 --slave_port= --workdir=/data/mysql//mysqltmp --target_version=5.7.-log --manager_version=0.56 --client_bindir=/usr/local/mysql/bin/ --relay_dir=/data/mysql//relaylog --current_relay_log=relaylog.  --slave_pass=xxx
Mon Mar  ::  - [info]   Connecting to root@192.168.0.20(192.168.0.20:)..
  Checking slave recovery environment settings..
    Relay log found at /data/mysql//relaylog, up to relaylog.
    Temporary relay log file is /data/mysql//relaylog/relaylog.
    Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Mon Mar  ::  - [info] Slaves settings check done.
Mon Mar  ::  - [info]
192.168.0.10(192.168.0.10:) (current master)
 +--192.168.0.20(192.168.0.20:)

Mon Mar  ::  - [info] Checking replication health on 192.168.0.20..
Mon Mar  ::  - [info]  ok.
Mon Mar  ::  - [info] Checking master_ip_failover_script status:
Mon Mar  ::  - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.0.10 --orig_master_ip=192.168.0.10 --orig_master_port= 

IN SCRIPT TEST====/sbin/ifconfig eth1: down==/sbin/ifconfig eth1: 192.168.0.88/===

Checking the Status of the script.. OK
Mon Mar  ::  - [info]  OK.
Mon Mar  ::  - [warning] shutdown_script is not defined.
Mon Mar  ::  - [info] Got exit code  (Not master dead).

MySQL Replication Health is OK.

主库（192.168.0.10）执行命令，启动vip：

/sbin/ifconfig eth1: 192.168.0.88/

在线切换，把主库切到192.168.0.20

masterha_master_switch --master_state=alive --conf=/data/mha//mha.cnf --new_master_host=192.168.0.20 --new_master_port= --orig_master_is_new_slave

输出如下：

Mon Mar  ::  - [info] MHA::MasterRotate version 0.56.
Mon Mar  ::  - [info] Starting online master switch..
Mon Mar  ::  - [info]
Mon Mar  ::  - [info] * Phase : Configuration Check Phase..
Mon Mar  ::  - [info]
Mon Mar  ::  - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Mar  ::  - [info] Reading application default configuration from /data/mha//mha.cnf..
Mon Mar  ::  - [info] Updating application default configuration from /usr/local/bin/load_cnf..
Mon Mar  ::  - [info] Reading server configuration from /data/mha//mha.cnf..
Mon Mar  ::  - [info] Setting max_ping_errors to , ping_interval to .
Mon Mar  ::  - [info] GTID failover mode =
Mon Mar  ::  - [info] Current Alive Master: 192.168.0.10(192.168.0.10:)
Mon Mar  ::  - [info] Alive Slaves:
Mon Mar  ::  - [info]   192.168.0.20(192.168.0.20:)  Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Mon Mar  ::  - [info]     Replicating from 192.168.0.10(192.168.0.10:)
Mon Mar  ::  - [info]     Primary candidate for the new Master (candidate_master is set)

It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.0.10(192.168.0.10:)? (YES/no): yes
Mon Mar  ::  - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Mon Mar  ::  - [info]  ok.
Mon Mar  ::  - [info] Checking MHA is not monitoring or doing failover..
Mon Mar  ::  - [info] Checking replication health on 192.168.0.20..
Mon Mar  ::  - [info]  ok.
Mon Mar  ::  - [info] 192.168.0.20 can be new master.
Mon Mar  ::  - [info]
From:
192.168.0.10(192.168.0.10:) (current master)
 +--192.168.0.20(192.168.0.20:)

To:
192.168.0.20(192.168.0.20:) (new master)
 +--192.168.0.10(192.168.0.10:)

Starting master switch from 192.168.0.10(192.168.0.10:) to 192.168.0.20(192.168.0.20:)? (yes/NO): yes
Mon Mar  ::  - [info] Checking whether 192.168.0.20(192.168.0.20:) is ok for the new master..
Mon Mar  ::  - [info]  ok.
Mon Mar  ::  - [info] 192.168.0.10(192.168.0.10:): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.
Mon Mar  ::  - [info] 192.168.0.10(192.168.0.10:): Resetting slave pointing to the dummy host.
Mon Mar  ::  - [info] ** Phase : Configuration Check Phase completed.
Mon Mar  ::  - [info]
Mon Mar  ::  - [info] * Phase : Rejecting updates Phase..
Mon Mar  ::  - [info]
Mon Mar  ::  - [info] Executing master ip online change script to disable write on the current master:
Mon Mar  ::  - [info]   /usr/local/bin/master_ip_online_change --command=stop --orig_master_host=192.168.0.10 --orig_master_ip=192.168.0.10 --orig_master_port= --orig_master_user='root' --orig_master_password='' --new_master_host=192.168.0.20 --new_master_ip=192.168.0.20 --new_master_port= --new_master_user='root' --new_master_password='' --orig_master_ssh_user=root --new_master_ssh_user=root   --orig_master_is_new_slave
Mon Mar  ::   Set read_only on the new master.. ok.
Mon Mar  ::   Drpping app user on the orig master..
Mon Mar  ::   Set read_only= on the orig master.. ok.
Mon Mar  ::   Killing all application threads..
Mon Mar  ::   done.
Mon Mar  ::  - [info]  ok.
Mon Mar  ::  - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Mon Mar  ::  - [info] Executing FLUSH TABLES WITH READ LOCK..
Mon Mar  ::  - [info]  ok.
Mon Mar  ::  - [info] Orig master binlog:pos is mysql-bin.:.
Mon Mar  ::  - [info]  Waiting to execute all relay logs on 192.168.0.20(192.168.0.20:)..
Mon Mar  ::  - [info]  master_pos_wait(mysql-bin.:) completed on 192.168.0.20(192.168.0.20:). Executed  events.
Mon Mar  ::  - [info]   done.
Mon Mar  ::  - [info] Getting new master's binlog name and position..
Mon Mar  ::  - [info]  mysql-bin.:
Mon Mar  ::  - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.0.20', MASTER_PORT=, MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Mon Mar  ::  - [info] Executing master ip online change script to allow write on the new master:
Mon Mar  ::  - [info]   /usr/local/bin/master_ip_online_change --command=start --orig_master_host=192.168.0.10 --orig_master_ip=192.168.0.10 --orig_master_port= --orig_master_user='root' --orig_master_password='' --new_master_host=192.168.0.20 --new_master_ip=192.168.0.20 --new_master_port= --new_master_user='root' --new_master_password='' --orig_master_ssh_user=root --new_master_ssh_user=root   --orig_master_is_new_slave
Mon Mar  ::   Set read_only= on the new master.
Mon Mar  ::   Creating app user on the new master..
Mon Mar  ::  - [info]  ok.
Mon Mar  ::  - [info]
Mon Mar  ::  - [info] * Switching slaves in parallel..
Mon Mar  ::  - [info]
Mon Mar  ::  - [info] Unlocking all tables on the orig master:
Mon Mar  ::  - [info] Executing UNLOCK TABLES..
Mon Mar  ::  - [info]  ok.
Mon Mar  ::  - [info] Starting orig master as a new slave..
Mon Mar  ::  - [info]  Resetting slave 192.168.0.10(192.168.0.10:) and starting replication from the new master 192.168.0.20(192.168.0.20:)..
Mon Mar  ::  - [info]  Executed CHANGE MASTER.
Mon Mar  ::  - [info]  Slave started.
Mon Mar  ::  - [info] All new slave servers switched successfully.
Mon Mar  ::  - [info]
Mon Mar  ::  - [info] * Phase : New master cleanup phase..
Mon Mar  ::  - [info]
Mon Mar  ::  - [info]  192.168.0.20: Resetting slave info succeeded.
Mon Mar  ::  - [info] Switching master to 192.168.0.20(192.168.0.20:) completed successfully.

关于配置文件中的参数：
max_ping_errors=40，这个是修改了源码，增加了检测次数的定义，默认是3次，太容易误切换。

启动管理进程：

/usr/bin/nohup /usr/local/bin/masterha_manager --conf=/data/mha//mha.cnf --ignore_last_failover > /data/mha//log/manager.log >& &

在主库（192.168.0.10）封掉ip，可以看到日志输出。

iptables -I INPUT -s 192.168.0.20 -j DROP

Mon Mar  ::  - [warning] Got error on MySQL connect:  (Can't connect to MySQL server on '192.168.0.10' (4))
Mon Mar  ::  - [warning] Connection failed  time(s)..
Mon Mar  ::  - [warning] Got error on MySQL connect:  (Can't connect to MySQL server on '192.168.0.10' (4))
Mon Mar  ::  - [warning] Connection failed  time(s)..
Mon Mar  ::  - [warning] Got error on MySQL connect:  (Can't connect to MySQL server on '192.168.0.10' (4))
Mon Mar  ::  - [warning] Connection failed  time(s)..
Mon Mar  ::  - [warning] Got error on MySQL connect:  (Can't connect to MySQL server on '192.168.0.10' (4))
Mon Mar  ::  - [warning] Connection failed  time(s)..
Mon Mar  ::  - [warning] Got error on MySQL connect:  (Can't connect to MySQL server on '192.168.0.10' (4))
Mon Mar  ::  - [warning] Connection failed  time(s)..
Mon Mar  ::  - [warning] Got error on MySQL connect:  (Can't connect to MySQL server on '192.168.0.10' (4))
Mon Mar  ::  - [warning] Connection failed  time(s)..
Mon Mar  ::  - [warning] Got error on MySQL connect:  (Can't connect to MySQL server on '192.168.0.10' (4))
Mon Mar  ::  - [warning] Connection failed  time(s)..
Mon Mar  ::  - [warning] Got error on MySQL connect:  (Can't connect to MySQL server on '192.168.0.10' (4))
Mon Mar  ::  - [warning] Connection failed  time(s)..
Mon Mar  ::  - [warning] Got error on MySQL connect:  (Can't connect to MySQL server on '192.168.0.10' (4))
Mon Mar  ::  - [warning] Connection failed  time(s)..
Mon Mar  ::  - [warning] Got error on MySQL connect:  (Can't connect to MySQL server on '192.168.0.10' (4))

修改以后的软件包下载地址：链接：http://pan.baidu.com/s/1c3qg70 密码：f9u2