mysql mha高可用架构的安装

MMM无法全然地保证数据的一致性，所以MMM适用于对数据的一致性要求不是非常高。可是又想最大程度的保证业务可用性的场景对于那些对数据一致性要求非常高的业务，非常不建议採用MMM的这样的高可用性架构。那么可以考虑使用MHA。在mysql故障切换的过程中。MHA可以在0-30s内自己主动完毕数据库的故障切换操作，而且MHA可以最大程度上保证数据的一致性，以达到真正意义上的高可用。

MHA软件由两部分组成，Manager工具包和Node工具包。详细的说明例如以下。

Manager工具包主要包含下面几个工具：

masterha_check_ssh              检查MHA的SSH配置状况

masterha_check_repl             检查MySQL复制状况

masterha_manger                 启动MHA

masterha_check_status           检測当前MHA执行状态

masterha_master_monitor         检測master是否宕机

masterha_master_switch          控制故障转移（自己主动或者手动）

masterha_conf_host              加入或删除配置的server信息

Node工具包（这些工具通常由MHA Manager的脚本触发。无需人为操作）主要包含下面几个工具：

save_binary_logs                保存和复制master的二进制日志

apply_diff_relay_logs           识别差异的中继日志事件并将其差异的事件应用于其它的slave

filter_mysqlbinlog              去除不必要的ROLLBACK事件（MHA已不再使用这个工具）

purge_relay_logs                清除中继日志（不会堵塞SQL线程）

注意：

（1）为了尽可能的降低主库硬件损坏宕机造成的数据丢失。因此在配置MHA的同一时候建议配置成MySQL 5.5的半同步复制

1.1、搭建好开发环境

<img src="http://img.blog.csdn.net/20160324175821383?

watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQv/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" />

1.2、yaolansvr安装ftp服务，并上传mha安装软件

（1）关闭selinux，否则vsftpd报226错误

# mkdir -p /yangsq/ftp

# useradd -d /yangsq/ftp -s /sbin/nologin uftp

# passwd uftp

# chown -R uftp:uftp /yangsq/ftp

# yum list all|grep vsftpd

# yum -y install vsftpd.x86_64

# cp /etc/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.conf.bak

anonymous_enable=YES改为anonymous_enable=NO

local_enable=YES

write_enable=YES

chroot_local_user=YES

# chkconfig vsftpd on

# service vsftpd start

# yum install ftp.x86_64 -y

# sestatus

SELinux status:                 enabled

SELinuxfs mount:                /selinux

Current mode:                   enforcing

Mode from config file:          enforcing

Policy version:                 24

Policy from config file:        targeted

# setenforce 0

# ftp 192.168.0.3 21

1.3、全部数据库节点安装Perl模块，同一时候做

yum -y install perl-DBD-MySQL

yum -y install perl-CPAN.x86_64

cd /yangsq/ftp

tar xvf mha4mysql-node-0.54.tar.gz

cd mha4mysql-node-0.54

perl Makefile.PL

make && make install

Installing /usr/local/share/perl5/MHA/BinlogPosFinderXid.pm

Installing /usr/local/share/perl5/MHA/SlaveUtil.pm

Installing /usr/local/share/perl5/MHA/NodeUtil.pm

Installing /usr/local/share/perl5/MHA/BinlogPosFinderElp.pm

Installing /usr/local/share/perl5/MHA/BinlogPosFindManager.pm

Installing /usr/local/share/perl5/MHA/BinlogPosFinder.pm

Installing /usr/local/share/perl5/MHA/BinlogManager.pm

Installing /usr/local/share/perl5/MHA/NodeConst.pm

Installing /usr/local/share/perl5/MHA/BinlogHeaderParser.pm

Installing /usr/local/share/man/man1/filter_mysqlbinlog.1

Installing /usr/local/share/man/man1/purge_relay_logs.1

Installing /usr/local/share/man/man1/apply_diff_relay_logs.1

Installing /usr/local/share/man/man1/save_binary_logs.1

Installing /usr/local/bin/filter_mysqlbinlog

Installing /usr/local/bin/apply_diff_relay_logs

Installing /usr/local/bin/save_binary_logs

Installing /usr/local/bin/purge_relay_logs

1.4、在yaolansvr_slave安装mha manager

（1）No package perl-Log-Dispatch available.

# mv CentOS-Base.repo CentOS-Base.repo.bak

sftp> put C:\Users\Yaolan\Downloads\CentOS6-Base-163.repo

# yum clean all

# yum makecache

或者

# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

yum -y install perl-Config-Tiny

yum install perl-Log-Dispatch -y

yum install perl-Parallel-ForkManager -y

yum install perl-Time-HiRes -y

# tar xvf mha4mysql-manager-0.54.tar.gz

# cd mha4mysql-manager-0.54

# perl Makefile.PL 

# make && make install

Installing /usr/local/share/perl5/MHA/ManagerAdminWrapper.pm

Installing /usr/local/share/perl5/MHA/ManagerUtil.pm

Installing /usr/local/share/perl5/MHA/MasterFailover.pm

Installing /usr/local/share/perl5/MHA/MasterMonitor.pm

Installing /usr/local/share/perl5/MHA/ManagerAdmin.pm

Installing /usr/local/share/perl5/MHA/Config.pm

Installing /usr/local/share/perl5/MHA/DBHelper.pm

Installing /usr/local/share/perl5/MHA/HealthCheck.pm

Installing /usr/local/share/perl5/MHA/FileStatus.pm

Installing /usr/local/share/perl5/MHA/MasterRotate.pm

Installing /usr/local/share/perl5/MHA/Server.pm

Installing /usr/local/share/perl5/MHA/ServerManager.pm

Installing /usr/local/share/perl5/MHA/SSHCheck.pm

Installing /usr/local/share/perl5/MHA/ManagerConst.pm

Installing /usr/local/share/man/man1/masterha_check_ssh.1

Installing /usr/local/share/man/man1/masterha_secondary_check.1

Installing /usr/local/share/man/man1/masterha_conf_host.1

Installing /usr/local/share/man/man1/masterha_check_status.1

Installing /usr/local/share/man/man1/masterha_stop.1

Installing /usr/local/share/man/man1/masterha_manager.1

Installing /usr/local/share/man/man1/masterha_master_monitor.1

Installing /usr/local/share/man/man1/masterha_check_repl.1

Installing /usr/local/share/man/man1/masterha_master_switch.1

Installing /usr/local/bin/masterha_manager

Installing /usr/local/bin/masterha_check_ssh

Installing /usr/local/bin/masterha_check_status

Installing /usr/local/bin/masterha_master_monitor

Installing /usr/local/bin/masterha_secondary_check

Installing /usr/local/bin/masterha_conf_host

Installing /usr/local/bin/masterha_check_repl

Installing /usr/local/bin/masterha_stop

Installing /usr/local/bin/masterha_master_switch

1.5、全部mysql服务器配置ssh登录无密码验证，sshd_config已改不生效，万能的重新启动

注意：

（1）ssh-copy-id: command not found，解决：yum install openssh-clients -y

（2）对自己本身也要运行ssh-copy-id

（3）关闭slave后重新启动mysql实例，slave相关的进程正常启动

# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa)

# yum install -y openssh-clients

# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.0.3

# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.0.4

# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.0.5

1.6、搭建主从复制环境

yaolansvr服务器master节点-----------

注意：

（1）datadir和server-id的设置，candidate master和slave仅仅改动server-id

（2）

# vi /usr/mysql/etc/my.cnf

[mysqld]

port = 3306

datadir=/data/mysql/data

#慢查询设置

slow-query-log-file=/var/log/MysqlQuery.log

long_query_time =5

slow_query_log=1

#server-id

server-id=16803

#binlog 设置

log-bin = /data/mysql/data/mysql-bin.log

binlog_cache_size = 8M

binlog_format=mixed

#全局

join_buffer_size = 2M

sort_buffer_size = 2M

read_rnd_buffer_size = 2M

read_buffer_size = 2M

max_heap_table_size = 64M

thread_cache_size=12

thread_concurrency = 12

query_cache_type = 1

query_cache_size = 32M

ft_min_word_len = 4

thread_stack = 192K

tmp_table_size = 64M

#myisam内存设置

key_buffer_size=1024M

#同意最大的复制传输

max_allowed_packet=64M

#跳过dns解析

skip-name-resolve

#连接数设置

max_connections = 1000

max_connect_errors = 200

#innodb设置

innodb_buffer_pool_size = 1G

innodb_additional_mem_pool_size = 16M

innodb_log_buffer_size = 8M

innodb_log_file_size = 512M

innodb_log_files_in_group = 3

innodb_file_per_table=1

innodb_stats_persistent_sample_pages=1000

innodb_write_io_threads = 8

innodb_read_io_threads = 8

innodb_thread_concurrency = 16

innodb_flush_log_at_trx_commit = 2

innodb_lock_wait_timeout = 30

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

# service mysqld start 

1.6.1、在yaolansvr做备份，并创建复制用户

# mysqldump  -A --flush-privileges --lock-all-tables --events --routines --triggers --master-data=2>/yangsq/ftp/`date +%Y-%m-%d`_all.sql

mysql> grant replication slave on *.* to 'repl1'@'192.168.0.%' identified by '123456';

mysql> flush privileges;

1.6.2、使用ftp从yaolansvr获取整库的备份。并恢复

# mysql<2015-06-29_all.sql

# head -n 30 2015-06-29_all.sql |grep -i "change master"

-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000008', MASTER_LOG_POS=120;

mysql> show slave status;

Empty set (0.00 sec)

mysql> change master to master_host='192.168.0.3',master_user='repl1',master_password='123456',master_port=3306,master_log_file='mysql-bin.000008',master_log_pos=120;

mysql> start slave;

Last_IO_Errno: 1593

Last_IO_Error: Fatal error: The slave I/O thread stops because master and slave have equal MySQL server UUIDs; these UUIDs must be different for replication to work.

解决：两台datadir/auto.cnf一样，select uuid()不同。所以删除candidate master上的auto.cnf。又一次启动实例

重新启动mysql实例后：

mysql> show slave status\G

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 192.168.0.3

                  Master_User: repl1

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000008

          Read_Master_Log_Pos: 408

               Relay_Log_File: yaolansvr_slave-relay-bin.000003

                Relay_Log_Pos: 571

        Relay_Master_Log_File: mysql-bin.000008

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

1.6.3、其它的slave节点设置read_only=1（不写入my.cnf。以供candidate master转为主后，提供写）

# mysql -e "set global read_only=1"

1.6.4、注意******************全部数据库节点都要创建复制用户

（1）candidate master不创建复制用户。则报错：

Mon Jun 29 17:28:00 2015 - [info] Alive Slaves:

Mon Jun 29 17:28:00 2015 - [info]   192.168.0.4(192.168.0.4:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Mon Jun 29 17:28:00 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Mon Jun 29 17:28:00 2015 - [info]     Primary candidate for the new Master (candidate_master is set)

Mon Jun 29 17:28:00 2015 - [info]   192.168.0.5(192.168.0.5:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Mon Jun 29 17:28:00 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Mon Jun 29 17:28:00 2015 - [info]     Not candidate for the new Master (no_master is set)

Mon Jun 29 17:28:00 2015 - [info] Current Alive Master: 192.168.0.3(192.168.0.3:3306)

Mon Jun 29 17:28:00 2015 - [info] Checking slave configurations..

Mon Jun 29 17:28:00 2015 - [info] Checking replication filtering settings..

Mon Jun 29 17:28:00 2015 - [info]  binlog_do_db= , binlog_ignore_db=

Mon Jun 29 17:28:00 2015 - [info]  Replication filtering check ok.

Mon Jun 29 17:28:00 2015 - [error][/usr/local/share/perl5/MHA/Server.pm, ln382] 192.168.0.4(192.168.0.4:3306): User repl1 does not exist or does not have REPLICATION SLAVE privilege! Other slaves can not start replication from this host.

（2）假设一个数据库节点不会成为备选master，且no_master=1，则不须要创建复制用户

仅仅在candidate master创建复制用户，必须与master的复制用户同样

mysql> grant replication slave on *.* to 'repl1'@'192.168.0.%' identified by '123456';

mysql> flush privileges;

1.6.5、注意******************全部数据库节点都要创建监控用户，监控用户必须要

仅仅在master上创建监控用户：可是会被复制

（1）其它节点必须创建监控用户，否则报错：

Mon Jun 29 18:02:41 2015 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln255] Got MySQL error when connecting 192.168.0.4(192.168.0.4:3306) :1045:Access denied for user 'monitor'@'192.168.0.4' (using password: YES), but this is not mysql crash. Check MySQL server settings.

mysql> grant all privileges on *.* to 'monitor'@'192.168.0.%' identified by '123456';

mysql> flush privileges;

1.7、配置MHA

（1）master_binlog_dir=/home/soft/mysql/3306/binlog必须是binlog所在文件夹

（2）假设一个数据库节点不会成为备选master。且no_master=1，则不须要创建复制用户

# mkdir -p /etc/mha/app1

# vi /etc/mha/app1/app1.cnf

[server default]

manager_workdir=/etc/mha/app1

manager_log=/var/log/manager.log

master_binlog_dir=/data/mysql/data/

ssh_user=root ####ssh_user是互信配置用户。就用这个账户运行mhamaster命令。原理：ssh roo@192.168.0.4

user=monitor

password=123456

repl_user=repl1

repl_password=123456

secondary_check_script=masterha_secondary_check -s 192.168.0.3 -s 192.168.0.5

ping_interval=3

#master_ip_failover_script=/etc/mha/app1/master_ip_failover

#shutdown_script=/script/masterha/power_manager

#report_script=/script/masterha/send_report

#master_ip_online_change_script=/etc/mha/master_ip_failover

[server1]

hostname=192.168.0.3

port=3306

#master_binlog_dir=/data/mysql/data

candidate_master=1

[server2]

hostname=192.168.0.4

port=3306

#master_binlog_dir=/data/mysql/data

candidate_master=1

[server3]

hostname=192.168.0.5

port=3306

no_master=1

1.7.2、关闭全部数据库节点 relay log的自己主动清除

# mysql -e "set global relay_log_purge=0"

1.7.3、全部数据库节点在环境变量中输出mysqlbinlog

# ln -sv /usr/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog

1.7.4、检查ssh的配置

注意：

（1）# cp /yangsq/ftp/mha4mysql-manager-0.54/samples/scripts/master_ip_failover /etc/mha/app1/master_ip_failover或将master_ip_failover_script=/etc/mha/app1/master_ip_failover凝视掉

否则报错：Mon Jun 29 17:05:26 2015 - [info]   /etc/mha/app1/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.0.3 --orig_master_ip=192.168.0.3 --orig_master_port=3306

Bareword "FIXME_xxx" not allowed while "strict subs" in use at /etc/mha/app1/master_ip_failover line 93.

（2）没有mysqlbinlog，报错：

Mon Jun 29 17:52:11 2015 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/var/tmp/save_binary_logs_test --manager_version=0.54 --start_file=mysql-bin.000008

Mon Jun 29 17:52:11 2015 - [info]   Connecting to root@192.168.0.3(192.168.0.3)..

Failed to save binary log: Binlog not found from /var/lib/mysql,/var/log/mysql! If you got this error at MHA Manager, please set "master_binlog_dir=/path/to/binlog_directory_of_the_master" correctly in the MHA Manager's configuration file and try again.

# masterha_check_ssh --conf=/etc/mha/app1/app1.cnf 

# masterha_check_ssh --conf=/etc/mha/app1/app1.cnf

Mon Jun 29 17:33:39 2015 - [info] Reading server configurations from /etc/mha/app1/app1.cnf..

Mon Jun 29 17:33:39 2015 - [info] MHA::MasterMonitor version 0.54.

Mon Jun 29 17:33:39 2015 - [info] Dead Servers:

Mon Jun 29 17:33:39 2015 - [info] Alive Servers:

Mon Jun 29 17:33:39 2015 - [info]   192.168.0.3(192.168.0.3:3306)

Mon Jun 29 17:33:39 2015 - [info]   192.168.0.4(192.168.0.4:3306)

Mon Jun 29 17:33:39 2015 - [info]   192.168.0.5(192.168.0.5:3306)

Mon Jun 29 17:33:39 2015 - [info] Alive Slaves:

Mon Jun 29 17:33:39 2015 - [info]   192.168.0.4(192.168.0.4:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Mon Jun 29 17:33:39 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Mon Jun 29 17:33:39 2015 - [info]     Primary candidate for the new Master (candidate_master is set)

Mon Jun 29 17:33:39 2015 - [info]   192.168.0.5(192.168.0.5:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Mon Jun 29 17:33:39 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Mon Jun 29 17:33:39 2015 - [info]     Not candidate for the new Master (no_master is set)

Mon Jun 29 17:33:39 2015 - [info] Current Alive Master: 192.168.0.3(192.168.0.3:3306)

Mon Jun 29 17:33:39 2015 - [info] Checking slave configurations..

Mon Jun 29 17:33:39 2015 - [info] Checking replication filtering settings..

Mon Jun 29 17:33:39 2015 - [info]  binlog_do_db= , binlog_ignore_db=

Mon Jun 29 17:33:39 2015 - [info]  Replication filtering check ok.

Mon Jun 29 17:33:39 2015 - [info] Starting SSH connection tests..

Mon Jun 29 17:34:21 2015 - [info] All SSH connection tests passed successfully.

Mon Jun 29 17:34:21 2015 - [info] Checking MHA Node version..

Mon Jun 29 17:34:31 2015 - [info]  Version check ok.

Mon Jun 29 17:34:31 2015 - [info] Checking SSH publickey authentication settings on the current master..

Mon Jun 29 17:34:32 2015 - [info] HealthCheck: SSH to 192.168.0.3 is reachable.

Mon Jun 29 17:34:32 2015 - [info] Master MHA Node version is 0.54.

Mon Jun 29 17:34:32 2015 - [info] Checking recovery script configurations on the current master..

Mon Jun 29 17:34:32 2015 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/data --output_file=/var/tmp/save_binary_logs_test --manager_version=0.54 --start_file=mysql-bin.000008

Mon Jun 29 17:34:32 2015 - [info]   Connecting to root@192.168.0.3(192.168.0.3)..

  Creating /var/tmp if not exists..    ok.

  Checking output directory is accessible or not..

   ok.

  Binlog found at /data/mysql/data, up to mysql-bin.000008

Mon Jun 29 17:34:32 2015 - [info] Master setting check done.

Mon Jun 29 17:34:32 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Mon Jun 29 17:34:32 2015 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=192.168.0.4 --slave_ip=192.168.0.4 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.20-log --manager_version=0.54 --relay_log_info=/data/mysql/data/relay-log.info  --relay_dir=/data/mysql/data/  --slave_pass=xxx

Mon Jun 29 17:34:32 2015 - [info]   Connecting to root@192.168.0.4(192.168.0.4:22)..

  Checking slave recovery environment settings..

    Opening /data/mysql/data/relay-log.info ... ok.

    Relay log found at /data/mysql/data, up to yaolansvr_slave-relay-bin.000003

    Temporary relay log file is /data/mysql/data/yaolansvr_slave-relay-bin.000003

    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Mon Jun 29 17:34:33 2015 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=192.168.0.5 --slave_ip=192.168.0.5 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.20-log --manager_version=0.54 --relay_log_info=/data/mysql/data/relay-log.info  --relay_dir=/data/mysql/data/  --slave_pass=xxx

Mon Jun 29 17:34:33 2015 - [info]   Connecting to root@192.168.0.5(192.168.0.5:22)..

reverse mapping checking getaddrinfo for bogon [192.168.0.5] failed - POSSIBLE BREAK-IN ATTEMPT!

  Checking slave recovery environment settings..

    Opening /data/mysql/data/relay-log.info ... ok.

    Relay log found at /data/mysql/data, up to yaolansvr_slave01-relay-bin.000002

    Temporary relay log file is /data/mysql/data/yaolansvr_slave01-relay-bin.000002

    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Mon Jun 29 17:34:43 2015 - [info] Slaves settings check done.

Mon Jun 29 17:34:43 2015 - [info]

192.168.0.3 (current master)

 +--192.168.0.4

 +--192.168.0.5

Mon Jun 29 17:34:43 2015 - [info] Checking replication health on 192.168.0.4..

Mon Jun 29 17:34:43 2015 - [info]  ok.

Mon Jun 29 17:34:43 2015 - [info] Checking replication health on 192.168.0.5..

Mon Jun 29 17:34:43 2015 - [info]  ok.

Mon Jun 29 17:34:43 2015 - [warning] master_ip_failover_script is not defined.

Mon Jun 29 17:34:43 2015 - [warning] shutdown_script is not defined.

Mon Jun 29 17:34:43 2015 - [info] Got exit code 0 (Not master dead).

1.8、vip配置

为了防止脑裂发生，推荐生产环境採用脚本的方式来管理虚拟ip，而不是使用keepalived来完毕

1.8.1、改动故障切换的脚本

# vi /etc/mha/app1/master_ip_failover

####加入变量

my $vip = '192.168.0.10/24';

my $key = '1';

my $ssh_start_vip = "/sbin/ifconfig eth1:$key $vip";

my $ssh_stop_vip = "/sbin/ifconfig eth1:$key down";

####首先主节点须要启动vip

# /sbin/ifconfig eth0:1 192.168.0.10/24

# /sbin/ifconfig eth0:1 down

# ip a

####然后检查复制环境状况

# masterha_check_repl --conf=/etc/mha/app1/app1.cnf 

# masterha_check_status --conf=/etc/mha/app1/app1.cnf

app1 is stopped(2:NOT_RUNNING).

# touch /etc/mha/app1/manager.log

# nohup masterha_manager --conf=/etc/mha/app1/app1.cnf </dev/null>/etc/mha/app1/manager.log 2>&1   &

# tail -f /var/log/manager.log

Tue Jun 30 10:21:25 2015 - [info] Got terminate signal. Exit.

Tue Jun 30 10:21:58 2015 - [info] MHA::MasterMonitor version 0.54.

Tue Jun 30 10:21:58 2015 - [info] Dead Servers:

Tue Jun 30 10:21:58 2015 - [info] Alive Servers:

Tue Jun 30 10:21:58 2015 - [info]   192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:21:58 2015 - [info]   192.168.0.4(192.168.0.4:3306)

Tue Jun 30 10:21:58 2015 - [info]   192.168.0.5(192.168.0.5:3306)

Tue Jun 30 10:21:58 2015 - [info] Alive Slaves:

Tue Jun 30 10:21:58 2015 - [info]   192.168.0.4(192.168.0.4:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:21:58 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:21:58 2015 - [info]     Primary candidate for the new Master (candidate_master is set)

Tue Jun 30 10:21:58 2015 - [info]   192.168.0.5(192.168.0.5:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:21:58 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:21:58 2015 - [info]     Not candidate for the new Master (no_master is set)

Tue Jun 30 10:21:58 2015 - [info] Current Alive Master: 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:21:58 2015 - [info] Checking slave configurations..

Tue Jun 30 10:21:58 2015 - [info]  read_only=1 is not set on slave 192.168.0.4(192.168.0.4:3306).

Tue Jun 30 10:21:58 2015 - [warning]  relay_log_purge=0 is not set on slave 192.168.0.4(192.168.0.4:3306).

Tue Jun 30 10:21:58 2015 - [info]  read_only=1 is not set on slave 192.168.0.5(192.168.0.5:3306).

Tue Jun 30 10:21:58 2015 - [warning]  relay_log_purge=0 is not set on slave 192.168.0.5(192.168.0.5:3306).

Tue Jun 30 10:21:58 2015 - [info] Checking replication filtering settings..

Tue Jun 30 10:21:58 2015 - [info]  binlog_do_db= , binlog_ignore_db=

Tue Jun 30 10:21:58 2015 - [info]  Replication filtering check ok.

Tue Jun 30 10:21:58 2015 - [info] Starting SSH connection tests..

Tue Jun 30 10:22:00 2015 - [info] All SSH connection tests passed successfully.

Tue Jun 30 10:22:00 2015 - [info] Checking MHA Node version..

Tue Jun 30 10:22:01 2015 - [info]  Version check ok.

Tue Jun 30 10:22:01 2015 - [info] Checking SSH publickey authentication settings on the current master..

Tue Jun 30 10:22:01 2015 - [info] HealthCheck: SSH to 192.168.0.3 is reachable.

Tue Jun 30 10:22:01 2015 - [info] Master MHA Node version is 0.54.

Tue Jun 30 10:22:01 2015 - [info] Checking recovery script configurations on the current master..

Tue Jun 30 10:22:01 2015 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/data/ --output_file=/var/tmp/save_binary_logs_test --manager_version=0.54 --start_file=mysql-bin.000009

Tue Jun 30 10:22:01 2015 - [info]   Connecting to root@192.168.0.3(192.168.0.3)..

  Creating /var/tmp if not exists..    ok.

  Checking output directory is accessible or not..

   ok.

  Binlog found at /data/mysql/data/, up to mysql-bin.000009

Tue Jun 30 10:22:01 2015 - [info] Master setting check done.

Tue Jun 30 10:22:01 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Tue Jun 30 10:22:01 2015 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=192.168.0.4 --slave_ip=192.168.0.4 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.20-log --manager_version=0.54 --relay_log_info=/data/mysql/data/relay-log.info  --relay_dir=/data/mysql/data/  --slave_pass=xxx

Tue Jun 30 10:22:01 2015 - [info]   Connecting to root@192.168.0.4(192.168.0.4:22)..

  Checking slave recovery environment settings..

    Opening /data/mysql/data/relay-log.info ... ok.

    Relay log found at /data/mysql/data, up to yaolansvr_slave-relay-bin.000006

    Temporary relay log file is /data/mysql/data/yaolansvr_slave-relay-bin.000006

    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Tue Jun 30 10:22:02 2015 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=192.168.0.5 --slave_ip=192.168.0.5 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.20-log --manager_version=0.54 --relay_log_info=/data/mysql/data/relay-log.info  --relay_dir=/data/mysql/data/  --slave_pass=xxx

Tue Jun 30 10:22:02 2015 - [info]   Connecting to root@192.168.0.5(192.168.0.5:22)..

  Checking slave recovery environment settings..

    Opening /data/mysql/data/relay-log.info ... ok.

    Relay log found at /data/mysql/data, up to yaolansvr_slave01-relay-bin.000005

    Temporary relay log file is /data/mysql/data/yaolansvr_slave01-relay-bin.000005

    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Tue Jun 30 10:22:02 2015 - [info] Slaves settings check done.

Tue Jun 30 10:22:02 2015 - [info]

192.168.0.3 (current master)

 +--192.168.0.4

 +--192.168.0.5

Tue Jun 30 10:22:02 2015 - [info] Checking master_ip_failover_script status:

Tue Jun 30 10:22:02 2015 - [info]   /etc/mha/app1/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.0.3 --orig_master_ip=192.168.0.3 --orig_master_port=3306

Tue Jun 30 10:22:02 2015 - [info]  OK.

Tue Jun 30 10:22:02 2015 - [warning] shutdown_script is not defined.

Tue Jun 30 10:22:02 2015 - [info] Set master ping interval 3 seconds.

Tue Jun 30 10:22:02 2015 - [info] Set secondary check script: masterha_secondary_check -s 192.168.0.3 -s 192.168.0.5

Tue Jun 30 10:22:02 2015 - [info] Starting ping health check on 192.168.0.3(192.168.0.3:3306)..

Tue Jun 30 10:22:02 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

# masterha_check_status --conf=/etc/mha/app1/app1.cnf

app1 (pid:2243) is running(0:PING_OK), master:192.168.0.3

# masterha_stop --conf=/etc/mha/app1/app1.cnf

进行切换后：

# tail -f /var/log/manager.log

Tue Jun 30 10:28:02 2015 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)

Tue Jun 30 10:28:02 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/data/ --output_file=/var/tmp/save_binary_logs_test --manager_version=0.54 --binlog_prefix=mysql-bin

Tue Jun 30 10:28:03 2015 - [info] Executing seconary network check script: masterha_secondary_check -s 192.168.0.3 -s 192.168.0.5  --user=root  --master_host=192.168.0.3  --master_ip=192.168.0.3  --master_port=3306

Tue Jun 30 10:28:03 2015 - [info] HealthCheck: SSH to 192.168.0.3 is reachable.

Monitoring server 192.168.0.3 is reachable, Master is not reachable from 192.168.0.3. OK.

Monitoring server 192.168.0.5 is reachable, Master is not reachable from 192.168.0.5. OK.

Tue Jun 30 10:28:03 2015 - [info] Master is not reachable from all other monitoring servers. Failover should start.

Tue Jun 30 10:28:05 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)

Tue Jun 30 10:28:05 2015 - [warning] Connection failed 1 time(s)..

Tue Jun 30 10:28:08 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)

Tue Jun 30 10:28:08 2015 - [warning] Connection failed 2 time(s)..

Tue Jun 30 10:28:11 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)

Tue Jun 30 10:28:11 2015 - [warning] Connection failed 3 time(s)..

Tue Jun 30 10:28:11 2015 - [warning] Master is not reachable from health checker!

Tue Jun 30 10:28:11 2015 - [warning] Master 192.168.0.3(192.168.0.3:3306) is not reachable!

Tue Jun 30 10:28:11 2015 - [warning] SSH is reachable.

Tue Jun 30 10:28:11 2015 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha/app1/app1.cnf again, and trying to connect to all servers to check server status..

Tue Jun 30 10:28:11 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Tue Jun 30 10:28:11 2015 - [info] Reading application default configurations from /etc/mha/app1/app1.cnf..

Tue Jun 30 10:28:11 2015 - [info] Reading server configurations from /etc/mha/app1/app1.cnf..

Tue Jun 30 10:28:12 2015 - [info] Dead Servers:

Tue Jun 30 10:28:12 2015 - [info]   192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:12 2015 - [info] Alive Servers:

Tue Jun 30 10:28:12 2015 - [info]   192.168.0.4(192.168.0.4:3306)

Tue Jun 30 10:28:12 2015 - [info]   192.168.0.5(192.168.0.5:3306)

Tue Jun 30 10:28:12 2015 - [info] Alive Slaves:

Tue Jun 30 10:28:12 2015 - [info]   192.168.0.4(192.168.0.4:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:28:12 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:12 2015 - [info]     Primary candidate for the new Master (candidate_master is set)

Tue Jun 30 10:28:12 2015 - [info]   192.168.0.5(192.168.0.5:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:28:12 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:12 2015 - [info]     Not candidate for the new Master (no_master is set)

Tue Jun 30 10:28:12 2015 - [info] Checking slave configurations..

Tue Jun 30 10:28:12 2015 - [warning]  relay_log_purge=0 is not set on slave 192.168.0.4(192.168.0.4:3306).

Tue Jun 30 10:28:12 2015 - [warning]  relay_log_purge=0 is not set on slave 192.168.0.5(192.168.0.5:3306).

Tue Jun 30 10:28:12 2015 - [info] Checking replication filtering settings..

Tue Jun 30 10:28:12 2015 - [info]  Replication filtering check ok.

Tue Jun 30 10:28:12 2015 - [info] Master is down!

Tue Jun 30 10:28:12 2015 - [info] Terminating monitoring script.

Tue Jun 30 10:28:12 2015 - [info] Got exit code 20 (Master dead).

Tue Jun 30 10:28:12 2015 - [info] MHA::MasterFailover version 0.54.

Tue Jun 30 10:28:12 2015 - [info] Starting master failover.

Tue Jun 30 10:28:12 2015 - [info]

Tue Jun 30 10:28:12 2015 - [info] * Phase 1: Configuration Check Phase..

Tue Jun 30 10:28:12 2015 - [info]

Tue Jun 30 10:28:12 2015 - [info] Dead Servers:

Tue Jun 30 10:28:12 2015 - [info]   192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:12 2015 - [info] Checking master reachability via mysql(double check)..

Tue Jun 30 10:28:12 2015 - [info]  ok.

Tue Jun 30 10:28:12 2015 - [info] Alive Servers:

Tue Jun 30 10:28:12 2015 - [info]   192.168.0.4(192.168.0.4:3306)

Tue Jun 30 10:28:12 2015 - [info]   192.168.0.5(192.168.0.5:3306)

Tue Jun 30 10:28:12 2015 - [info] Alive Slaves:

Tue Jun 30 10:28:12 2015 - [info]   192.168.0.4(192.168.0.4:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:28:12 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:12 2015 - [info]     Primary candidate for the new Master (candidate_master is set)

Tue Jun 30 10:28:12 2015 - [info]   192.168.0.5(192.168.0.5:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:28:12 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:12 2015 - [info]     Not candidate for the new Master (no_master is set)

Tue Jun 30 10:28:12 2015 - [info] ** Phase 1: Configuration Check Phase completed.

Tue Jun 30 10:28:12 2015 - [info]

Tue Jun 30 10:28:12 2015 - [info] * Phase 2: Dead Master Shutdown Phase..

Tue Jun 30 10:28:12 2015 - [info]

Tue Jun 30 10:28:12 2015 - [info] Forcing shutdown so that applications never connect to the current master..

Tue Jun 30 10:28:12 2015 - [info] Executing master IP deactivatation script:

Tue Jun 30 10:28:12 2015 - [info]   /etc/mha/app1/master_ip_failover --orig_master_host=192.168.0.3 --orig_master_ip=192.168.0.3 --orig_master_port=3306 --command=stopssh --ssh_user=root

Disabling the VIP on old master: 192.168.0.3

Tue Jun 30 10:28:13 2015 - [info]  done.

Tue Jun 30 10:28:13 2015 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.

Tue Jun 30 10:28:13 2015 - [info] * Phase 2: Dead Master Shutdown Phase completed.

Tue Jun 30 10:28:13 2015 - [info]

Tue Jun 30 10:28:13 2015 - [info] * Phase 3: Master Recovery Phase..

Tue Jun 30 10:28:13 2015 - [info]

Tue Jun 30 10:28:13 2015 - [info] * Phase 3.1: Getting Latest Slaves Phase..

Tue Jun 30 10:28:13 2015 - [info]

Tue Jun 30 10:28:13 2015 - [info] The latest binary log file/position on all slaves is mysql-bin.000009:120

Tue Jun 30 10:28:13 2015 - [info] Latest slaves (Slaves that received relay log files to the latest):

Tue Jun 30 10:28:13 2015 - [info]   192.168.0.4(192.168.0.4:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:28:13 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:13 2015 - [info]     Primary candidate for the new Master (candidate_master is set)

Tue Jun 30 10:28:13 2015 - [info]   192.168.0.5(192.168.0.5:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:28:13 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:13 2015 - [info]     Not candidate for the new Master (no_master is set)

Tue Jun 30 10:28:13 2015 - [info] The oldest binary log file/position on all slaves is mysql-bin.000009:120

Tue Jun 30 10:28:13 2015 - [info] Oldest slaves:

Tue Jun 30 10:28:13 2015 - [info]   192.168.0.4(192.168.0.4:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:28:13 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:13 2015 - [info]     Primary candidate for the new Master (candidate_master is set)

Tue Jun 30 10:28:13 2015 - [info]   192.168.0.5(192.168.0.5:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:28:13 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:13 2015 - [info]     Not candidate for the new Master (no_master is set)

Tue Jun 30 10:28:13 2015 - [info]

Tue Jun 30 10:28:13 2015 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase.. XXXXXXXXXXXXXXXXXXXXXX

Tue Jun 30 10:28:13 2015 - [info]

Tue Jun 30 10:28:13 2015 - [info] Fetching dead master's binary logs..

Tue Jun 30 10:28:13 2015 - [info] Executing command on the dead master 192.168.0.3(192.168.0.3:3306): save_binary_logs --command=save --start_file=mysql-bin.000009  --start_pos=120 --binlog_dir=/data/mysql/data/ --output_file=/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.54

  Creating /var/tmp if not exists..    ok.

 Concat binary/relay logs from mysql-bin.000009 pos 120 to mysql-bin.000009 EOF into /var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog ..

  Dumping binlog format description event, from position 0 to 120.. ok.

  Dumping effective binlog data from /data/mysql/data//mysql-bin.000009 position 120 to tail(143).. ok.

 Concat succeeded.

Tue Jun 30 10:28:14 2015 - [info] scp from root@192.168.0.3:/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog to local:/etc/mha/app1/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog succeeded.

Tue Jun 30 10:28:14 2015 - [info] HealthCheck: SSH to 192.168.0.4 is reachable.

Tue Jun 30 10:28:15 2015 - [info] HealthCheck: SSH to 192.168.0.5 is reachable.

Tue Jun 30 10:28:15 2015 - [info]

Tue Jun 30 10:28:15 2015 - [info] * Phase 3.3: Determining New Master Phase..

Tue Jun 30 10:28:15 2015 - [info]

Tue Jun 30 10:28:15 2015 - [info] XXXXXXXXX  Finding the latest slave that has all relay logs for recovering other slaves..

Tue Jun 30 10:28:15 2015 - [info] XXXXXXXXX  All slaves received relay logs to the same position. No need to resync each other.

Tue Jun 30 10:28:15 2015 - [info] Searching new master from slaves..

Tue Jun 30 10:28:15 2015 - [info]  Candidate masters from the configuration file:

Tue Jun 30 10:28:15 2015 - [info]   192.168.0.4(192.168.0.4:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:28:15 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:15 2015 - [info]     Primary candidate for the new Master (candidate_master is set)

Tue Jun 30 10:28:15 2015 - [info]  Non-candidate masters:

Tue Jun 30 10:28:15 2015 - [info]   192.168.0.5(192.168.0.5:3306)  Version=5.6.20-log (oldest major version between slaves) log-bin:enabled

Tue Jun 30 10:28:15 2015 - [info]     Replicating from 192.168.0.3(192.168.0.3:3306)

Tue Jun 30 10:28:15 2015 - [info]     Not candidate for the new Master (no_master is set)

Tue Jun 30 10:28:15 2015 - [info]  Searching from candidate_master slaves which have received the latest relay log events..

Tue Jun 30 10:28:15 2015 - [info] New master is 192.168.0.4(192.168.0.4:3306)

Tue Jun 30 10:28:15 2015 - [info] Starting master failover..

Tue Jun 30 10:28:15 2015 - [info]

From:

192.168.0.3 (current master)

 +--192.168.0.4

 +--192.168.0.5

To:

192.168.0.4 (new master)

 +--192.168.0.5

Tue Jun 30 10:28:15 2015 - [info]

Tue Jun 30 10:28:15 2015 - [info] * Phase 3.3: New Master Diff Log Generation Phase..

Tue Jun 30 10:28:15 2015 - [info]

Tue Jun 30 10:28:15 2015 - [info] XXXXXXX  This server has all relay logs. No need to generate diff files from the latest slave.

Tue Jun 30 10:28:15 2015 - [info] Sending binlog..

Tue Jun 30 10:28:15 2015 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog to root@192.168.0.4:/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog succeeded.

Tue Jun 30 10:28:15 2015 - [info]

Tue Jun 30 10:28:15 2015 - [info] * Phase 3.4: Master Log Apply Phase..

Tue Jun 30 10:28:15 2015 - [info]

Tue Jun 30 10:28:15 2015 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.

Tue Jun 30 10:28:15 2015 - [info] Starting recovery on 192.168.0.4(192.168.0.4:3306)..

Tue Jun 30 10:28:15 2015 - [info]  Generating diffs succeeded.

Tue Jun 30 10:28:15 2015 - [info] Waiting until all relay logs are applied.

Tue Jun 30 10:28:15 2015 - [info]  done.

Tue Jun 30 10:28:16 2015 - [info] Getting slave status..

Tue Jun 30 10:28:16 2015 - [info] This slave(192.168.0.4)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000009:120). No need to recover from Exec_Master_Log_Pos.

Tue Jun 30 10:28:16 2015 - [info] Connecting to the target slave host 192.168.0.4, running recover script..

Tue Jun 30 10:28:16 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='monitor' --slave_host=192.168.0.4 --slave_ip=192.168.0.4  --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog --workdir=/var/tmp --target_version=5.6.20-log --timestamp=20150630102812 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.54 --slave_pass=xxx

Tue Jun 30 10:28:16 2015 - [info]

MySQL client version is 5.6.20. Using --binary-mode.

Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog on 192.168.0.4:3306. This may take long time...

Applying log files succeeded.

Tue Jun 30 10:28:16 2015 - [info]  All relay logs were successfully applied.

Tue Jun 30 10:28:16 2015 - [info] Getting new master's binlog name and position..

Tue Jun 30 10:28:16 2015 - [info]  mysql-bin.000015:120

Tue Jun 30 10:28:16 2015 - [info] XXXXXXX  all other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.0.4', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000015', MASTER_LOG_POS=120, MASTER_USER='repl1', MASTER_PASSWORD='xxx';

Tue Jun 30 10:28:16 2015 - [info] Executing master IP activate script:

Tue Jun 30 10:28:16 2015 - [info]   /etc/mha/app1/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.0.3 --orig_master_ip=192.168.0.3 --orig_master_port=3306 --new_master_host=192.168.0.4 --new_master_ip=192.168.0.4 --new_master_port=3306 --new_master_user='monitor' --new_master_password='123456'

Enabling the VIP - 192.168.0.10/24 on the new master - 192.168.0.4

Tue Jun 30 10:28:16 2015 - [info]  OK.

Tue Jun 30 10:28:16 2015 - [info] Setting read_only=0 on 192.168.0.4(192.168.0.4:3306)..

Tue Jun 30 10:28:16 2015 - [info]  ok.

Tue Jun 30 10:28:16 2015 - [info] ** Finished master recovery successfully.

Tue Jun 30 10:28:16 2015 - [info] * Phase 3: Master Recovery Phase completed.

Tue Jun 30 10:28:16 2015 - [info]

Tue Jun 30 10:28:16 2015 - [info] * Phase 4: Slaves Recovery Phase..

Tue Jun 30 10:28:16 2015 - [info]

Tue Jun 30 10:28:16 2015 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..

Tue Jun 30 10:28:16 2015 - [info]

Tue Jun 30 10:28:16 2015 - [info] -- Slave diff file generation on host 192.168.0.5(192.168.0.5:3306) started, pid: 2658. Check tmp log /etc/mha/app1/192.168.0.5_3306_20150630102812.log if it takes time..

Tue Jun 30 10:28:16 2015 - [info]

Tue Jun 30 10:28:16 2015 - [info] Log messages from 192.168.0.5 ...

Tue Jun 30 10:28:16 2015 - [info]

Tue Jun 30 10:28:16 2015 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.

Tue Jun 30 10:28:16 2015 - [info] End of log messages from 192.168.0.5.

Tue Jun 30 10:28:16 2015 - [info] -- 192.168.0.5(192.168.0.5:3306) has the latest relay log events.

Tue Jun 30 10:28:16 2015 - [info] Generating relay diff files from the latest slave succeeded.

Tue Jun 30 10:28:16 2015 - [info]

Tue Jun 30 10:28:16 2015 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..

Tue Jun 30 10:28:16 2015 - [info]

Tue Jun 30 10:28:16 2015 - [info] -- Slave recovery on host 192.168.0.5(192.168.0.5:3306) started, pid: 2660. Check tmp log /etc/mha/app1/192.168.0.5_3306_20150630102812.log if it takes time..

Tue Jun 30 10:28:18 2015 - [info]

Tue Jun 30 10:28:18 2015 - [info] Log messages from 192.168.0.5 ...

Tue Jun 30 10:28:18 2015 - [info]

Tue Jun 30 10:28:16 2015 - [info] Sending binlog..

Tue Jun 30 10:28:17 2015 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog to root@192.168.0.5:/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog succeeded.

Tue Jun 30 10:28:17 2015 - [info] Starting recovery on 192.168.0.5(192.168.0.5:3306)..

Tue Jun 30 10:28:17 2015 - [info]  Generating diffs succeeded.

Tue Jun 30 10:28:17 2015 - [info] Waiting until all relay logs are applied.

Tue Jun 30 10:28:17 2015 - [info]  done.

Tue Jun 30 10:28:17 2015 - [info] Getting slave status..

Tue Jun 30 10:28:17 2015 - [info] This slave(192.168.0.5)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000009:120). No need to recover from Exec_Master_Log_Pos.

Tue Jun 30 10:28:17 2015 - [info] Connecting to the target slave host 192.168.0.5, running recover script..

Tue Jun 30 10:28:17 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='monitor' --slave_host=192.168.0.5 --slave_ip=192.168.0.5  --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog --workdir=/var/tmp --target_version=5.6.20-log --timestamp=20150630102812 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.54 --slave_pass=xxx

Tue Jun 30 10:28:17 2015 - [info]

MySQL client version is 5.6.20. Using --binary-mode.

Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog on 192.168.0.5:3306. This may take long time...

Applying log files succeeded.

Tue Jun 30 10:28:17 2015 - [info]  All relay logs were successfully applied.

Tue Jun 30 10:28:17 2015 - [info]  Resetting slave 192.168.0.5(192.168.0.5:3306) and starting replication from the new master 192.168.0.4(192.168.0.4:3306)..

Tue Jun 30 10:28:18 2015 - [info]  Executed CHANGE MASTER.

Tue Jun 30 10:28:18 2015 - [info]  Slave started.

Tue Jun 30 10:28:18 2015 - [info] End of log messages from 192.168.0.5.

Tue Jun 30 10:28:18 2015 - [info] -- Slave recovery on host 192.168.0.5(192.168.0.5:3306) succeeded.

Tue Jun 30 10:28:18 2015 - [info] All new slave servers recovered successfully.

Tue Jun 30 10:28:18 2015 - [info]

Tue Jun 30 10:28:18 2015 - [info] * Phase 5: New master cleanup phase..

Tue Jun 30 10:28:18 2015 - [info]

Tue Jun 30 10:28:18 2015 - [info] Resetting slave info on the new master..

Tue Jun 30 10:28:18 2015 - [info]  192.168.0.4: Resetting slave info succeeded.

Tue Jun 30 10:28:18 2015 - [info] Master failover to 192.168.0.4(192.168.0.4:3306) completed successfully.

Tue Jun 30 10:28:18 2015 - [info] 

----- Failover Report -----

app1: MySQL Master failover 192.168.0.3 to 192.168.0.4 succeeded

Master 192.168.0.3 is down!

Check MHA Manager logs at yaolansvr_slave:/var/log/manager.log for details.

Started automated(non-interactive) failover.

Invalidated master IP address on 192.168.0.3.

The latest slave 192.168.0.4(192.168.0.4:3306) has all relay logs for recovery.

Selected 192.168.0.4 as a new master.

192.168.0.4: OK: Applying all logs succeeded.

192.168.0.4: OK: Activated master IP address.

192.168.0.5: This host has the latest relay log events.

Generating relay diff files from the latest slave succeeded.

192.168.0.5: OK: Applying all logs succeeded. Slave started, replicating from 192.168.0.4.

192.168.0.4: Resetting slave info succeeded.

Master failover to 192.168.0.4(192.168.0.4:3306) completed successfully.

######192.168.0.4提升为主之后。查看状态：

# masterha_check_status --conf=/etc/mha/app1/app1.cnf

app1 is stopped(2:NOT_RUNNING).

mysql> select @@read_only;

+-------------+

| @@read_only |

+-------------+

|           0 |

+-------------+

######修复宕机的机器

首先cat /var/log/manager.log|grep -i "All other slaves should start"确定change master命令。把宕掉的数据库给启动。登陆进去后，slave status为空，使用change master命令设置应用的主节点，启动slave进程

然后设置read_only=1。最后检查复制环境，必须启动mha manager的监控（ps aux|grep perl）并查看状态，删除app1.failover.complete，并把# mysql -e "set global relay_log_purge=0"

192.168.0.4关闭mysql后，192.168.0.3提升为主的过程中报错：

Tue Jun 30 11:50:37 2015 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln297] Last failover was done at 2015/06/30 10:05:18. Current time is too early to do failover again. If you want to do failover, manually remove /etc/mha/app1/app1.failover.complete and run this script again.

Tue Jun 30 11:50:37 2015 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln178] Got ERROR:  at /usr/local/bin/masterha_manager line 65

而且masterha_manager会马上死掉

注意：

（1）一旦重新启动slave，记得须要将mysql -e "set global read_only=1"

mysql mha高可用架构的安装的更多相关文章

美团点评MySQL数据库高可用架构从MMM到MHA+Zebra以及MHA+Proxy的演进
本文介绍最近几年美团点评MySQL数据库高可用架构的演进过程,以及我们在开源技术基础上做的一些创新.同时,也和业界其它方案进行综合对比,了解业界在高可用方面的进展,和未来我们的一些规划和展望. MMM ...
MySQL MHA 高可用集群部署及故障切换
MySQL MHA 高可用集群部署及故障切换 1.概念 2.搭建MySQL + MHA 1.概念: a)MHA概念 : MHA(MasterHigh Availability)是一套优秀的MySQL高 ...
搭建MySQL MHA高可用
本文内容参考:http://www.ttlsa.com/mysql/step-one-by-one-deploy-mysql-mha-cluster/ MySQL MHA 高可用集群环境: Linu ...
mysql复制(高可用架构方案的基础)
mysql复制:把一个数据库实例上所有改变复制到另外一个数据库库服务器实例的过程特点:1.没有改变就无所谓复制 ;改变是复制的根本与数据源2.所有的改变:是指可以复制全部改变,也可以复制部分改变可以 ...
Oracle Compute云快速搭建MySQL Keepalived高可用架构
最近有个客户在测试Oracle Compute云,他们的应用需要使用MySQL数据库,由于是企业级应用一定要考虑高可用架构,因此有需求要在Oracle Compute云上搭建MySQL高可用集群.客户 ...
mysql实现高可用架构之MHA
一.简介 MHA(Master HA)是一款开源的 MySQL 的高可用程序,它为 MySQL 主从复制架构提供了 automating master failover 功能.MHA 在监控到 mas ...
Mysql MHA高可用集群架构
** 记得之前发过一篇文章,名字叫<浅析MySQL高可用架构>,之后一直有很多小伙伴在公众号后台或其它渠道问我,何时有相关的深入配置管理文章出来,因此,民工哥,也将对前面的各类架构逐一进行 ...
MHA高可用架构与Atlas读写分离
1.1 MHA简介 1.1.1 MHA软件介绍 MHA(Master High Availability)目前在MySQL高可用方面是一个相对成熟的解决方案,它由日本DeNA公司youshimaton ...
MySQL集群搭建(5)-MHA高可用架构
1 概述 1.1 MHA 简介 MHA - Master High Availability 是由 Perl 实现的一款高可用程序,出现故障时,MHA 以最小的停机时间(通常10-30秒)执行 mas ...

随机推荐

php学习笔记6
PHP 字符串变量 PHP 中的字符串变量字符串变量用于包含有字符的值. 在创建字符串之后,我们就可以对它进行操作了.您可以直接在函数中使用字符串,或者把它存储在变量中. 在下面的实例中,我们创建一 ...
前端上传文件后端PHP获取文件
<body> <form action="03-post-file.php" method="post" enctype="mult ...
BZOJ3926: [Zjoi2015]诸神眷顾的幻想乡（广义后缀自动机）
Description 幽香是全幻想乡里最受人欢迎的萌妹子,这天,是幽香的2600岁生日,无数幽香的粉丝到了幽香家门前的太阳花田上来为幽香庆祝生日. 粉丝们非常热情,自发组织表演了一系列节目给幽香看. ...
杭电1102 Constructing Roads
Constructing Roads Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 65536/32768 K (Java/Other ...
103.tcp通信实现远程控制
客户端代码 #define _CRT_SECURE_NO_WARNINGS #include<stdio.h> #include<stdlib.h> #include < ...
一个很详细的web.xml讲解（转）
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE web-app PUBLIC "- ...
【Codeforces Round #450 (Div. 2) B】Position in Fraction
[链接] 我是链接,点我呀:) [题意] 在这里输入题意 [题解] 找循环节就好. ->其实可以不用找出来整个循环节. 有找到c就直接输出. 找到了循环节还没找到的话,直接输出无解. [代码] ...
洛谷 P1170 兔八哥与猎人
P1170 兔八哥与猎人题目描述兔八哥躲藏在树林旁边的果园里.果园有M × N棵树,组成一个M行N列的矩阵,水平或垂直相邻的两棵树的距离为1.兔八哥在一棵果树下. 猎人背着猎枪走进了果园,他爬上一 ...
Maven在dos窗口中的命令
转自:https://www.cnblogs.com/zyjava/p/4310957.html 1.配置环境变量 MAVEN_HOME : D:\apache-maven-3.0.2 MAVEN : ...
Android 迭代器 Iteraor迭代器以及foreach的使用
Iterator是一个迭代器接口,专门用来迭代各种Collection集合,包括Set集合和List集合. Java要求各种集合都提供一个iteratot()方法,该方法返回一个Iterator用于遍 ...

mysql mha高可用架构的安装

mysql mha高可用架构的安装的更多相关文章

随机推荐

热门专题