MySQL高可用之MHA切换测试（switchover & failover）
Preface
I've installed MasterHA yesterday,Now let's test the master-slave switch and failover feature.
Framework
| Hostname | IP | Port | Identity | OS Version | MySQL Version |
| zlm2 | 192.168.1.101 | 3306 | master | CentOS 7.0 | 5.7.21 |
| zlm3 | 192.168.1.102 | 3306 | slave/mha-manager | CentOS 7.0 | 5.7.21 |
| null | 192.168.1.200 | null | vip | null | null |
Procedure
Test 1：Manual master switchover
Check state of MHA-manager on zlm3.
[root@zlm3 :: ~]
#masterha_check_ssh --conf=/etc/masterha/app1.conf
Fri Aug :: - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Starting SSH connection tests..
Fri Aug :: - [debug]
Fri Aug :: - [debug] Connecting via SSH from root@192.168.1.101(192.168.1.101:) to root@192.168.1.102(192.168.1.102:)..
Fri Aug :: - [debug] ok.
Fri Aug :: - [debug]
Fri Aug :: - [debug] Connecting via SSH from root@192.168.1.102(192.168.1.102:) to root@192.168.1.101(192.168.1.101:)..
Fri Aug :: - [debug] ok.
Fri Aug :: - [info] All SSH connection tests passed successfully. [root@zlm3 :: ~]
#masterha_check_repl --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] MHA::MasterMonitor version 0.56.
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Checking slave configurations..
Fri Aug :: - [info] read_only= is not set on slave 192.168.1.102(192.168.1.102:).
Fri Aug :: - [info] Checking replication filtering settings..
Fri Aug :: - [info] binlog_do_db= , binlog_ignore_db=
Fri Aug :: - [info] Replication filtering check ok.
Fri Aug :: - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Fri Aug :: - [info] Checking SSH publickey authentication settings on the current master..
ssh_exchange_identification: Connection closed by remote host
Fri Aug :: - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
Fri Aug :: - [info]
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) Fri Aug :: - [info] Checking replication health on 192.168.1.102..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Checking master_ip_failover_script status:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_ssh_port=
Fri Aug :: - [info] OK.
Fri Aug :: - [warning] shutdown_script is not defined.
Fri Aug :: - [info] Got exit code (Not master dead). MySQL Replication Health is OK. [root@zlm3 :: ~]
#Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
ssh_exchange_identification: Connection closed by remote host
^C [root@zlm3 :: ~]
#masterha_check_status --conf=/etc/masterha/app1.conf
app1 (pid:) is running(:PING_OK), master:192.168.1.101
Switch master to slave and make it become a new slave of new master.
[root@zlm3 :: ~]
#masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --master_state=alive --new_master_host=192.168.1.102 --orig_master_is_new_slave --running_updates_limit=
Fri Aug :: - [info] MHA::MasterRotate version 0.56.
Fri Aug :: - [info] Starting online master switch..
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Configuration Check Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set) It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.1.101(192.168.1.101:)? (YES/no): yes
Fri Aug :: - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Checking MHA is not monitoring or doing failover..
Fri Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/MasterRotate.pm, ln142] Getting advisory lock failed on the current master. MHA Monitor runs on the current master. Stop MHA Manager/Monitor and try again.
Fri Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/bin/masterha_master_switch line . //It means that we should stop MHA-manager when donging switchover master. [root@zlm3 :: ~]
#masterha_stop --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf
Stopped app1 successfully.
[]+ Exit masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf [root@zlm3 :: ~]
#masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --master_state=alive --new_master_host=192.168.1.102 --orig_master_is_new_slave --running_updates_limit=
Fri Aug :: - [info] MHA::MasterRotate version 0.56.
Fri Aug :: - [info] Starting online master switch..
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Configuration Check Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set) It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.1.101(192.168.1.101:)? (YES/no): yes
Fri Aug :: - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Checking MHA is not monitoring or doing failover..
Fri Aug :: - [info] Checking replication health on 192.168.1.102..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] 192.168.1.102 can be new master.
Fri Aug :: - [info]
From:
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) To:
192.168.1.102(192.168.1.102:) (new master)
+--192.168.1.101(192.168.1.101:) Starting master switch from 192.168.1.101(192.168.1.101:) to 192.168.1.102(192.168.1.102:)? (yes/NO): yes
Fri Aug :: - [info] Checking whether 192.168.1.102(192.168.1.102:) is ok for the new master..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:): Resetting slave pointing to the dummy host.
Fri Aug :: - [info] ** Phase : Configuration Check Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Rejecting updates Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Executing master ip online change script to disable write on the current master:
Fri Aug :: - [info] /etc/masterha/master_ip_online_change --command=stop --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_user='zlm' --orig_master_password='zlmzlm' --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port= --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_ssh_port= --new_master_ssh_port= --orig_master_is_new_slave
Unknown option: new_master_ssh_port
Fri Aug :: Set read_only on the new master.. ok.
Fri Aug :: drop vip 10.33.101.239..
ssh_exchange_identification: Connection closed by remote host
Fri Aug :: Waiting all running threads are disconnected.. (max milliseconds)
{'Time' => '','db' => undef,'Id' => '','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
Fri Aug :: Waiting all running threads are disconnected.. (max milliseconds)
{'Time' => '','db' => undef,'Id' => '','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
Fri Aug :: Waiting all running threads are disconnected.. (max milliseconds)
{'Time' => '','db' => undef,'Id' => '','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
Fri Aug :: Set read_only= on the orig master.. ok.
Fri Aug :: Waiting all running queries are disconnected.. (max milliseconds)
{'Time' => '','db' => undef,'Id' => '','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
Fri Aug :: Killing all application threads..
Fri Aug :: done.
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Fri Aug :: - [info] Executing FLUSH TABLES WITH READ LOCK..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Orig master binlog:pos is mysql-bin.:.
Fri Aug :: - [info] Waiting to execute all relay logs on 192.168.1.102(192.168.1.102:)..
Fri Aug :: - [info] master_pos_wait(mysql-bin.:) completed on 192.168.1.102(192.168.1.102:). Executed events.
Fri Aug :: - [info] done.
Fri Aug :: - [info] Getting new master's binlog name and position..
Fri Aug :: - [info] mysql-bin.:
Fri Aug :: - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.102', MASTER_PORT=, MASTER_AUTO_POSITION=, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Fri Aug :: - [info] Executing master ip online change script to allow write on the new master:
Fri Aug :: - [info] /etc/masterha/master_ip_online_change --command=start --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_user='zlm' --orig_master_password='zlmzlm' --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port= --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_ssh_port= --new_master_ssh_port= --orig_master_is_new_slave
Unknown option: new_master_ssh_port
Fri Aug :: Set read_only= on the new master.
Fri Aug :: 328259Add vip 10.33.101.239 on p3p1..
ssh_exchange_identification: Connection closed by remote host
Fri Aug :: - [info] ok.
Fri Aug :: - [info]
Fri Aug :: - [info] * Switching slaves in parallel..
Fri Aug :: - [info]
Fri Aug :: - [info] Unlocking all tables on the orig master:
Fri Aug :: - [info] Executing UNLOCK TABLES..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Starting orig master as a new slave..
Fri Aug :: - [info] Resetting slave 192.168.1.101(192.168.1.101:) and starting replication from the new master 192.168.1.102(192.168.1.102:)..
Fri Aug :: - [info] Executed CHANGE MASTER.
Fri Aug :: - [info] Slave started.
Fri Aug :: - [info] All new slave servers switched successfully.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : New master cleanup phase..
Fri Aug :: - [info]
Fri Aug :: - [info] 192.168.1.102: Resetting slave info succeeded.
Fri Aug :: - [info] Switching master to 192.168.1.102(192.168.1.102:) completed successfully. [root@zlm3 :: ~]
#
Check the master-slave replication status.
//New master(original slave)
(zlm@192.168.1.102 )[(none)]>show master status;
+------------------+----------+--------------+------------------+------------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+------------------------------------------------+
| mysql-bin. | | | | 1b7181ee-6eaf-11e8-998e-080027de0e0e:- |
+------------------+----------+--------------+------------------+------------------------------------------------+
row in set (0.00 sec) (zlm@192.168.1.102 )[(none)]>show slave status\G
Empty set (0.00 sec) //New slave(original master)
(zlm@192.168.1.101 )[(none)]>show master status;
+------------------+----------+--------------+------------------+------------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+------------------------------------------------+
| mysql-bin. | | | | 1b7181ee-6eaf-11e8-998e-080027de0e0e:- |
+------------------+----------+--------------+------------------+------------------------------------------------+
row in set (0.01 sec) (zlm@192.168.1.101 )[(none)]>show slave status\G
*************************** . row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.1.102
Master_User: repl
Master_Port:
Connect_Retry:
Master_Log_File: mysql-bin.
Read_Master_Log_Pos:
Relay_Log_File: relay-bin.
Relay_Log_Pos:
Relay_Master_Log_File: mysql-bin.
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno:
Last_Error:
Skip_Counter:
Exec_Master_Log_Pos:
Relay_Log_Space:
Until_Condition: None
Until_Log_File:
Until_Log_Pos:
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master:
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno:
Last_IO_Error:
Last_SQL_Errno:
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id:
Master_UUID: 842ea497--11e8-83ca-080027de0e0e
Master_Info_File: mysql.slave_master_info
SQL_Delay:
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count:
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 1b7181ee-6eaf-11e8-998e-080027de0e0e:-
Auto_Position:
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
row in set (0.00 sec)
Check the log of MasterHA on zlm3.
[root@zlm3 :: ~]
#cd /var/log/masterha/app1 [root@zlm3 :: /var/log/masterha/app1]
#cat app1.log
Fri Aug :: - [info] MHA::MasterMonitor version 0.56.
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Checking slave configurations..
Fri Aug :: - [info] read_only= is not set on slave 192.168.1.102(192.168.1.102:).
Fri Aug :: - [info] Checking replication filtering settings..
Fri Aug :: - [info] binlog_do_db= , binlog_ignore_db=
Fri Aug :: - [info] Replication filtering check ok.
Fri Aug :: - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Fri Aug :: - [info] Checking SSH publickey authentication settings on the current master..
Fri Aug :: - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
Fri Aug :: - [info]
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) Fri Aug :: - [info] Checking master_ip_failover_script status:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_ssh_port=
Fri Aug :: - [info] OK.
Fri Aug :: - [warning] shutdown_script is not defined.
Fri Aug :: - [info] Set master ping interval seconds.
Fri Aug :: - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Fri Aug :: - [info] Starting ping health check on 192.168.1.101(192.168.1.101:)..
Fri Aug :: - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Fri Aug :: - [info] Got terminate signal. Exit.
Fri Aug :: - [info] MHA::MasterMonitor version 0.56.
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Checking slave configurations..
Fri Aug :: - [info] read_only= is not set on slave 192.168.1.102(192.168.1.102:).
Fri Aug :: - [info] Checking replication filtering settings..
Fri Aug :: - [info] binlog_do_db= , binlog_ignore_db=
Fri Aug :: - [info] Replication filtering check ok.
Fri Aug :: - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Fri Aug :: - [info] Checking SSH publickey authentication settings on the current master..
Fri Aug :: - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
Fri Aug :: - [info]
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) Fri Aug :: - [info] Checking master_ip_failover_script status:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_ssh_port=
Fri Aug :: - [info] OK.
Fri Aug :: - [warning] shutdown_script is not defined.
Fri Aug :: - [info] Set master ping interval seconds.
Fri Aug :: - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Fri Aug :: - [info] Starting ping health check on 192.168.1.101(192.168.1.101:)..
Fri Aug :: - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Fri Aug :: - [info] Got terminate signal. Exit.
Test 2：Manual master failover
Execute "masterha_master_switch" script again to generate a failover on zlm2.
[root@zlm2 :: ~]
#masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --dead_master_host=192.168.1.102 --master_state=dead --new_master_host=192.168.1.101 --ignore_last_failover
--dead_master_ip=<dead_master_ip> is not set. Using 192.168.1.102.
--dead_master_port=<dead_master_port> is not set. Using .
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] MHA::MasterFailover version 0.56.
Fri Aug :: - [info] Starting master failover.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Configuration Check Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln187] None of server is dead. Stop failover.
Fri Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/bin/masterha_master_switch line . //Stop mysqld of master on zlm3.
[root@zlm3 :: ~]
#mysqladmin shutdown [root@zlm3 :: ~]
#ps aux|grep mysqld
mysql 0.0 19.6 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3306/my.cnf
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld [root@zlm3 :: ~]
#ps aux|grep mysqld
mysql 0.0 19.6 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3306/my.cnf
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld [root@zlm3 :: ~]
#ps aux|grep mysqld
mysql 0.0 19.1 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3306/my.cnf
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld [root@zlm3 :: ~]
#ps aux|grep mysqld
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld //Execute the above command again on zlm2.
[root@zlm2 :: ~]
#masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --dead_master_host=192.168.1.102 --master_state=dead --new_master_host=192.168.1.101 --ignore_last_failover
--dead_master_ip=<dead_master_ip> is not set. Using 192.168.1.102.
--dead_master_port=<dead_master_port> is not set. Using .
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] MHA::MasterFailover version 0.56.
Fri Aug :: - [info] Starting master failover.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Configuration Check Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Checking master reachability via MySQL(double check)...
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Master 192.168.1.102(192.168.1.102:) is dead. Proceed? (yes/NO): yes
Fri Aug :: - [info] Starting GTID based failover.
Fri Aug :: - [info]
Fri Aug :: - [info] ** Phase : Configuration Check Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Dead Master Shutdown Phase..
Fri Aug :: - [info]
ssh: connect to host 192.168.1.102 port : Connection refused
Fri Aug :: - [warning] HealthCheck: SSH to 192.168.1.102 is NOT reachable.
Fri Aug :: - [info] Forcing shutdown so that applications never connect to the current master..
Fri Aug :: - [info] Executing master IP deactivation script:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --orig_master_host=192.168.1.102 --orig_master_ip=192.168.1.102 --orig_master_port= --command=stop --orig_master_ssh_port=
ssh: connect to host 192.168.1.102 port : Connection refused
Fri Aug :: - [info] done.
Fri Aug :: - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Fri Aug :: - [info] * Phase : Dead Master Shutdown Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Master Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.1: Getting Latest Slaves Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] The latest binary log file/position on all slaves is mysql-bin.:
Fri Aug :: - [info] Latest slaves (Slaves that received relay log files to the latest):
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] The oldest binary log file/position on all slaves is mysql-bin.:
Fri Aug :: - [info] Oldest slaves:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.3: Determining New Master Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] 192.168.1.101 can be new master.
Fri Aug :: - [info] New master is 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Starting master failover..
Fri Aug :: - [info]
From:
192.168.1.102(192.168.1.102:) (current master)
+--192.168.1.101(192.168.1.101:) To:
192.168.1.101(192.168.1.101:) (new master) Starting master switch from 192.168.1.102(192.168.1.102:) to 192.168.1.101(192.168.1.101:)? (yes/NO): yes
Fri Aug :: - [info] New master decided manually is 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.3: New Master Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Waiting all logs to be applied..
Fri Aug :: - [info] done.
Fri Aug :: - [info] Getting new master's binlog name and position..
Fri Aug :: - [info] mysql-bin.:
Fri Aug :: - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.101', MASTER_PORT=, MASTER_AUTO_POSITION=, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Fri Aug :: - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin., , 1b7181ee-6eaf-11e8-998e-080027de0e0e:-
Fri Aug :: - [info] Executing master IP activate script:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.102 --orig_master_ip=192.168.1.102 --orig_master_port= --new_master_host=192.168.1.101 --new_master_ip=192.168.1.101 --new_master_port= --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_port= --new_master_ssh_port=
Unknown option: new_master_ssh_port
Set read_only= on the new master.
ssh_exchange_identification: Connection closed by remote host
Fri Aug :: - [info] OK.
Fri Aug :: - [info] ** Finished master recovery successfully.
Fri Aug :: - [info] * Phase : Master Recovery Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Slaves Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 4.1: Starting Slaves in parallel..
Fri Aug :: - [info]
Fri Aug :: - [info] All new slave servers recovered successfully.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : New master cleanup phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Resetting slave info on the new master..
Fri Aug :: - [info] 192.168.1.101: Resetting slave info succeeded.
Fri Aug :: - [info] Master failover to 192.168.1.101(192.168.1.101:) completed successfully.
Fri Aug :: - [info] ----- Failover Report ----- app1: MySQL Master failover 192.168.1.102(192.168.1.102:) to 192.168.1.101(192.168.1.101:) succeeded Master 192.168.1.102(192.168.1.102:) is down! Check MHA Manager logs at zlm2 for details. Started manual(interactive) failover.
Invalidated master IP address on 192.168.1.102(192.168.1.102:)
Selected 192.168.1.101(192.168.1.101:) as a new master.
192.168.1.101(192.168.1.101:): OK: Applying all logs succeeded.
192.168.1.101(192.168.1.101:): OK: Activated master IP address.
192.168.1.101(192.168.1.101:): Resetting slave info succeeded.
Master failover to 192.168.1.101(192.168.1.101:) completed successfully.
Check the status of new master on zlm2.
(zlm@192.168.1.101 )[(none)]>show master status;
+------------------+----------+--------------+------------------+------------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+------------------------------------------------+
| mysql-bin. | | | | 1b7181ee-6eaf-11e8-998e-080027de0e0e:- |
+------------------+----------+--------------+------------------+------------------------------------------------+
row in set (0.00 sec) (zlm@192.168.1.101 )[(none)]>show slave status\G
Empty set (0.00 sec)
Check the file and log on MasterHA.
[root@zlm2 :: ~]
#cd /var/log/masterha/app1 [root@zlm2 :: /var/log/masterha/app1]
#ls -l
total
-rw-r--r-- root root Aug : app1.failover.complete
-rw-r--r-- root root Aug : app1.log //The option of "--ignore_last_failover" can neglect the influence of existence of "app1.failover.complete".Otherwise,the failover operation will be terminated by error.
//This file will be created after a failover operation and it will be created only on the original slave who wants to become a new master. [root@zlm2 :: /var/log/masterha/app1]
#cat app1.log
Thu Aug :: - [info] MHA::MasterMonitor version 0.56.
Thu Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301] Got MySQL error when connecting 192.168.1.101(192.168.1.101:) ::Access denied for user 'root'@'zlm2' (using password: NO), but this is not a MySQL crash. Check MySQL server settings.
at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line .
Thu Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301] Got MySQL error when connecting 192.168.1.102(192.168.1.102:) ::Access denied for user 'root'@'zlm2' (using password: NO), but this is not a MySQL crash. Check MySQL server settings.
at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line .
Thu Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln309] Got fatal error, stopping operations
Thu Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm line .
Thu Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers.
Thu Aug :: - [info] Got exit code (Not master dead).
Thu Aug :: - [info] MHA::MasterMonitor version 0.56.
Thu Aug :: - [info] GTID failover mode =
Thu Aug :: - [info] Dead Servers:
Thu Aug :: - [info] Alive Servers:
Thu Aug :: - [info] 192.168.1.101(192.168.1.101:)
Thu Aug :: - [info] 192.168.1.102(192.168.1.102:)
Thu Aug :: - [info] Alive Slaves:
Thu Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Thu Aug :: - [info] GTID ON
Thu Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Thu Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Thu Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Thu Aug :: - [info] Checking slave configurations..
Thu Aug :: - [info] read_only= is not set on slave 192.168.1.102(192.168.1.102:).
Thu Aug :: - [info] Checking replication filtering settings..
Thu Aug :: - [info] binlog_do_db= , binlog_ignore_db=
Thu Aug :: - [info] Replication filtering check ok.
Thu Aug :: - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Thu Aug :: - [info] Checking SSH publickey authentication settings on the current master..
Thu Aug :: - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
Thu Aug :: - [info]
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) Thu Aug :: - [info] Checking master_ip_failover_script status:
Thu Aug :: - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_ssh_port=
Thu Aug :: - [info] OK.
Thu Aug :: - [warning] shutdown_script is not defined.
Thu Aug :: - [info] Set master ping interval seconds.
Thu Aug :: - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Thu Aug :: - [info] Starting ping health check on 192.168.1.101(192.168.1.101:)..
Thu Aug :: - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Thu Aug :: - [info] Got terminate signal. Exit.
Test3: Automation of master switchover.
Repair the salve replication on zlm3.
[root@zlm3 :: ~]
#mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.-log MySQL Community Server (GPL) Copyright (c) , , Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. (zlm@192.168.1.102 )[(none)]>change master to \
-> master_host='192.168.1.101',\
-> master_port=,\
-> master_user='repl',\
-> master_password='repl4slave',\
-> master_auto_position=;
Query OK, rows affected, warnings (0.02 sec) (zlm@192.168.1.102 )[(none)]>show slave status\G
*************************** . row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.1.101
Master_User: repl
Master_Port:
Connect_Retry:
Master_Log_File: mysql-bin.
Read_Master_Log_Pos:
Relay_Log_File: relay-bin.
Relay_Log_Pos:
Relay_Master_Log_File: mysql-bin.
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno:
Last_Error:
Skip_Counter:
Exec_Master_Log_Pos:
Relay_Log_Space:
Until_Condition: None
Until_Log_File:
Until_Log_Pos:
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master:
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno:
Last_IO_Error:
Last_SQL_Errno:
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id:
Master_UUID: 1b7181ee-6eaf-11e8-998e-080027de0e0e
Master_Info_File: mysql.slave_master_info
SQL_Delay:
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count:
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 1b7181ee-6eaf-11e8-998e-080027de0e0e:-
Auto_Position:
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
row in set (0.00 sec)
Start MasterHA-manager.
[root@zlm3 :: /var/log/masterha/app1]
#nohup masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf &
[]
nohup: ignoring input and appending output to ‘nohup.out’ [root@zlm3 :: /var/log/masterha/app1]
#ls -l
total
-rw-r--r-- root root Aug : app1.log
-rw-r--r-- root root Aug : app1.master_status.health //This file is created only when MasterHA-manager is running.It will continuously record the health status between slave and master.
-rw------- root root Aug : nohup.out [root@zlm3 :: /var/log/masterha/app1]
#cat nohup.out
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
ssh_exchange_identification: Connection closed by remote host [root@zlm3 :: /var/log/masterha/app1]
#cat app1.master_status.health
:PING_OK master:192.168.1.101
[root@zlm3 :: /var/log/masterha/app1]
#ps aux|grep manager
root 0.6 2.1 pts/ S : : perl /usr/bin/masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf
root 0.0 0.0 pts/ R+ : : grep --color=auto manager
Kill mysqld on zlm2 to pretend the master is dead.
[root@zlm2 :: /var/log/masterha/app1]
#pkill mysqld [root@zlm2 :: /var/log/masterha/app1]
#ps aux|grep mysqld
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld
Observe the app1.log on zlm3.
[root@zlm3 :: /var/log/masterha/app1]
#echo ''> app1.log [root@zlm3 :: /var/log/masterha/app1]
#tail -f app1.log Fri Aug :: - [warning] Got error on MySQL select ping: (MySQL server has gone away)
Fri Aug :: - [info] Executing SSH check script: exit
Fri Aug :: - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
Fri Aug :: - [warning] Got error on MySQL connect: (Can't connect to MySQL server on '192.168.1.101' (111))
Fri Aug :: - [warning] Connection failed time(s)..
Fri Aug :: - [warning] Got error on MySQL connect: (Can't connect to MySQL server on '192.168.1.101' (111))
Fri Aug :: - [warning] Connection failed time(s)..
Fri Aug :: - [warning] Got error on MySQL connect: (Can't connect to MySQL server on '192.168.1.101' (111))
Fri Aug :: - [warning] Connection failed time(s)..
Fri Aug :: - [warning] Master is not reachable from health checker!
Fri Aug :: - [warning] Master 192.168.1.101(192.168.1.101:) is not reachable!
Fri Aug :: - [warning] SSH is NOT reachable.
Fri Aug :: - [info] Connecting to a master server failed. Reading configuration file /etc/masterha/masterha_default.conf and /etc/masterha/app1.conf again, and trying to connect to all servers to check server status..
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Checking slave configurations..
Fri Aug :: - [info] read_only= is not set on slave 192.168.1.102(192.168.1.102:).
Fri Aug :: - [info] Checking replication filtering settings..
Fri Aug :: - [info] Replication filtering check ok.
Fri Aug :: - [info] Master is down!
Fri Aug :: - [info] Terminating monitoring script.
Fri Aug :: - [info] Got exit code (Master dead).
Fri Aug :: - [info] MHA::MasterFailover version 0.56.
Fri Aug :: - [info] Starting master failover.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Configuration Check Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Checking master reachability via MySQL(double check)...
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Starting GTID based failover.
Fri Aug :: - [info]
Fri Aug :: - [info] ** Phase : Configuration Check Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Dead Master Shutdown Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Forcing shutdown so that applications never connect to the current master..
Fri Aug :: - [info] Executing master IP deactivation script:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --command=stop --orig_master_ssh_port=
ssh: connect to host 192.168.1.101 port : Connection refused
Fri Aug :: - [info] done.
Fri Aug :: - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Fri Aug :: - [info] * Phase : Dead Master Shutdown Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Master Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.1: Getting Latest Slaves Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] The latest binary log file/position on all slaves is mysql-bin.:
Fri Aug :: - [info] Latest slaves (Slaves that received relay log files to the latest):
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] The oldest binary log file/position on all slaves is mysql-bin.:
Fri Aug :: - [info] Oldest slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.3: Determining New Master Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Searching new master from slaves..
Fri Aug :: - [info] Candidate masters from the configuration file:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Non-candidate masters:
Fri Aug :: - [info] Searching from candidate_master slaves which have received the latest relay log events..
Fri Aug :: - [info] New master is 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Starting master failover..
Fri Aug :: - [info]
From:
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) To:
192.168.1.102(192.168.1.102:) (new master)
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.3: New Master Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Waiting all logs to be applied..
Fri Aug :: - [info] done.
Fri Aug :: - [info] Getting new master's binlog name and position..
Fri Aug :: - [info] mysql-bin.:
Fri Aug :: - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.102', MASTER_PORT=, MASTER_AUTO_POSITION=, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Fri Aug :: - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin., , 1b7181ee-6eaf-11e8-998e-080027de0e0e:-
Fri Aug :: - [info] Executing master IP activate script:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port= --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_port= --new_master_ssh_port=
Unknown option: new_master_ssh_port
Set read_only= on the new master.
ssh_exchange_identification: Connection closed by remote host
Fri Aug :: - [info] OK.
Fri Aug :: - [info] ** Finished master recovery successfully.
Fri Aug :: - [info] * Phase : Master Recovery Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Slaves Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 4.1: Starting Slaves in parallel..
Fri Aug :: - [info]
Fri Aug :: - [info] All new slave servers recovered successfully.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : New master cleanup phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Resetting slave info on the new master..
Fri Aug :: - [info] 192.168.1.102: Resetting slave info succeeded.
Fri Aug :: - [info] Master failover to 192.168.1.102(192.168.1.102:) completed successfully.
Fri Aug :: - [info] ----- Failover Report ----- app1: MySQL Master failover 192.168.1.101(192.168.1.101:) to 192.168.1.102(192.168.1.102:) succeeded Master 192.168.1.101(192.168.1.101:) is down! Check MHA Manager logs at zlm3:/var/log/masterha/app1/app1.log for details. Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.1.101(192.168.1.101:)
Selected 192.168.1.102(192.168.1.102:) as a new master.
192.168.1.102(192.168.1.102:): OK: Applying all logs succeeded.
192.168.1.102(192.168.1.102:): OK: Activated master IP address.
192.168.1.102(192.168.1.102:): Resetting slave info succeeded.
Master failover to 192.168.1.102(192.168.1.102:) completed successfully. //Above failover report shows all the evidence and results of automation master switchover.All of the steps are executed successfully.
MySQL高可用之MHA切换测试（switchover & failover）的更多相关文章
- MySQL高可用方案MHA在线切换的步骤及原理
在日常工作中,会碰到如下的场景,如mysql数据库升级,主服务器硬件升级等,这个时候就需要将写操作切换到另外一台服务器上,那么如何进行在线切换呢?同时,要求切换过程短,对业务的影响比较小. MHA就提 ...
- MySQL高可用方案MHA自动Failover与手动Failover的实践及原理
集群信息 角色 IP地址 ServerID 类型 Master ...
- MySQL高可用方案--MHA部署及故障转移
架构设计及必要配置 主机环境 IP 主机名 担任角色 192.168.192.128 node_master MySQL-Master| ...
- 【DB宝42】MySQL高可用架构MHA+ProxySQL实现读写分离和负载均衡
目录 一.MHA+ProxySQL架构 二.快速搭建MHA环境 2.1 下载MHA镜像 2.2 编辑yml文件,创建MHA相关容器 2.3 安装docker-compose软件(若已安装,可忽略) 2 ...
- mysql高可用方案MHA介绍
mysql高可用方案MHA介绍 概述 MHA是一位日本MySQL大牛用Perl写的一套MySQL故障切换方案,来保证数据库系统的高可用.在宕机的时间内(通常10-30秒内),完成故障切换,部署MHA, ...
- MySQL高可用之MHA (转)
MySQL高可用之MHA MHA简介 MHA是由日本人yoshinorim(原就职于DeNA现就职于FaceBook)开发的比较成熟的MySQL高可用方案.MHA能够在30秒内实现故障切换,并能在故障 ...
- 【DB宝19】在Docker中使用MySQL高可用之MHA
目录 一.MHA简介和架构 1.1 MHA简介 1.2 MHA工具包的组成 1.3 MHA架构 二.准备MHA环境 2.1 下载MHA镜像 2.2 编辑yml文件,创建MHA相关容器 2.3 安装do ...
- MySQL高可用之MHA的搭建 转
http://www.cnblogs.com/muhu/p/4045780.html http://www.cnblogs.com/gomysql/p/3675429.html http://www ...
- MySQL高可用之MHA的搭建
MySQL MHA架构介绍: MHA(Master High Availability)目前在MySQL高可用方面是一个相对成熟的解决方案,它由日本DeNA公司youshimaton(现就职于Face ...
随机推荐
- 线段拟合(带拉格朗日乘子，HGL)
线段特征上的扫描点满足 (1).本文的线段特征定义为:L: [dL, φL, PLs, PLe]T,如图1所示.其中,dL为笛卡尔坐标系中原点(激光传感器所在位置)到线段的距离, φL为线段特征的倾角 ...
- [转]关于VC++ MFC中的空闲Idle处理机制！
关键词: 先根据空闲标志以及消息队列是否为空这两个条件判断当前线程是否处于空闲状态(这个“空闲”的含义同操作系统的含义不同,是MFC自己所谓的“空闲”),如果是,就调用CWinThread::OnId ...
- P1800 software_NOI导刊2010提高（06）
P1800 software_NOI导刊2010提高(06) 题目描述 一个软件开发公司同时要开发两个软件,并且要同时交付给用户,现在公司为了尽快完成这一任务,将每个软件划分成m个模块,由公司里的技术 ...
- 【洛谷P1196】[NOI2002]银河英雄传说
银河英雄传说 题目链接 并查集时记录下以i为首的队列的长度(如果存在这个队列)num[i],便于合并, 和点i到队首的距离front[i],便于查询(在find时维护) #include<ios ...
- 自定义组件v-model的实质性理解
用了几个月Vue一直很纠结自定义组件的v-model实现,最近开始学习React时,React中受控组件与状态提升的理念与v-model不谋而合. 转载请注明地址: https://www.cnblo ...
- Qt数据库编程1
Qt中数据编程主要分为以下两点:1.利用qt提供类 访问数据库或者成为简单的数据库编程2.数据库编程中引入model/view编程模型 qt中数据库编程的步骤: 1.加载数据库驱动 QSqlDatab ...
- leetcode笔记（六）740. Delete and Earn
题目描述 Given an array nums of integers, you can perform operations on the array. In each operation, yo ...
- poj 1236 Network of Schools ： 求需要添加多少条边成为强连通图 tarjan O(E)
/** problem: http://poj.org/problem?id=1236 缩点后入度为0的点的总数为需要发放软件的学校个数 缩点后出度为0的点的总数和入度为0的点的总数的最大值为需要增加 ...
- SAP ABAP 日期，时间 相关函数
获的两个日期之间的分钟数 data min TYPE i. CALL FUNCTION 'DELTA_TIME_DAY_HOUR' EXPORTING T1 = ' T2 = ' D1 = ' D2 ...
- 根据坐标计算距离（mysql函数）
CREATE DEFINER=`root`@`localhost` FUNCTION `getDistance`(lng1 ) BEGIN DECLARE result double; DECLARE ...