MHA手动切换 原创1（主故障)

MHA提供了3种方式用于实现故障转移，分别自动故障转移，需要启用MHA监控；

在无监控的情况下的手动故障转移以及基于在线手动切换。

三种方式可以应对MySQL主从故障的任意场景。本文主要描述在无监控的情形是手动实现故障转移。供大家参考。

有关MHA的其他两种切换方式，可以参考：
            MHA 在线切换过程            MHA 自动故障转移步骤及过程剖析

1、手动故障转移的特点
    a、在监控节点未启用masterha_manager
    b、master库已经宕机或者转移到高性能服务器
    c、手动故障转移支持交互或非交互两种模式
    d、切换样例：$ masterha_master_switch --master_state=dead --conf=/etc/app1.cnf --dead_master_host=host1

2、masterha_master_switch切换的几个参数
--master_state=dead
      强制参数为"dead" 或者 "alive". dead为手动故障转移，alive为在线切换。
  
--dead_master_host=(hostname)
      强制参数为主机名，另2个--dead_master_ip --dead_master_port(缺省3306)可选。
  
--new_master_host=(hostname)
      可选参数，用于指定新master，如果未指定则按candidate_master参数设定值。
  
--interactive=(0|1)
      可选参数，指定是否交互。缺省为1，表明交互

1.server1:
  service mysql.server stop

2.monitor:
[root@monitor tmp]# masterha_master_switch --master_state=dead --conf=/etc/masterha/app1.conf --dead_master_host=server1 --dead_master_port= --new_master_host=slave1 --new_master_port=
--dead_master_ip=<dead_master_ip> is not set. Using 10.24.220.232.
Mon May  ::  - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon May  ::  - [info] Reading application default configuration from /etc/masterha/app1.conf..
Mon May  ::  - [info] Reading server configuration from /etc/masterha/app1.conf..
Mon May  ::  - [info] MHA::MasterFailover version 0.56.
Mon May  ::  - [info] Starting master failover.
Mon May  ::  - [info]
Mon May  ::  - [info] * Phase : Configuration Check Phase..
Mon May  ::  - [info]
Mon May  ::  - [debug] Connecting to servers..
Mon May  ::  - [debug]  Connected to: slave1(10.24.220.70:), user=root
Mon May  ::  - [debug]  Number of slave worker threads on host slave1(10.24.220.70:):
Mon May  ::  - [debug]  Connected to: slave2(10.169.214.33:), user=root
Mon May  ::  - [debug]  Number of slave worker threads on host slave2(10.169.214.33:):
Mon May  ::  - [debug]  Comparing MySQL versions..
Mon May  ::  - [debug]   Comparing MySQL versions done.
Mon May  ::  - [debug] Connecting to servers done.
Mon May  ::  - [info] GTID failover mode =
Mon May  ::  - [info] Dead Servers:
Mon May  ::  - [info]   server1(10.24.220.232:)
Mon May  ::  - [info] Checking master reachability via MySQL(double check)...
Mon May  ::  - [info]  ok.
Mon May  ::  - [info] Alive Servers:
Mon May  ::  - [info]   slave1(10.24.220.70:)
Mon May  ::  - [info]   slave2(10.169.214.33:)
Mon May  ::  - [info] Alive Slaves:
Mon May  ::  - [info]   slave1(10.24.220.70:)  Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Mon May  ::  - [info]     GTID ON
Mon May  ::  - [debug]    Relay log info repository: FILE
Mon May  ::  - [info]     Replicating from 10.24.220.232(10.24.220.232:)
Mon May  ::  - [info]     Primary candidate for the new Master (candidate_master is set)
Mon May  ::  - [info]   slave2(10.169.214.33:)  Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Mon May  ::  - [info]     GTID ON
Mon May  ::  - [debug]    Relay log info repository: FILE
Mon May  ::  - [info]     Replicating from 10.24.220.232(10.24.220.232:)
Mon May  ::  - [info]     Not candidate for the new Master (no_master is set)
Master server1(10.24.220.232:) is dead. Proceed? (yes/NO): yes
Mon May  ::  - [info] Starting GTID based failover.
Mon May  ::  - [info]
Mon May  ::  - [info] ** Phase : Configuration Check Phase completed.
Mon May  ::  - [info]
Mon May  ::  - [info] * Phase : Dead Master Shutdown Phase..
Mon May  ::  - [info]
Mon May  ::  - [debug] SSH connection test to server1, option -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -o ConnectTimeout=, timeout
Mon May  ::  - [debug]  Stopping IO thread on slave2(10.169.214.33:)..
Mon May  ::  - [debug]  Stopping IO thread on slave1(10.24.220.70:)..
Mon May  ::  - [debug]  Stop IO thread on slave2(10.169.214.33:) done.
Mon May  ::  - [debug]  Stop IO thread on slave1(10.24.220.70:) done.
Mon May  ::  - [info] HealthCheck: SSH to server1 is reachable.
Mon May  ::  - [info] Forcing shutdown so that applications never connect to the current master..
Mon May  ::  - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.
Mon May  ::  - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Mon May  ::  - [info] * Phase : Dead Master Shutdown Phase completed.
Mon May  ::  - [info]
Mon May  ::  - [info] * Phase : Master Recovery Phase..
Mon May  ::  - [info]
Mon May  ::  - [info] * Phase 3.1: Getting Latest Slaves Phase..
Mon May  ::  - [info]
Mon May  ::  - [debug] Fetching current slave status..
Mon May  ::  - [debug]  Fetching current slave status done.
Mon May  ::  - [info] The latest binary log file/position on all slaves is log.:
Mon May  ::  - [info] Retrieved Gtid Set: 191f7a9f-ffa2-11e5-a825-00163e00242a:-
Mon May  ::  - [info] Latest slaves (Slaves that received relay log files to the latest):
Mon May  ::  - [info]   slave1(10.24.220.70:)  Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Mon May  ::  - [info]     GTID ON
Mon May  ::  - [debug]    Relay log info repository: FILE
Mon May  ::  - [info]     Replicating from 10.24.220.232(10.24.220.232:)
Mon May  ::  - [info]     Primary candidate for the new Master (candidate_master is set)
Mon May  ::  - [info]   slave2(10.169.214.33:)  Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Mon May  ::  - [info]     GTID ON
Mon May  ::  - [debug]    Relay log info repository: FILE
Mon May  ::  - [info]     Replicating from 10.24.220.232(10.24.220.232:)
Mon May  ::  - [info]     Not candidate for the new Master (no_master is set)
Mon May  ::  - [info] The oldest binary log file/position on all slaves is log.:
Mon May  ::  - [info] Retrieved Gtid Set: 191f7a9f-ffa2-11e5-a825-00163e00242a:-
Mon May  ::  - [info] Oldest slaves:
Mon May  ::  - [info]   slave1(10.24.220.70:)  Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Mon May  ::  - [info]     GTID ON
Mon May  ::  - [debug]    Relay log info repository: FILE
Mon May  ::  - [info]     Replicating from 10.24.220.232(10.24.220.232:)
Mon May  ::  - [info]     Primary candidate for the new Master (candidate_master is set)
Mon May  ::  - [info]   slave2(10.169.214.33:)  Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Mon May  ::  - [info]     GTID ON
Mon May  ::  - [debug]    Relay log info repository: FILE
Mon May  ::  - [info]     Replicating from 10.24.220.232(10.24.220.232:)
Mon May  ::  - [info]     Not candidate for the new Master (no_master is set)
Mon May  ::  - [info]
Mon May  ::  - [info] * Phase 3.3: Determining New Master Phase..
Mon May  ::  - [info]
Mon May  ::  - [info] slave1 can be new master.
Mon May  ::  - [info] New master is slave1(10.24.220.70:)
Mon May  ::  - [info] Starting master failover..
Mon May  ::  - [info]
From:
server1(10.24.220.232:) (current master)
 +--slave1(10.24.220.70:)
 +--slave2(10.169.214.33:)

To:
slave1(10.24.220.70:) (new master)
 +--slave2(10.169.214.33:)

Starting master switch from server1(10.24.220.232:) to slave1(10.24.220.70:)? (yes/NO): yes
Mon May  ::  - [info] New master decided manually is slave1(10.24.220.70:)
Mon May  ::  - [info]
Mon May  ::  - [info] * Phase 3.3: New Master Recovery Phase..
Mon May  ::  - [info]
Mon May  ::  - [info]  Waiting all logs to be applied..
Mon May  ::  - [info]   done.
Mon May  ::  - [debug]  Stopping slave IO/SQL thread on slave1(10.24.220.70:)..
Mon May  ::  - [debug]   done.
Mon May  ::  - [info] Getting new master's binlog name and position..
Mon May  ::  - [info]  log.:
Mon May  ::  - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='slave1 or 10.24.220.70', MASTER_PORT=, MASTER_AUTO_POSITION=, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Mon May  ::  - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: log., , 191f7a9f-ffa2-11e5-a825-00163e00242a:-
Mon May  ::  - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.
Mon May  ::  - [info] ** Finished master recovery successfully.
Mon May  ::  - [info] * Phase : Master Recovery Phase completed.
Mon May  ::  - [info]
Mon May  ::  - [info] * Phase : Slaves Recovery Phase..
Mon May  ::  - [info]
Mon May  ::  - [info]
Mon May  ::  - [info] * Phase 4.1: Starting Slaves in parallel..
Mon May  ::  - [info]
Mon May  ::  - [info] -- Slave recovery on host slave2(10.169.214.33:) started, pid: . Check tmp log /var/log/masterha/app1/slave2_3306_20160516091938.log if it takes time..
Mon May  ::  - [info]
Mon May  ::  - [info] Log messages from slave2 ...
Mon May  ::  - [info]
Mon May  ::  - [info]  Resetting slave slave2(10.169.214.33:) and starting replication from the new master slave1(10.24.220.70:)..
Mon May  ::  - [debug]  Stopping slave IO/SQL thread on slave2(10.169.214.33:)..
Mon May  ::  - [debug]   done.
Mon May  ::  - [info]  Executed CHANGE MASTER.
Mon May  ::  - [debug]  Starting slave IO/SQL thread on slave2(10.169.214.33:)..
Mon May  ::  - [debug]   done.
Mon May  ::  - [info]  Slave started.
Mon May  ::  - [info]  gtid_wait(191f7a9f-ffa2-11e5-a825-00163e00242a:-) completed on slave2(10.169.214.33:). Executed  events.
Mon May  ::  - [info] End of log messages from slave2.
Mon May  ::  - [info] -- Slave on host slave2(10.169.214.33:) started.
Mon May  ::  - [info] All new slave servers recovered successfully.
Mon May  ::  - [info]
Mon May  ::  - [info] * Phase : New master cleanup phase..
Mon May  ::  - [info]
Mon May  ::  - [info] Resetting slave info on the new master..
Mon May  ::  - [debug]  Clearing slave info..
Mon May  ::  - [debug]  Stopping slave IO/SQL thread on slave1(10.24.220.70:)..
Mon May  ::  - [debug]   done.
Mon May  ::  - [debug]  SHOW SLAVE STATUS shows new master does not replicate from anywhere. OK.
Mon May  ::  - [info]  slave1: Resetting slave info succeeded.
Mon May  ::  - [info] Master failover to slave1(10.24.220.70:) completed successfully.
Mon May  ::  - [debug]  Disconnected from slave1(10.24.220.70:)
Mon May  ::  - [debug]  Disconnected from slave2(10.169.214.33:)
Mon May  ::  - [info] 

----- Failover Report -----

app1: MySQL Master failover server1(10.24.220.232:) to slave1(10.24.220.70:) succeeded

Master server1(10.24.220.232:) is down!

Check MHA Manager logs at monitor for details.

Started manual(interactive) failover.
Selected slave1(10.24.220.70:) as a new master.
slave1(10.24.220.70:): OK: Applying all logs succeeded.
slave2(10.169.214.33:): OK: Slave started, replicating from slave1(10.24.220.70:)
slave1(10.24.220.70:): Resetting slave info succeeded.
Master failover to slave1(10.24.220.70:) completed successfully.