MySQL 5.7传统复制到GTID在线切换（一主一从）

Preface

 

    Classic replication is commonly used in previous version of MySQL.It's really tough in managing them when our replications get into failures.Many new features are also depend on GTID.So it's urgent to use GTID replication as soon as possible.I'm gonna to demenstrate how to change classic replication to GTID replication online with two servers.Here we go.

 

Framework

Hostname	IP/Port	Identity	OS Version	MySQL Version	GTID Mode	Binlog Format
zlm2	192.168.1.101/3306	master	CentOS 7.0	5.7.21	off	row
zlm3	192.168.1.102/3306	slave	CentOS 7.0	5.7.21	off	row

 

Procedure

 

Check  parameter "gtid_mode" and is "OFF" on both master and slave in the replication group.

 //Master
 (root@localhost mysql3306.sock)[(none)]>show variables like 'gtid_mode';
 +---------------+-------+
 | Variable_name | Value |
 +---------------+-------+
 | gtid_mode     | OFF   |
 +---------------+-------+
  row in set (0.01 sec

 //Slave
 (root@localhost mysql3306.sock)[(none)]>show variables like 'gtid_mode';
 +---------------+-------+
 | Variable_name | Value |
 +---------------+-------+
 | gtid_mode     | OFF   |
 +---------------+-------+
  row in set (0.00 sec)

Execute sysbench to generate some transactions continuously on master.

 [root@zlm2 :: ~/sysbench-1.0/src/lua]
 #sysbench oltp_insert.lua --mysql-host=192.168.1.101 --mysql-port= --mysql-user=zlm --mysql-password=zlmzlm --mysql-db=sysbench --tables= --table-size= --mysql-storage-engine=innodb prepare
 sysbench 1.0. (using bundled LuaJIT 2.1.-beta2)

 Creating table 'sbtest1'...
 Inserting  records into 'sbtest1'
 Creating a secondary index on 'sbtest1'...
 Creating table 'sbtest2'...
 Inserting  records into 'sbtest2'
 Creating a secondary index on 'sbtest2'...
 ...

 [root@zlm2 :: ~/sysbench-1.0/src/lua]
 #sysbench oltp_insert.lua --mysql-host=192.168.1.101 --mysql-port= --mysql-user=zlm --mysql-password=zlmzlm --mysql-db=sysbench --threads= --time= --report-interval= --rand-type=uniform run
 sysbench 1.0. (using bundled LuaJIT 2.1.-beta2)

 Running the test with following options:
 Number of threads:
 Report intermediate results every  second(s)
 Initializing random number generator from current time

 Initializing worker threads...

 Threads started!

 [ 60s ] thds:  tps: 1623.71 qps: 1623.71 (r/w/o: 0.00/1623.71/0.00) lat (ms,%): 2.97 err/s: 0.00 reconn/s: 0.00
 [ 120s ] thds:  tps: 1844.96 qps: 1844.96 (r/w/o: 0.00/1844.96/0.00) lat (ms,%): 2.61 err/s: 0.00 reconn/s: 0.00
 [ 180s ] thds:  tps: 1894.37 qps: 1894.37 (r/w/o: 0.00/1894.37/0.00) lat (ms,%): 2.61 err/s: 0.00 reconn/s: 0.00
 ...

 //Check the output of processlist.
 (root@localhost mysql3306.sock)[(none)]>show processlist;
 +----+------+------------+----------+-------------+------+---------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
 | Id | User | Host       | db       | Command     | Time | State                                                         | Info                                                                                                 |
 +----+------+------------+----------+-------------+------+---------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
 |  | root | localhost  | NULL     | Query       |     | starting                                                      | show processlist                                                                                     |
 |  | repl | zlm3: | NULL     | Binlog Dump |   | Master has sent all binlog to slave; waiting for more updates | NULL                                                                                                 |
 |  | zlm  | zlm2: | sysbench | Query       |     | update                                                        | INSERT INTO sbtest1 (id, k, c, pad) VALUES (, , '57837919367-24452778030-14591605115-8049012633 |
 |  | zlm  | zlm2: | sysbench | Query       |     | update                                                        | INSERT INTO sbtest1 (id, k, c, pad) VALUES (, , '45087463438-93604980565-67881991526-9944080034 |
 |  | zlm  | zlm2: | sysbench | Query       |     | update                                                        | INSERT INTO sbtest1 (id, k, c, pad) VALUES (, , '01822437471-94427682076-39418270545-9867829936 |
 +----+------+------------+----------+-------------+------+---------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  rows in set (0.00 sec)

Make sure that the classic replication is working normally on slave.

 (root@localhost mysql3306.sock)[(none)]>show slave status\G
 *************************** . row ***************************
                Slave_IO_State: Waiting for master to send event
                   Master_Host: 192.168.1.101
                   Master_User: repl
                   Master_Port:
                 Connect_Retry:
               Master_Log_File: mysql-bin.
           Read_Master_Log_Pos:
                Relay_Log_File: relay-bin.
                 Relay_Log_Pos:
         Relay_Master_Log_File: mysql-bin.
              Slave_IO_Running: Yes
             Slave_SQL_Running: Yes
               Replicate_Do_DB:
           Replicate_Ignore_DB:
            Replicate_Do_Table:
        Replicate_Ignore_Table:
       Replicate_Wild_Do_Table:
   Replicate_Wild_Ignore_Table:
                    Last_Errno:
                    Last_Error:
                  Skip_Counter:
           Exec_Master_Log_Pos:
               Relay_Log_Space:
               Until_Condition: None
                Until_Log_File:
                 Until_Log_Pos:
            Master_SSL_Allowed: No
            Master_SSL_CA_File:
            Master_SSL_CA_Path:
               Master_SSL_Cert:
             Master_SSL_Cipher:
                Master_SSL_Key:
         Seconds_Behind_Master:
 Master_SSL_Verify_Server_Cert: No
                 Last_IO_Errno:
                 Last_IO_Error:
                Last_SQL_Errno:
                Last_SQL_Error:
   Replicate_Ignore_Server_Ids:
              Master_Server_Id:
                   Master_UUID: 1b7181ee-6eaf-11e8-998e-080027de0e0e
              Master_Info_File: mysql.slave_master_info
                     SQL_Delay:
           SQL_Remaining_Delay: NULL
       Slave_SQL_Running_State: System lock
            Master_Retry_Count:
                   Master_Bind:
       Last_IO_Error_Timestamp:
      Last_SQL_Error_Timestamp:
                Master_SSL_Crl:
            Master_SSL_Crlpath:
            Retrieved_Gtid_Set:
             Executed_Gtid_Set:
                 Auto_Position: 0 //This means we are using the classic replication now.
          Replicate_Rewrite_DB:
                  Channel_Name:
            Master_TLS_Version:
  row in set (0.00 sec)

Change the parameter "enforce_gitd_consistency" to "warn" on both master and slave.

 //Master
 (root@localhost mysql3306.sock)[(none)]>set @@global.enforce_gtid_consistency=warn;
 Query OK,  rows affected (0.13 sec)

 (root@localhost mysql3306.sock)[(none)]>select @@global.enforce_gtid_consistency;
 +-----------------------------------+
 | @@global.enforce_gtid_consistency |
 +-----------------------------------+
 | WARN                              |
 +-----------------------------------+
  row in set (0.06 sec)

 //Error log of master
 --13T07::56.877416+:  [Note] Changed ENFORCE_GTID_CONSISTENCY from OFF to WARN.
 --13T07::15.748645+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 8825ms. The settings might not be optimal. (flushed= and evicted=, during the time.)

 //Slave
 (root@localhost mysql3306.sock)[(none)]>set @@global.enforce_gtid_consistency=warn;
 Query OK,  rows affected (0.49 sec)

 (root@localhost mysql3306.sock)[(none)]>select @@global.enforce_gtid_consistency;
 +-----------------------------------+
 | @@global.enforce_gtid_consistency |
 +-----------------------------------+
 | WARN                              |
 +-----------------------------------+
  row in set (1.35 sec)

 //Error log of slave
 --13T07::02.556232+:  [Note] Changed ENFORCE_GTID_CONSISTENCY from OFF to WARN.

 //Make sure there's no warning messages on both master and slave.

Change the parameter "enforce_gitd_consistency" to "on" on both master and slave.

 //Master
 (root@localhost mysql3306.sock)[(none)]>set @@global.enforce_gtid_consistency=on;
 Query OK,  rows affected (0.00 sec)

 (root@localhost mysql3306.sock)[(none)]>select @@global.enforce_gtid_consistency;
 +-----------------------------------+
 | @@global.enforce_gtid_consistency |
 +-----------------------------------+
 | ON                                |
 +-----------------------------------+
  row in set (0.00 sec)

 //Slave
 (root@localhost mysql3306.sock)[(none)]>set @@global.enforce_gtid_consistency=on;
 Query OK,  rows affected (0.03 sec)

 (root@localhost mysql3306.sock)[(none)]>select @@global.enforce_gtid_consistency;
 +-----------------------------------+
 | @@global.enforce_gtid_consistency |
 +-----------------------------------+
 | ON                                |
 +-----------------------------------+
  row in set (0.00 sec)

Change  the parameter "gtid_mode" to "off_permissive"  on both master and slave.

 //Master
 (root@localhost mysql3306.sock)[(none)]>set @@globa.gtid_mode=off_permissive;
 Query OK,  rows affected (0.72 sec)

 (root@localhost mysql3306.sock)[(none)]>select @@global.gtid_mode;
 +--------------------+
 | @@global.gtid_mode |
 +--------------------+
 | OFF_PERMISSIVE     |
 +--------------------+
  row in set (0.01 sec)

 //Error log of master
 --13T07::56.877416+:  [Note] Changed ENFORCE_GTID_CONSISTENCY from OFF to WARN.
 --13T07::15.748645+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 8825ms. The settings might not be optimal. (flushed= and evicted=, during the time.)
 --13T07::38.472436+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 8569ms. The settings might not be optimal. (flushed= and evicted=, during the time.)
 --13T07::03.886312+:  [Note] Changed ENFORCE_GTID_CONSISTENCY from WARN to ON.
 --13T07::04.137251+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 5067ms. The settings might not be optimal. (flushed= and evicted=, during the time.)
 --13T07::39.586306+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 5394ms. The settings might not be optimal. (flushed= and evicted=, during the time.)
 --13T07::38.441594+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 4927ms. The settings might not be optimal. (flushed= and evicted=, during the time.)
 --13T07::19.070954+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 4539ms. The settings might not be optimal. (flushed= and evicted=, during the time.)
 --13T07::20.930564+:  [Note] Changed GTID_MODE from OFF to OFF_PERMISSIVE.
 --13T07::36.490470+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 4602ms. The settings might not be optimal. (flushed= and evicted=, during the time.)

 //Slave
 (root@localhost mysql3306.sock)[(none)]>set @@global.gtid_mode=off_permissive;
 Query OK,  rows affected (3.02 sec)

 (root@localhost mysql3306.sock)[(none)]>select @@global.gtid_mode;
 +--------------------+
 | @@global.gtid_mode |
 +--------------------+
 | OFF_PERMISSIVE     |
 +--------------------+
  row in set (0.00 sec)

 //Error log of slave
 --13T07::02.556232+:  [Note] Changed ENFORCE_GTID_CONSISTENCY from OFF to WARN.
 --13T07::22.628014+:  [Note] Changed ENFORCE_GTID_CONSISTENCY from WARN to ON.
 --13T07::33.136288+:  [Note] Aborted connection  to db: 'unconnected' user: 'root' host: 'localhost' (Got timeout reading communication packets)
 --13T07::27.360767+:  [Note] Changed GTID_MODE from OFF to OFF_PERMISSIVE.
 --13T07::39.972826+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 10489ms. The settings might not be optimal. (flushed= and evicted=, during the time.)

Change  the parameter "gtid_mode" to "on_permissive"  on both master and slave.

 //Master
 (root@localhost mysql3306.sock)[(none)]>set @@global.gtid_mode=on_permissive;
 Query OK,  rows affected (3.26 sec)

 (root@localhost mysql3306.sock)[(none)]>select @@global.gtid_mode;
 +--------------------+
 | @@global.gtid_mode |
 +--------------------+
 | ON_PERMISSIVE      |
 +--------------------+
  row in set (0.00 sec)

 //Error log of master
 --13T07::16.796632+:  [Note] Changed GTID_MODE from OFF_PERMISSIVE to ON_PERMISSIVE.
 --13T07::20.034425+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 4954ms. The settings might not be optimal. (flushed= and evicted=, during the time.)

 //Slave
 (root@localhost mysql3306.sock)[(none)]>set @@global.gtid_mode=on_permissive;
 Query OK,  rows affected (2.22 sec)

 (root@localhost mysql3306.sock)[(none)]>select @@global.gtid_mode;
 +--------------------+
 | @@global.gtid_mode |
 +--------------------+
 | ON_PERMISSIVE      |
 +--------------------+
  row in set (0.06 sec)

 //Error log of slave
 --13T07::57.921081+:  [Note] Changed GTID_MODE from OFF_PERMISSIVE to ON_PERMISSIVE.
 --13T07::03.109628+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 5853ms. The settings might not be optimal. (flushed= and evicted=, during the time.)

 //I'm afraid it's better to execut "set gtid_mode=on_permissive;" on slave first for best practice even though sometimes it's not obliged to do that.

Make sure all the binlogs generated by classic replication has been disappeared on both master and slave by checking parameter 'ongoing_anonymous_transaction_count' whether it returns "0".

 //Master
 (root@localhost mysql3306.sock)[(none)]>show status like 'ongoing_anonymous_transaction_count';
 +-------------------------------------+-------+
 | Variable_name                       | Value |
 +-------------------------------------+-------+
 | Ongoing_anonymous_transaction_count |      |
 +-------------------------------------+-------+
  row in set (0.66 sec)

 //Slave
 (root@localhost mysql3306.sock)[(none)]>show status like 'ongoing_anonymous_transaction_count';
 +-------------------------------------+-------+
 | Variable_name                       | Value |
 +-------------------------------------+-------+
 | Ongoing_anonymous_transaction_count |      |
 +-------------------------------------+-------+
  row in set (3.34 sec)

 //The value of 'ongoing_anonymous_transaction_count' become "0" what means there arn't non-gtid events in binlogs anymore.Therefore,we can do the last step,that is,to change the "gtid_mode" to "on".

Change the parameter "gtid_mode" to "on" on both master and slave.

 //Master
 (root@localhost mysql3306.sock)[(none)]>select @@global.gtid_mode;
 +--------------------+
 | @@global.gtid_mode |
 +--------------------+
 | ON                 |
 +--------------------+
  row in set (0.00 sec)

 //Error log of master
 --13T08::59.853460+:  [Note] Changed GTID_MODE from ON_PERMISSIVE to ON.
 --13T08::01.804678+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 6035ms. The settings might not be optimal. (flushed= and evicted=, during the time.)
 --13T08::56.202081+:  [Note] Aborted connection  to db: 'unconnected' user: 'repl' host: 'zlm3' (Failed on my_net_write())

 //Slave
 (root@localhost mysql3306.sock)[(none)]>set @@global.gtid_mode=on;
 ERROR  (HY000): MySQL server has gone away
 No connection. Trying to reconnect...
 Connection id:
 Current database: *** NONE ***

 //It's stuck here.Oh my!!!

 //Check the error log of slave see what has happened.
 --13T08::49.070915+:  [ERROR] Disk is full writing './relay-bin.000044' (Errcode:  - No space left on device). Waiting for someone to free space...
 --13T08::49.070948+:  [ERROR] Retry in  secs. Message reprinted in  secs
 --13T08::49.104353+:  [ERROR] Disk is full writing '/data/mysql/mysql3306/logs/mysql-bin.000011' (Errcode:  - No space left on device). Waiting for someone to free space...
 --13T08::49.104382+:  [ERROR] Retry in  secs. Message reprinted in  secs
 --13T08::51.712891+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 4001ms. The settings might not be optimal. (flushed= and evicted=, during the time.)
 --13T08::00.346384+:  [Note] InnoDB: page_cleaner: 1000ms intended loop took 7634ms. The settings might not be optimal. (flushed= and evicted=, during the time.)

 //It shows "[ERROR] Disk is full writing ... ".The test tables have been inserted too many data.

 [root@zlm3 :: ~]
 #df -h
 Filesystem               Size  Used Avail Use% Mounted on
 /dev/mapper/centos-root  .4G  .4G   20K % /  //The root directory is full.
 devtmpfs                 488M       488M   % /dev
 tmpfs                    497M       497M   % /dev/shm
 tmpfs                    497M  6.6M  491M   % /run
 tmpfs                    497M       497M   % /sys/fs/cgroup
 /dev/sda1                497M  118M  379M  % /boot
 none                      87G   80G  .1G  % /vagrant

 //Unfortunately,the disk on salve has been writen fully.

    Notwithstanding the demonstrating was interupted accidentally but the porcedure of changing classic replication to GTID replicatioin is correct.Onlyif  the slave has finished to change the "gtid_mode" to "on",the implementing is accomplished.

    One more thing need to do is to modify your "my.cnf" file to make them support GTID replication after restarting your mysqld process.Make sure these three parameters:"enforce_gtid_consistency=on","gtid_mode=on","log_slave_updates=on" are right in your configuration file "my.cnf".

    The last thing to do in this case is to stop slave,set "master_auto_position=1"  and start slave again.I'm not going to do these last steps here('cause the environment has been destroyed.oops!).

 

Some error masseages may occur if you don't implement follow the sequence above.

 //The output of "show salve status\G"
 Last_IO_Errno:
                 Last_IO_Error: The replication receiver thread cannot start because the master has GTID_MODE = ON and this server has GTID_MODE = OFF

 //You cannot modify "gtid_mode" to "on" directly.
 (root@localhost mysql3306.sock)[(none)]>set @@global.gtid_mode=on;
 ERROR  (HY000): The value of @@GLOBAL.GTID_MODE can only be changed one step at a time: OFF <-> OFF_PERMISSIVE <-> ON_PERMISSIVE <-> ON. Also note that this value must be stepped up or down simultaneously on all servers. See the Manual for instructions.

Summary

• GTID replication is the best practice in MySQL replicaiton now,especially in 5.7 version above.More and more new good features are relies on GTID,such as "Group Replication","Group Commit","Parallel Replication",etc.
• We'd better replace all the classic replication to GTID replication in our product environment in order to get more benifits and work efficiently.
• Chang classic replicaiton to GTID replicaiton online should follow the order of "off -> off_permissive -> on_permissive -> on" and execute them on both master and slaves.
• Notice that change online is only support on MySQL 5.7.6 and above.