mysql主从之多线程复制

多线程复制

mysql 主从复制原理：

1. master 节点上的binlogdump 线程，在slave 与其正常连接的情况下，将binlog 发送到slave 上。

2. slave 节点的I/O Thread ，通过读取master 节点binlog 日志名称以及偏移量信息将其拷贝到本地relay log 日志文件。

3. slave 节点的SQL Thread，该线程读取relay log 日志信息，将在master 节点上提交的事务在本地回放，达到与主库数据保持一致的目的。

MySQL5.5 及以前的复制

一般主从复制有三个线程且都是单线程：

Binlog Dump（主） ‐‐> IO Thread（从） ‐‐> SQL Thread（从）。

而master 这边是通过并发线程提交，事务通过LSN 写入binlog；但是Slave 只有一个IO 线程和SQL 线程，是单线程，所以在业务大的情况下就很容易造成主从延时.

如果在MySQL 5.6 版本开启并行复制功能（slave_parallel_workers > 0），那么SQL 线程就变为了coordinator 线程，coordinator 线程主要负责以下两部分内容：

Coordinator+worker（多个）

若判断可以并行执行，那么选择worker 线程执行事务的二进制日志。

若判断不可以并行执行，如该操作是DDL，亦或者是事务跨schema 操作，则等待所有的worker 线程执行完成之后，再执行当前的日志。

这意味着coordinator 线程并不是仅将日志发送给worker 线程，自己也可以回放日志，但是所有可以并行的操作交付由worker 线程完成。

上述机制实现的基于schema 的并行复制，存在的问题是：

这样设计的并行复制效果并不高，如果用户实例仅有一个库，那么就无法实现并行回放，甚至性能会比原来的单线程更差，而单库多表是比多库多表更为常见的一种情形。

MySQL5.7 的MTS（Enhanced Muti‐threadedslaves）MySQL 5.7 引入了新的机制来实现并行复制，不再有基于库的并行复制限制，主要思想就是slave 服务器的回放与主机是一致的，即master 服务器上是怎么并行执行，slave 上就怎样进行并行回放。

mysql v5.7.2 进行了优化，增加了参数slave_parallel_type，参数有两个选项：

LOGICAL_CLOCK：基于逻辑时钟 ，可以在一个DATABASE 中并发执行relay log 事务DATABASE： 基于数据库，v5.6 默认是这个参数，改参数每个库只能一个线程；

slave‐parallel‐type，其可以配置的值有：

DATABASE：默认值，基于库的并行复制方式

LOGICAL_CLOCK：基于组提交的并行复制方式

从库:

mysql> show variables like 'slave_parallel%';
+------------------------+----------+
| Variable_name          | Value    |
+------------------------+----------+
| slave_parallel_type    | DATABASE |
| slave_parallel_workers |         |
+------------------------+----------+
mysql> stop slave;
mysql> set global slave_parallel_type='LOGICAL_CLOCK';
Query OK,  rows affected (0.00 sec)
mysql> set global slave_parallel_type='LOGICAL_CLOCK';
Query OK,  rows affected (0.00 sec)
mysql> set global slave_parallel_workers=;
Query OK,  rows affected (0.00 sec)
mysql> start slave;
Query OK,  rows affected (0.01 sec)
mysql> show processlist;
+-------+-------------+-----------+------+---------+------+--------------------------------------------------------+------------------+
| Id    | User        | Host      | db   | Command | Time | State                                                  | Info             |
+-------+-------------+-----------+------+---------+------+--------------------------------------------------------+------------------+
|  | root        | localhost | NULL | Query   |     | starting                                               | show processlist |
|  | system user |           | NULL | Connect |    | Waiting for master to send event                       | NULL             |
|  | system user |           | NULL | Connect |    | Slave has read all relay log; waiting for more updates | NULL             |
|  | system user |           | NULL | Connect |    | Waiting for an event from Coordinator                  | NULL             |
|  | system user |           | NULL | Connect |    | Waiting for an event from Coordinator                  | NULL             |
+-------+-------------+-----------+------+---------+------+--------------------------------------------------------+------------------+
mysql> show variables like 'slave_parallel%';
+------------------------+---------------+
| Variable_name          | Value         |
+------------------------+---------------+
| slave_parallel_type    | LOGICAL_CLOCK |
| slave_parallel_workers |              |
+------------------------+---------------+

master_info_repository=TABLE （开启MTS 功能后，会频繁更新master.info，设置为TABLE 减小开销）relay_log_info_repository=TABLE

slave_master_info 记录了首次同步master 的位置relay_log_recovery=ON （slave IO 线程crash，如果relay‐log损坏，则自动放弃所有未执行的relay‐log，重新从master 上获取日志，保证relay‐log 的完整性）

slave_preserve_commit_order=ON （保证提交的顺序性）在slave 上应用事务的顺序是无序的，和relay log 中记录的事务顺序不一样，这样数据一致性是无法保证的，为了保证事务是按照relay log 中记录的顺序来回放，就需要开启参数slave_preserve_commit_order。

虽然mysql5.7 添加MTS 后，虽然slave 可以并行应用relay log，但commit 部分仍然是顺序提交，其中可能会有等待的情况。

优化参数

[mysqld]
bind-address=0.0.0.0
port=3306
datadir=/data/mysql
socket=/data/mysql/mysql.sock
user=mysql
skip-name-resolve
slow_query_log=on
long_query_time=1
slow_query_log_file=/data/mysql/mysql-slow.log
innodb-file-per-table=1
innodb_flush_log_at_trx_commit = 2
log_warnings = 1
connect_timeout = 60
net_read_timeout = 120
performance_schema_max_table_instances = 400
server-id = 2
relay-log = relay-log
relay-log-index = relay-log.index
slave_parallel_type=LOGICAL_CLOCK
slave_parallel_workers=2
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON
slave_preserve_commit_order=ON
log_bin=mysql‐bin
log_slave_updates=ON

[mysqld_safe]
log-error=/data/mysql/mysqld.log
pid-file=/data/mysql/mysqld.pid

重启mysqld进入数据库

mysql> show variables like '%slave_parallel_type%';
+---------------------+---------------+
| Variable_name       | Value         |
+---------------------+---------------+
| slave_parallel_type | LOGICAL_CLOCK |
+---------------------+---------------+
mysql> show variables like '%slave_parallel_worker%';
+------------------------+-------+
| Variable_name          | Value |
+------------------------+-------+
| slave_parallel_workers |      |
+------------------------+-------+
mysql> show variables like '%master_info_repository%';
+------------------------+-------+
| Variable_name          | Value |
+------------------------+-------+
| master_info_repository | TABLE |
+------------------------+-------+
mysql> set global relay_log_info_repository='TABLE';
Query OK,  rows affected (0.00 sec)

mysql> show variables like '%relay_log_info_repository%';
+---------------------------+-------+
| Variable_name             | Value |
+---------------------------+-------+
| relay_log_info_repository | TABLE |
+---------------------------+-------+
mysql> show variables like '%relay_log_recovery%';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| relay_log_recovery | ON    |
+--------------------+-------+
 row in set (0.00 sec)

mysql> show variables like '%slave_preserve_commit_order%';
+-----------------------------+-------+
| Variable_name               | Value |
+-----------------------------+-------+
| slave_preserve_commit_order | ON    |
+-----------------------------+-------+

开启slave

mysql> start slave;
Query OK,  rows affected (0.00 sec)

mysql> show slave  status\G;
*************************** . row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.132.121
                  Master_User: replication
                  Master_Port:
                Connect_Retry:
              Master_Log_File: master-bin.
          Read_Master_Log_Pos:
               Relay_Log_File: relay-log.
                Relay_Log_Pos:
        Relay_Master_Log_File: master-bin.
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno:
                   Last_Error:
                 Skip_Counter:
          Exec_Master_Log_Pos:
              Relay_Log_Space:
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos:
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master:
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno:
                Last_IO_Error:
               Last_SQL_Errno:
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id:
                  Master_UUID: 6b00724f-a094-11e9-8f47-000c2991dd19
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay:
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count:
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position:
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version: 

主库写入数据

[root@master mysql]#mysql -uroot -p123456  -e 'use darren;insert into test values (1);
[root@master mysql]#mysql -uroot -p123456  -e 'use darren;insert into test values (1);
[root@master mysql]#mysql -uroot -p123456
mysql> mysql> show master status;
+-------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| master-bin. |  | | | |
+-------------------+----------+--------------+------------------+-------------------+

从库

mysql> use mysql;
Database changed
mysql> show tables;
+---------------------------+
| Tables_in_mysql           |
+---------------------------+
| columns_priv              |
| db                        |
| engine_cost               |
| event                     |
| func                      |
| general_log               |
| gtid_executed             |
| help_category             |
| help_keyword              |
| help_relation             |
| help_topic                |
| innodb_index_stats        |
| innodb_table_stats        |
| ndb_binlog_index          |
| plugin                    |
| proc                      |
| procs_priv                |
| proxies_priv              |
| server_cost               |
| servers                   |
| slave_master_info         |
| slave_relay_log_info      |
| slave_worker_info         |
| slow_log                  |
| tables_priv               |
| time_zone                 |
| time_zone_leap_second     |
| time_zone_name            |
| time_zone_transition      |
| time_zone_transition_type |
| user                      |
+---------------------------+
mysql> select * from slave_master_info;
+-----------------+-------------------+----------------+-----------------+-------------+---------------+------+---------------+-------------+--------+------------+----------+------------+---------+------------------------+-----------+------+--------------------+--------------------------------------+-------------+---------+-------------+-----------------------+--------------+-------------+
| Number_of_lines | Master_log_name   | Master_log_pos | Host            | User_name   | User_password | Port | Connect_retry | Enabled_ssl | Ssl_ca | Ssl_capath | Ssl_cert | Ssl_cipher | Ssl_key | Ssl_verify_server_cert | Heartbeat | Bind | Ignored_server_ids | Uuid                                 | Retry_count | Ssl_crl | Ssl_crlpath | Enabled_auto_position | Channel_name | Tls_version |
+-----------------+-------------------+----------------+-----------------+-------------+---------------+------+---------------+-------------+--------+------------+----------+------------+---------+------------------------+-----------+------+--------------------+--------------------------------------+-------------+---------+-------------+-----------------------+--------------+-------------+
|               | master-bin. |           | 192.168.132.121 | replication |        |  |             |            |        |            |          |            |         |                       |         |      |                   | 6b00724f-a094-11e9-8f47-000c2991dd19 |        |         |             |                      |              |             |
+-----------------+-------------------+----------------+-----------------+-------------+---------------+------+---------------+-------------+--------+------------+----------+------------+---------+------------------------+-----------+------+--------------------+--------------------------------------+-------------+---------+-------------+-----------------------+--------------+-------------+
mysql> select * from slave_relay_log_info;
+-----------------+--------------------+---------------+-------------------+----------------+-----------+-------------------+----+--------------+
| Number_of_lines | Relay_log_name     | Relay_log_pos | Master_log_name   | Master_log_pos | Sql_delay | Number_of_workers | Id | Channel_name |
+-----------------+--------------------+---------------+-------------------+----------------+-----------+-------------------+----+--------------+
|                | ./relay-log. |            | master-bin. |           |          |                  |   |              |
+-----------------+--------------------+---------------+-------------------+----------------+-----------+-------------------+----+--------------+

查看线程

+----+-------------+-----------+-------+---------+------+--------------------------------------------------------+-------------------+
| Id | User        | Host      | db    | Command | Time | State                                                  | Info              |
+----+-------------+-----------+-------+---------+------+--------------------------------------------------------+-------------------+
|   | root        | localhost | mysql | Query   |     | starting                                               | show  processlist |
|  | system user |           | NULL  | Connect |   | Waiting for master to send event                       | NULL              |
|  | system user |           | NULL  | Connect |   | Slave has read all relay log; waiting for more updates | NULL              |
|  | system user |           | NULL  | Connect |   | Waiting for an event from Coordinator                  | NULL              |
|  | system user |           | NULL  | Connect |   | Waiting for an event from Coordinator                  | NULL              |
+----+-------------+-----------+-------+---------+------+--------------------------------------------------------+-------------------+