MySQL5.7 MTS work线程stack

复制现象是，slave线程状态正常，但是sql 线程不应用，所以delay越来越大，查看复制状态

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: System lock
                  Master_Host: 
                  Master_User: repl
                  Master_Port: xxxx
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000056
          Read_Master_Log_Pos: 384142784
               Relay_Log_File: relay-log.000014
                Relay_Log_Pos: 6517796
        Relay_Master_Log_File: mysql-bin.000053
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 6517623
              Relay_Log_Space: 4657584528
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 206184
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 102121028
                  Master_UUID: 7eeec54f-6464-11e7-a16a-20f17c59daa2
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Waiting for Slave Worker queue
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 7eeec54f-6464-11e7-a16a-20f17c59daa2:53936753-54857989
            Executed_Gtid_Set: 7eeec54f-6464-11e7-a16a-20f17c59daa2:1-54494096,
ff8f1aa3-7431-11e6-888c-1051722626c0:42306882-42306884
                Auto_Position: 1
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version:

---TRANSACTION 339088058, ACTIVE 43636 sec setting auto-inc lock
mysql tables in use 1, locked 1
1 lock struct(s), heap size 1136, 0 row lock(s), undo log entries 7842577
MySQL thread id 16, OS thread handle 140560924931840, query id 38813829 System lock
--------

mysql> select * from information_schema.innodb_trx\G
*************************** 1. row ***************************
                    trx_id: 339088058
                 trx_state: RUNNING
               trx_started: 2017-11-03 20:53:34
     trx_requested_lock_id: NULL
          trx_wait_started: NULL
                trx_weight: 7846747
       trx_mysql_thread_id: 16
                 trx_query: NULL
       trx_operation_state: setting auto-inc lock
         trx_tables_in_use: 1
         trx_tables_locked: 1
          trx_lock_structs: 1
     trx_lock_memory_bytes: 1136
           trx_rows_locked: 0
         trx_rows_modified: 7846746
   trx_concurrency_tickets: 0
       trx_isolation_level: READ COMMITTED
         trx_unique_checks: 1
    trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
 trx_adaptive_hash_latched: 0
 trx_adaptive_hash_timeout: 0
          trx_is_read_only: 0
trx_autocommit_non_locking: 0
1 row in set (0.00 sec)

mysql> show processlist;
+------+-------------+--------------------+-------------------+---------+--------+---------------------------------------+------------------+
| Id   | User        | Host               | db                | Command | Time   | State                                 | Info             |
+------+-------------+--------------------+-------------------+---------+--------+---------------------------------------+------------------+
|   14 | system user |                    | NULL              | Connect | 155776 | Waiting for master to send event      | NULL             |
|   15 | system user |                    | NULL              | Connect |      0 | Waiting for Slave Worker queue        | NULL             |
|   16 | system user |                    | NULL              | Connect | 209694 | System lock                           | NULL             |
|   17 | system user |                    | NULL              | Connect | 207404 | Waiting for an event from Coordinator | NULL             |
|   18 | system user |                    | NULL              | Connect | 213656 | Waiting for an event from Coordinator | NULL             |
|   19 | system user |                    | NULL              | Connect | 217230 | Waiting for an event from Coordinator | NULL             |
| 1519 | checksum    |                    |                   | Sleep   |      1 |                                       | NULL             |
| 1628 | root        | localhost          | NULL              | Query   |      0 | starting                              | show processlist |
+------+-------------+--------------------+-------------------+---------+--------+---------------------------------------+------------------+
8 rows in set (0.00 sec)

发现id号为16的work线程从2017-11-03 20:53:34卡到现在，查看正在应用的relay log

[root@mysql]# mysqlbinlog -vv --base64-output='decode-rows' --start-position=6517796 relay-log.000014 | more
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 6517796
#171101 23:40:01 server id 102121028  end_log_pos 6517688 CRC32 0xa6c47c8a      GTID    last_committed=2245     sequence_number=2246
SET @@SESSION.GTID_NEXT= '7eeec54f-6464-11e7-a16a-20f17c59daa2:54494097'/*!*/;
# at 6517861
#171101 21:41:33 server id 102121028  end_log_pos 6517773 CRC32 0x5abb9023      Query   thread_id=203424280     exec_time=0     error_code=0
SET TIMESTAMP=1509543693/*!*/;
SET @@session.pseudo_thread_id=203424280/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=1344798720/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=33/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
BEGIN
/*!*/;
# at 6517946
#171101 21:41:33 server id 102121028  end_log_pos 6517867 CRC32 0x474f27f3      Table_map: `nxin_qlw_business`.`wm_midwarehousebalance` mappe
d to number 58798
# at 6518040
#171101 21:41:33 server id 102121028  end_log_pos 6517993 CRC32 0x6c1671e8      Write_rows: table id 58798 flags: STMT_END_F
### INSERT INTO .....
......
# at 6518166
#171101 21:41:33 server id 102121028  end_log_pos 6518087 CRC32 0x4b319525      Table_map: `nxin_qlw_business`.`wm_midwarehousebalance` mappe
d to number 58798
# at 6518260
#171101 21:41:33 server id 102121028  end_log_pos 6518213 CRC32 0x2342008d      Write_rows: table id 58798 flags: STMT_END_F
--More--

16号work线程正在执行的事务是批量的insert，线上语句是 insert into ... select ... ，是一个大的事务。这种情况复制会很慢

解决方法是

过滤正在操作的表不再复制

stop slave;
# mysql 5.7 新增 在线filter
CHANGE REPLICATION FILTER REPLICATE_WILD_IGNORE_TABLE = ('db.t');

start slave

此时再看sql 线程应用正常，待主从无延迟后，恢复db.t表

关闭过滤

stop slave ;
CHANGE REPLICATION FILTER REPLICATE_WILD_IGNORE_TABLE = ();

---恢复内容结束---