mysql Last_SQL_Errno: 1197 Coordinator stopped because there were error(s) in the worker(s)问题处理

Last_SQL_Errno: 1197 Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log , end_log_pos 1103545303. See error log and/or performance_schema.replication_applier_status_by_worker table for more details

mysql报警主从不同步，查看发现是sql_running断了
问题分析：
有一个aiticls表大概有20W条数据开发要对它进行操作，提前在主库直接复制备份了一份
可能是数据量太大，一时无法处理

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: cmsmarketing.wai.prod.mysql.ws
                  Master_User: rep
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000046
          Read_Master_Log_Pos: 9219397
               Relay_Log_File: mysql-relay-bin.000199
                Relay_Log_Pos: 444825125
        Relay_Master_Log_File: mysql-bin.000045
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB: information_schema,performance_schema,undolog,for_nagios,information_schema,performance_schema,undolog,for_nagios
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table: information_schema.%,performance_schema.%,information_schema.%,performance_schema.%
                   Last_Errno: 1197
                   Last_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log , end_log_pos 1103545303. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 969112684
              Relay_Log_Space: 2692467349
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 1197
               Last_SQL_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log , end_log_pos 1103545303. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 16
                  Master_UUID: bda16924-6fe2-11e8-9376-005056912d4a
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State:
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp: 190830 00:31:48
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

mysql> show global variables like '%binlog_cache%';

+-----------------------+-----------+
| Variable_name         | Value     |
+-----------------------+-----------+
| binlog_cache_size     | 65536     |
| max_binlog_cache_size | 134217728 |
+-----------------------+-----------+
2 rows in set (0.01 sec)

mysql> show global status like '%binlog_cache%';
+-----------------------+---------+
| Variable_name         | Value   |
+-----------------------+---------+
| Binlog_cache_disk_use | 266     |
| Binlog_cache_use      | 3186562 |
+-----------------------+---------+
2 rows in set (0.00 sec)

mysql> set @@global.max_binlog_cache_size=500000000;
Query OK, 0 rows affected, 1 warning (0.00 sec)

stop slave
start slave

过了一会再次报错：
mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: cmsmarketing.wai.prod.mysql.ws
                  Master_User: rep
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000046
          Read_Master_Log_Pos: 10459073
               Relay_Log_File: mysql-relay-bin.000199
                Relay_Log_Pos: 444825125
        Relay_Master_Log_File: mysql-bin.000045
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB: information_schema,performance_schema,undolog,for_nagios,information_schema,performance_schema,undolog,for_nagios
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table: information_schema.%,performance_schema.%,information_schema.%,performance_schema.%
                   Last_Errno: 1197
                   Last_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log , end_log_pos 1469407519. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 969112684
              Relay_Log_Space: 2693707771
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 1197
               Last_SQL_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction 'ANONYMOUS' at master log , end_log_pos 1469407519. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 16
                  Master_UUID: bda16924-6fe2-11e8-9376-005056912d4a
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Waiting for workers to exit
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp: 190830 00:38:37
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

# 把处理的cache_size配置为1G，问题解决
>set global max_binlog_cache_size=10737418240;

另外一个机房的从库报错记录：

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: cms_marketing.wai.prod.mysql.eus
                  Master_User: rep
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000013
          Read_Master_Log_Pos: 7477932
               Relay_Log_File: mysql-relay-bin.000064
                Relay_Log_Pos: 36069896
        Relay_Master_Log_File: mysql-bin.000012
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB: information_schema,performance_schema,undolog,for_nagios,information_schema,performance_schema,undolog,for_nagios
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table: information_schema.%,performance_schema.%,information_schema.%,performance_schema.%
                   Last_Errno: 1197
                   Last_Error: Could not execute Write_rows event on table cms_marketing.articles_copy1; Multi-statement transaction required more than 'max_binlog_cache_size' bytes of storage; increase this mysqld variable and try again, Error_code: 1197; Writing one row to the row-based binary log failed, Error_code: 1534; handler error HA_ERR_RBR_LOGGING_FAILED; the event's master log mysql-bin.000012, end_log_pos 336399959
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 202043090
              Relay_Log_Space: 1922987726
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 1197
               Last_SQL_Error: Could not execute Write_rows event on table cms_marketing.articles_copy1; Multi-statement transaction required more than 'max_binlog_cache_size' bytes of storage; increase this mysqld variable and try again, Error_code: 1197; Writing one row to the row-based binary log failed, Error_code: 1534; handler error HA_ERR_RBR_LOGGING_FAILED; the event's master log mysql-bin.000012, end_log_pos 336399959
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 164
                  Master_UUID: 968e1d7f-957c-11e8-b650-00163e00923d
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State:
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp: 190830 00:47:25
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)