Percona-Tookit工具包之pt-table-checksum

 

Preface

 

    The master-slave replication is commonly used in our product evironment.On account of network lag or replicaton mode,slaves probablly be delayed with master.How to solve the consistency of tables between master and slaves?pt-table-checksum is a suitable tool which you can use efficiently.I'm gonna use the tool to do some tests today.

 

Introduce

 

    pt-table-checksum is a memeber of Percona-Toolkit,it's used to check consistency of tables in online replication environment by execute querries on master(put results into a replica called chckesum table).It will indicate whether there're some inconsistent tables bewteen master and slaves by output on screen with parameter "--print".

 

Procedure

 

1.Comman parameter introduce.

 Connection relevent:
 --ask-pass -- Ask user to input a password when executing.
 -h hostname/ip
 -u username
 -p password(specify the password in command line)
 -P port
 -S socket
 -D database
 -t tables

 Important parameter:
 --create-replicate-table -- Create database and table(checksum table) mentioned in "--replicate".
 --no-check-binlog-format -- Don't check binlog format on all server.
 --recursion-method -- Specify the prefferd mode to find slaves if you've got multiple slaves.
 --replicate -- Specify the table(default is "percona.checksums") into which the results will be write.
 --replicate-check-only -- Check consistency on replica without executing checksum queries on master.Furthermore,it only checks riplica for differences found by previous checksuming.

 Output relevent:
 --explain -- Show without really execute checksum querries.
 --progress -- Print progress report(default  seconds).
 --quite -- Print only important informations on screen(will disable "--progress").

 Safety relevent:
 --no-check-slave-tables -- Only if you're confirmed that all tables on slave is the same with master's.Then you can set it to avoid breaking of replication when executing pt-table-checksum.
 --check-replication-filters -- Don't checksum if any replication filters are set on target replica.
 --chunk-size-limit -- Limit the chunk size to avoid performance issues.

2.Examples.

 Master:
 (root@localhost mysql3306.sock)[zlm]::>show tables;
 +----------------+
 | Tables_in_zlm  |
 +----------------+
 | test_ddl       |
 | test_ddl_no_pk |
 | test_innodb    |
 | test_myisam    |
 +----------------+
  rows in set (0.00 sec)

 Slave:
 (root@localhost mysql3306.sock)[zlm]::>show tables;
 +----------------+
 | Tables_in_zlm  |
 +----------------+
 | t1             |
 | t2             |
 | t3             |
 | test_ddl       |
 | test_ddl_no_pk |
 | test_innodb    |
 | test_myisam    |
 +----------------+
  rows in set (0.00 sec)

 ###My pt-table-checksum is installed on master server.Execute the command below on master.###

 [root@zlm2 :: ~]
 #pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --check-slave-tables -hzlm2 -P3306 -uroot --ask-pass
 Enter MySQL password:
 -21T10:: DBI connect(';host=zlm2;port=3306;mysql_read_default_group=client','root',...) failed: Access denied for user 'root'@'localhost' (using password: YES) at /usr/bin/pt-table-checksum line .

 [root@zlm2 :: ~]
 #pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --check-slave-tables -hlocalhost -P3306 -uroot --ask-pass
 Enter MySQL password:
 Checking if all tables can be checksummed ...
 Starting checksum ...
 Cannot connect to P=,h=zlm3,p=...,u=root -- It's due to the "root" user cannot login with TCP/IP mode.
 Diffs cannot be detected because no slaves were found.  Please read the --recursion-method documentation for information.
 *******************************************************************
  Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
  is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
  possibly with SSL_ca_file|SSL_ca_path for verification.
  If you really don't want to verify the certificate and keep the
  connection open to Man-In-The-Middle attacks please set
  SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
 *******************************************************************
   at /usr/bin/pt-table-checksum line .
 *******************************************************************
  Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
  is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
  possibly with SSL_ca_file|SSL_ca_path for verification.
  If you really don't want to verify the certificate and keep the
  connection open to Man-In-The-Middle attacks please set
  SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
 *******************************************************************
   at /usr/bin/pt-table-checksum line .

 # A software update is available:
             TS ERRORS  DIFFS     ROWS  DIFF_ROWS  CHUNKS SKIPPED    TIME TABLE
 -21T10::                                               0.006 mysql.columns_priv
 -21T10::                                               0.012 mysql.db
 -21T10::                                               0.010 mysql.engine_cost
 -21T10::                                               0.008 mysql.event
 -21T10::                                               0.009 mysql.func
 -21T10::                                              0.008 mysql.help_category
 -21T10::                                             0.010 mysql.help_keyword
 -21T10::                                            0.011 mysql.help_relation
 -21T10::                                             0.032 mysql.help_topic
 -21T10::                                               0.008 mysql.ndb_binlog_index
 -21T10::                                               0.011 mysql.plugin
 -21T10::                                              0.012 mysql.proc
 -21T10::                                               0.009 mysql.procs_priv
 -21T10::                                               0.010 mysql.proxies_priv
 -21T10::                                               0.009 mysql.server_cost
 -21T10::                                               0.010 mysql.servers
 -21T10::                                               0.010 mysql.tables_priv
 -21T10::                                               0.009 mysql.time_zone
 -21T10::                                               0.009 mysql.time_zone_leap_second
 -21T10::                                               0.009 mysql.time_zone_name
 -21T10::                                               0.009 mysql.time_zone_transition
 -21T10::                                               0.008 mysql.time_zone_transition_type
 -21T10::                                               0.007 mysql.user
 -21T10::                                               0.011 sys.sys_config
 -21T10::                                               0.008 zlm.test_ddl
 -21T10::                                               0.008 zlm.test_ddl_no_pk
 -21T10::                                               0.009 zlm.test_innodb
 -21T10::                                               0.009 zlm.test_myisam

 ###See the detail of checksum table.###
 (root@localhost mysql3306.sock)[zlm]::>show tables;
 +----------------+
 | Tables_in_zlm  |
 +----------------+
 | checksums      |  -- the table checksums was created.
 | test_ddl       |
 | test_ddl_no_pk |
 | test_innodb    |
 | test_myisam    |
 +----------------+
  rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]::>show create table checksums\G
 *************************** . row ***************************
        Table: checksums
 Create Table: CREATE TABLE `checksums` (
   `db` char() NOT NULL,
   `tbl` char() NOT NULL,
   `chunk` int() NOT NULL,
   `chunk_time` float DEFAULT NULL,
   `chunk_index` varchar() DEFAULT NULL,
   `lower_boundary` text,
   `upper_boundary` text,
   `this_crc` char() NOT NULL,
   `this_cnt` int() NOT NULL,
   `master_crc` char() DEFAULT NULL,
   `master_cnt` int() DEFAULT NULL,
   `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
   PRIMARY KEY (`db`,`tbl`,`chunk`),
   KEY `ts_db_tbl` (`ts`,`db`,`tbl`)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8
  row in set (0.00 sec)

 ###Check slave hosts information.###
 (root@localhost mysql3306.sock)[zlm]::>show slave hosts;
 +-----------+------+------+-----------+--------------------------------------+
 | Server_id | Host | Port | Master_id | Slave_UUID                           |
 +-----------+------+------+-----------+--------------------------------------+
 |    |      |  |    | 5c77c31b-4add-11e8-81e2-080027de0e0e |
 +-----------+------+------+-----------+--------------------------------------+
  row in set (0.00 sec)

 ###Change another user 'repl'@'192.168.1.%' and grant all privileges to it.###
 (root@localhost mysql3306.sock)[zlm]::>grant all privileges on *.* to 'repl'@'192.168.1.%';
 Query OK,  rows affected (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]::>show grants for 'repl'@'192.168.1.%';
 +-----------------------------------------------------+
 | Grants for repl@192.168..%                         |
 +-----------------------------------------------------+
 | GRANT ALL PRIVILEGES ON *.* TO 'repl'@'192.168.1.%' |
 +-----------------------------------------------------+
  row in set (0.00 sec)

 ###Execute pt-table-checksum again.###
 [root@zlm2 :: ~]
 #pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --check-slave-tables -h192.168.1. -P3306 -urepl --ask-pass
 Enter MySQL password:
 Checking if all tables can be checksummed ...
 Starting checksum ...
 *******************************************************************
  Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
  is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
  possibly with SSL_ca_file|SSL_ca_path for verification.
  If you really don't want to verify the certificate and keep the
  connection open to Man-In-The-Middle attacks please set
  SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
 *******************************************************************
   at /usr/bin/pt-table-checksum line .
 *******************************************************************
  Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
  is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
  possibly with SSL_ca_file|SSL_ca_path for verification.
  If you really don't want to verify the certificate and keep the
  connection open to Man-In-The-Middle attacks please set
  SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
 *******************************************************************
   at /usr/bin/pt-table-checksum line .

 # A software update is available:
             TS ERRORS  DIFFS     ROWS  DIFF_ROWS  CHUNKS SKIPPED    TIME TABLE
 -21T11::                                               0.018 mysql.columns_priv
 -21T11::                                               0.020 mysql.db
 -21T11::                                               0.016 mysql.engine_cost
 -21T11::                                               0.017 mysql.event
 -21T11::                                               0.014 mysql.func
 -21T11::                                              0.018 mysql.help_category
 -21T11::                                             0.016 mysql.help_keyword
 -21T11::                                            0.015 mysql.help_relation
 -21T11::                                             0.019 mysql.help_topic
 -21T11::                                               0.013 mysql.ndb_binlog_index
 -21T11::                                               0.012 mysql.plugin
 -21T11::                                              0.015 mysql.proc
 -21T11::                                               0.012 mysql.procs_priv
 -21T11::                                               0.016 mysql.proxies_priv
 -21T11::                                               0.015 mysql.server_cost
 -21T11::                                               0.015 mysql.servers
 -21T11::                                               0.014 mysql.tables_priv
 -21T11::                                               0.013 mysql.time_zone
 -21T11::                                               0.013 mysql.time_zone_leap_second
 -21T11::                                               0.015 mysql.time_zone_name
 -21T11::                                               0.015 mysql.time_zone_transition
 -21T11::                                               0.012 mysql.time_zone_transition_type
 -21T11::                                               0.013 mysql.user
 -21T11::                                               0.015 sys.sys_config
 -21T11::                                               0.014 zlm.test_ddl
 -21T11::                                               0.015 zlm.test_ddl_no_pk
 -21T11::                                               0.021 zlm.test_innodb
 -21T11::                                               0.016 zlm.test_myisam

 ###Check whether there're records in "checksums" table or not on slave.###
 (root@localhost mysql3306.sock)[zlm]::>select * from checksums where this_cnt<>master_cnt;
 +-------+------+-------+------------+-------------+----------------+----------------+----------+----------+------------+------------+---------------------+
 | db    | tbl  | chunk | chunk_time | chunk_index | lower_boundary | upper_boundary | this_crc | this_cnt | master_crc | master_cnt | ts                  |
 +-------+------+-------+------------+-------------+----------------+----------------+----------+----------+------------+------------+---------------------+
 | mysql | proc |      |   0.001277 | NULL        | NULL           | NULL           | 9e5a007c |        | 4e0f05d9   |          | -- :: |
 | mysql | user |      |   0.000667 | NULL        | NULL           | NULL           | 7de55b47 |        | 587dfc7    |           | -- :: |
 +-------+------+-------+------------+-------------+----------------+----------------+----------+----------+------------+------------+---------------------+
  rows in set (0.00 sec)

Supplement

 

1.Drop a table "testl_ddl" on slave when master-slave replication is normally running.

 (root@localhost mysql3306.sock)[zlm]::>show tables;
 +----------------+
 | Tables_in_zlm  |
 +----------------+
 | checksums      |
 | t1             |
 | t2             |
 | t3             |
 | test_ddl       |
 | test_ddl_no_pk |
 | test_innodb    |
 | test_myisam    |
 +----------------+
  rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]::>drop table test_ddl;
 Query OK,  rows affected (0.00 sec)

2.Execute pt-table-checksum again.

 [root@zlm2 :: ~]
 #pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --check-slave-tables -h192.168.1. -P3306 -urepl --ask-pass
 Enter MySQL password:
 Checking if all tables can be checksummed ...
 Starting checksum ...
             TS ERRORS  DIFFS     ROWS  DIFF_ROWS  CHUNKS SKIPPED    TIME TABLE
 -22T04::                                               0.012 mysql.columns_priv
 -22T04::                                               0.015 mysql.db
 -22T04::                                               0.013 mysql.engine_cost
 -22T04::                                               0.016 mysql.event
 -22T04::                                               0.014 mysql.func
 -22T04::                                              0.011 mysql.help_category
 -22T04::                                             0.015 mysql.help_keyword
 -22T04::                                            0.017 mysql.help_relation
 -22T04::                                             0.018 mysql.help_topic
 -22T04::                                               0.014 mysql.ndb_binlog_index
 -22T04::                                               0.013 mysql.plugin
 -22T04::                                              0.013 mysql.proc
 -22T04::                                               0.013 mysql.procs_priv
 -22T04::                                               0.014 mysql.proxies_priv
 -22T04::                                               0.012 mysql.server_cost
 -22T04::                                               0.012 mysql.servers
 -22T04::                                               0.014 mysql.tables_priv
 -22T04::                                               0.014 mysql.time_zone
 -22T04::                                               0.013 mysql.time_zone_leap_second
 -22T04::                                               0.012 mysql.time_zone_name
 -22T04::                                               0.014 mysql.time_zone_transition
 -22T04::                                               0.014 mysql.time_zone_transition_type
 -22T04::                                               0.015 mysql.user
 -22T04::                                               0.012 sys.sys_config
 -22T04:: Skipping table zlm.test_ddl because it has problems on these replicas:
 Table zlm.test_ddl does not exist on replica zlm3
 This can break replication.  If you understand the risks, specify --no-check-slave-tables to disable this check.
 -22T04:: Error checksumming table zlm.test_ddl: Error getting row count estimate of table zlm.test_ddl on replica zlm3: DBD::mysql::db selectrow_hashref failed: Table 'zlm.test_ddl' doesn't exist [for Statement "EXPLAIN SELECT * FROM `zlm`.`test_ddl` WHERE 1=1"] at /usr/bin/pt-table-checksum line 6823.

 -22T04::                                               0.002 zlm.test_ddl
 -22T04::                                               0.014 zlm.test_ddl_no_pk
 -22T04::                                               0.014 zlm.test_innodb
 -22T04::                                               0.014 zlm.test_myisam

 ###This will always lead to "ERROR 1146" what is shown below.###
                    Last_Errno:
                    Last_Error: Error 'Table 'zlm.test_ddl' doesn't exist' on query. Default database: 'zlm'. Query: 'REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT 'zlm', 'test_ddl', '', NULL, NULL, NULL, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(CRC32(CONCAT_WS('#', `id`, convert(`name` using utf8mb4), `tel`, CONCAT(ISNULL(`tel`)))) AS UNSIGNED)), , )), ) AS crc FROM `zlm`.`test_ddl` /*checksum table*/'
                ...
                Last_SQL_Errno:
                Last_SQL_Error: Error 'Table 'zlm.test_ddl' doesn't exist' on query. Default database: 'zlm'. Query: 'REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT 'zlm', 'test_ddl', '', NULL, NULL, NULL, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(CRC32(CONCAT_WS('#', `id`, convert(`name` using utf8mb4), `tel`, CONCAT(ISNULL(`tel`)))) AS UNSIGNED)), , )), ) AS crc FROM `zlm`.`test_ddl` /*checksum table*/'

3.Use parameter "--no-check-slave-tables" when execute pt-table-checksum.

 [root@zlm2 :: ~]
 #pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --no-check-slave-tables -h192.168.1. -P3306 -urepl --ask-pass
 Enter MySQL password:
 Checking if all tables can be checksummed ...
 Starting checksum ...
             TS ERRORS  DIFFS     ROWS  DIFF_ROWS  CHUNKS SKIPPED    TIME TABLE
 -22T04::                                               0.017 mysql.columns_priv
 -22T04::                                               0.015 mysql.db
 -22T04::                                               0.013 mysql.engine_cost
 -22T04::                                               0.011 mysql.event
 -22T04::                                               0.010 mysql.func
 -22T04::                                              0.011 mysql.help_category
 -22T04::                                             0.015 mysql.help_keyword
 -22T04::                                            0.012 mysql.help_relation
 -22T04::                                             0.015 mysql.help_topic
 -22T04::                                               0.013 mysql.ndb_binlog_index
 -22T04::                                               0.012 mysql.plugin
 -22T04::                                              0.012 mysql.proc
 -22T04::                                               0.012 mysql.procs_priv
 -22T04::                                               0.015 mysql.proxies_priv
 -22T04::                                               0.013 mysql.server_cost
 -22T04::                                               0.014 mysql.servers
 -22T04::                                               0.013 mysql.tables_priv
 -22T04::                                               0.012 mysql.time_zone
 -22T04::                                               0.013 mysql.time_zone_leap_second
 -22T04::                                               0.013 mysql.time_zone_name
 -22T04::                                               0.011 mysql.time_zone_transition
 -22T04::                                               0.012 mysql.time_zone_transition_type
 -22T04::                                               0.012 mysql.user
 -22T04::                                               0.011 sys.sys_config
 -22T04:: Error checksumming table zlm.test_ddl: Error getting row count estimate of table zlm.test_ddl on replica zlm3: DBD::mysql::db selectrow_hashref failed: Table 'zlm.test_ddl' doesn't exist [for Statement "EXPLAIN SELECT * FROM `zlm`.`test_ddl` WHERE 1=1"] at /usr/bin/pt-table-checksum line 6823.

 -22T04::                                               0.001 zlm.test_ddl
 -22T04::                                               0.013 zlm.test_ddl_no_pk
 -22T04::                                               0.011 zlm.test_innodb
 -22T04::                                               0.014 zlm.test_myisam

 ###If slave has already down,then it will wait until you handle the issue and restart slave.###
 [root@zlm2 :: ~]
 #pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --no-check-slave-tables -h192.168.1. -P3306 -urepl --ask-pass
 Enter MySQL password:
 Checking if all tables can be checksummed ...
 Starting checksum ...
 Replica zlm3 is stopped.  Waiting.
 Replica zlm3 is stopped.  Waiting.
 Replica zlm3 is stopped.  Waiting.

Summary

• pt-table-checksum is a light tool with less influence of performance.
  
• pt-table-checksum use CRC arlgorism instead of MD5 and SHA1 to reduce consumption of CPU.
  
• Parameter "--no-check-binlog-format" is necessary while replications are using "row" binlog format.
• You can implement pt-table-checksum tool on either master or slave even other third server which is not belongs to replicaitons at all.