Percona-Tookit工具包之pt-table-checksum
Preface
The master-slave replication is commonly used in our product evironment.On account of network lag or replicaton mode,slaves probablly be delayed with master.How to solve the consistency of tables between master and slaves?pt-table-checksum is a suitable tool which you can use efficiently.I'm gonna use the tool to do some tests today.
Introduce
pt-table-checksum is a memeber of Percona-Toolkit,it's used to check consistency of tables in online replication environment by execute querries on master(put results into a replica called chckesum table).It will indicate whether there're some inconsistent tables bewteen master and slaves by output on screen with parameter "--print".
Procedure
1.Comman parameter introduce.
Connection relevent:
--ask-pass -- Ask user to input a password when executing.
-h hostname/ip
-u username
-p password(specify the password in command line)
-P port
-S socket
-D database
-t tables Important parameter:
--create-replicate-table -- Create database and table(checksum table) mentioned in "--replicate".
--no-check-binlog-format -- Don't check binlog format on all server.
--recursion-method -- Specify the prefferd mode to find slaves if you've got multiple slaves.
--replicate -- Specify the table(default is "percona.checksums") into which the results will be write.
--replicate-check-only -- Check consistency on replica without executing checksum queries on master.Furthermore,it only checks riplica for differences found by previous checksuming. Output relevent:
--explain -- Show without really execute checksum querries.
--progress -- Print progress report(default seconds).
--quite -- Print only important informations on screen(will disable "--progress"). Safety relevent:
--no-check-slave-tables -- Only if you're confirmed that all tables on slave is the same with master's.Then you can set it to avoid breaking of replication when executing pt-table-checksum.
--check-replication-filters -- Don't checksum if any replication filters are set on target replica.
--chunk-size-limit -- Limit the chunk size to avoid performance issues.
2.Examples.
Master:
(root@localhost mysql3306.sock)[zlm]::>show tables;
+----------------+
| Tables_in_zlm |
+----------------+
| test_ddl |
| test_ddl_no_pk |
| test_innodb |
| test_myisam |
+----------------+
rows in set (0.00 sec) Slave:
(root@localhost mysql3306.sock)[zlm]::>show tables;
+----------------+
| Tables_in_zlm |
+----------------+
| t1 |
| t2 |
| t3 |
| test_ddl |
| test_ddl_no_pk |
| test_innodb |
| test_myisam |
+----------------+
rows in set (0.00 sec) ###My pt-table-checksum is installed on master server.Execute the command below on master.### [root@zlm2 :: ~]
#pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --check-slave-tables -hzlm2 -P3306 -uroot --ask-pass
Enter MySQL password:
-21T10:: DBI connect(';host=zlm2;port=3306;mysql_read_default_group=client','root',...) failed: Access denied for user 'root'@'localhost' (using password: YES) at /usr/bin/pt-table-checksum line . [root@zlm2 :: ~]
#pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --check-slave-tables -hlocalhost -P3306 -uroot --ask-pass
Enter MySQL password:
Checking if all tables can be checksummed ...
Starting checksum ...
Cannot connect to P=,h=zlm3,p=...,u=root -- It's due to the "root" user cannot login with TCP/IP mode.
Diffs cannot be detected because no slaves were found. Please read the --recursion-method documentation for information.
*******************************************************************
Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
possibly with SSL_ca_file|SSL_ca_path for verification.
If you really don't want to verify the certificate and keep the
connection open to Man-In-The-Middle attacks please set
SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
at /usr/bin/pt-table-checksum line .
*******************************************************************
Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
possibly with SSL_ca_file|SSL_ca_path for verification.
If you really don't want to verify the certificate and keep the
connection open to Man-In-The-Middle attacks please set
SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
at /usr/bin/pt-table-checksum line . # A software update is available:
TS ERRORS DIFFS ROWS DIFF_ROWS CHUNKS SKIPPED TIME TABLE
-21T10:: 0.006 mysql.columns_priv
-21T10:: 0.012 mysql.db
-21T10:: 0.010 mysql.engine_cost
-21T10:: 0.008 mysql.event
-21T10:: 0.009 mysql.func
-21T10:: 0.008 mysql.help_category
-21T10:: 0.010 mysql.help_keyword
-21T10:: 0.011 mysql.help_relation
-21T10:: 0.032 mysql.help_topic
-21T10:: 0.008 mysql.ndb_binlog_index
-21T10:: 0.011 mysql.plugin
-21T10:: 0.012 mysql.proc
-21T10:: 0.009 mysql.procs_priv
-21T10:: 0.010 mysql.proxies_priv
-21T10:: 0.009 mysql.server_cost
-21T10:: 0.010 mysql.servers
-21T10:: 0.010 mysql.tables_priv
-21T10:: 0.009 mysql.time_zone
-21T10:: 0.009 mysql.time_zone_leap_second
-21T10:: 0.009 mysql.time_zone_name
-21T10:: 0.009 mysql.time_zone_transition
-21T10:: 0.008 mysql.time_zone_transition_type
-21T10:: 0.007 mysql.user
-21T10:: 0.011 sys.sys_config
-21T10:: 0.008 zlm.test_ddl
-21T10:: 0.008 zlm.test_ddl_no_pk
-21T10:: 0.009 zlm.test_innodb
-21T10:: 0.009 zlm.test_myisam ###See the detail of checksum table.###
(root@localhost mysql3306.sock)[zlm]::>show tables;
+----------------+
| Tables_in_zlm |
+----------------+
| checksums | -- the table checksums was created.
| test_ddl |
| test_ddl_no_pk |
| test_innodb |
| test_myisam |
+----------------+
rows in set (0.00 sec) (root@localhost mysql3306.sock)[zlm]::>show create table checksums\G
*************************** . row ***************************
Table: checksums
Create Table: CREATE TABLE `checksums` (
`db` char() NOT NULL,
`tbl` char() NOT NULL,
`chunk` int() NOT NULL,
`chunk_time` float DEFAULT NULL,
`chunk_index` varchar() DEFAULT NULL,
`lower_boundary` text,
`upper_boundary` text,
`this_crc` char() NOT NULL,
`this_cnt` int() NOT NULL,
`master_crc` char() DEFAULT NULL,
`master_cnt` int() DEFAULT NULL,
`ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`db`,`tbl`,`chunk`),
KEY `ts_db_tbl` (`ts`,`db`,`tbl`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
row in set (0.00 sec) ###Check slave hosts information.###
(root@localhost mysql3306.sock)[zlm]::>show slave hosts;
+-----------+------+------+-----------+--------------------------------------+
| Server_id | Host | Port | Master_id | Slave_UUID |
+-----------+------+------+-----------+--------------------------------------+
| | | | | 5c77c31b-4add-11e8-81e2-080027de0e0e |
+-----------+------+------+-----------+--------------------------------------+
row in set (0.00 sec) ###Change another user 'repl'@'192.168.1.%' and grant all privileges to it.###
(root@localhost mysql3306.sock)[zlm]::>grant all privileges on *.* to 'repl'@'192.168.1.%';
Query OK, rows affected (0.00 sec) (root@localhost mysql3306.sock)[zlm]::>show grants for 'repl'@'192.168.1.%';
+-----------------------------------------------------+
| Grants for repl@192.168..% |
+-----------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'repl'@'192.168.1.%' |
+-----------------------------------------------------+
row in set (0.00 sec) ###Execute pt-table-checksum again.###
[root@zlm2 :: ~]
#pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --check-slave-tables -h192.168.1. -P3306 -urepl --ask-pass
Enter MySQL password:
Checking if all tables can be checksummed ...
Starting checksum ...
*******************************************************************
Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
possibly with SSL_ca_file|SSL_ca_path for verification.
If you really don't want to verify the certificate and keep the
connection open to Man-In-The-Middle attacks please set
SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
at /usr/bin/pt-table-checksum line .
*******************************************************************
Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
possibly with SSL_ca_file|SSL_ca_path for verification.
If you really don't want to verify the certificate and keep the
connection open to Man-In-The-Middle attacks please set
SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
at /usr/bin/pt-table-checksum line . # A software update is available:
TS ERRORS DIFFS ROWS DIFF_ROWS CHUNKS SKIPPED TIME TABLE
-21T11:: 0.018 mysql.columns_priv
-21T11:: 0.020 mysql.db
-21T11:: 0.016 mysql.engine_cost
-21T11:: 0.017 mysql.event
-21T11:: 0.014 mysql.func
-21T11:: 0.018 mysql.help_category
-21T11:: 0.016 mysql.help_keyword
-21T11:: 0.015 mysql.help_relation
-21T11:: 0.019 mysql.help_topic
-21T11:: 0.013 mysql.ndb_binlog_index
-21T11:: 0.012 mysql.plugin
-21T11:: 0.015 mysql.proc
-21T11:: 0.012 mysql.procs_priv
-21T11:: 0.016 mysql.proxies_priv
-21T11:: 0.015 mysql.server_cost
-21T11:: 0.015 mysql.servers
-21T11:: 0.014 mysql.tables_priv
-21T11:: 0.013 mysql.time_zone
-21T11:: 0.013 mysql.time_zone_leap_second
-21T11:: 0.015 mysql.time_zone_name
-21T11:: 0.015 mysql.time_zone_transition
-21T11:: 0.012 mysql.time_zone_transition_type
-21T11:: 0.013 mysql.user
-21T11:: 0.015 sys.sys_config
-21T11:: 0.014 zlm.test_ddl
-21T11:: 0.015 zlm.test_ddl_no_pk
-21T11:: 0.021 zlm.test_innodb
-21T11:: 0.016 zlm.test_myisam ###Check whether there're records in "checksums" table or not on slave.###
(root@localhost mysql3306.sock)[zlm]::>select * from checksums where this_cnt<>master_cnt;
+-------+------+-------+------------+-------------+----------------+----------------+----------+----------+------------+------------+---------------------+
| db | tbl | chunk | chunk_time | chunk_index | lower_boundary | upper_boundary | this_crc | this_cnt | master_crc | master_cnt | ts |
+-------+------+-------+------------+-------------+----------------+----------------+----------+----------+------------+------------+---------------------+
| mysql | proc | | 0.001277 | NULL | NULL | NULL | 9e5a007c | | 4e0f05d9 | | -- :: |
| mysql | user | | 0.000667 | NULL | NULL | NULL | 7de55b47 | | 587dfc7 | | -- :: |
+-------+------+-------+------------+-------------+----------------+----------------+----------+----------+------------+------------+---------------------+
rows in set (0.00 sec)
Supplement
1.Drop a table "testl_ddl" on slave when master-slave replication is normally running.
(root@localhost mysql3306.sock)[zlm]::>show tables;
+----------------+
| Tables_in_zlm |
+----------------+
| checksums |
| t1 |
| t2 |
| t3 |
| test_ddl |
| test_ddl_no_pk |
| test_innodb |
| test_myisam |
+----------------+
rows in set (0.00 sec) (root@localhost mysql3306.sock)[zlm]::>drop table test_ddl;
Query OK, rows affected (0.00 sec)
2.Execute pt-table-checksum again.
[root@zlm2 :: ~]
#pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --check-slave-tables -h192.168.1. -P3306 -urepl --ask-pass
Enter MySQL password:
Checking if all tables can be checksummed ...
Starting checksum ...
TS ERRORS DIFFS ROWS DIFF_ROWS CHUNKS SKIPPED TIME TABLE
-22T04:: 0.012 mysql.columns_priv
-22T04:: 0.015 mysql.db
-22T04:: 0.013 mysql.engine_cost
-22T04:: 0.016 mysql.event
-22T04:: 0.014 mysql.func
-22T04:: 0.011 mysql.help_category
-22T04:: 0.015 mysql.help_keyword
-22T04:: 0.017 mysql.help_relation
-22T04:: 0.018 mysql.help_topic
-22T04:: 0.014 mysql.ndb_binlog_index
-22T04:: 0.013 mysql.plugin
-22T04:: 0.013 mysql.proc
-22T04:: 0.013 mysql.procs_priv
-22T04:: 0.014 mysql.proxies_priv
-22T04:: 0.012 mysql.server_cost
-22T04:: 0.012 mysql.servers
-22T04:: 0.014 mysql.tables_priv
-22T04:: 0.014 mysql.time_zone
-22T04:: 0.013 mysql.time_zone_leap_second
-22T04:: 0.012 mysql.time_zone_name
-22T04:: 0.014 mysql.time_zone_transition
-22T04:: 0.014 mysql.time_zone_transition_type
-22T04:: 0.015 mysql.user
-22T04:: 0.012 sys.sys_config
-22T04:: Skipping table zlm.test_ddl because it has problems on these replicas:
Table zlm.test_ddl does not exist on replica zlm3
This can break replication. If you understand the risks, specify --no-check-slave-tables to disable this check.
-22T04:: Error checksumming table zlm.test_ddl: Error getting row count estimate of table zlm.test_ddl on replica zlm3: DBD::mysql::db selectrow_hashref failed: Table 'zlm.test_ddl' doesn't exist [for Statement "EXPLAIN SELECT * FROM `zlm`.`test_ddl` WHERE 1=1"] at /usr/bin/pt-table-checksum line 6823. -22T04:: 0.002 zlm.test_ddl
-22T04:: 0.014 zlm.test_ddl_no_pk
-22T04:: 0.014 zlm.test_innodb
-22T04:: 0.014 zlm.test_myisam ###This will always lead to "ERROR 1146" what is shown below.###
Last_Errno:
Last_Error: Error 'Table 'zlm.test_ddl' doesn't exist' on query. Default database: 'zlm'. Query: 'REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT 'zlm', 'test_ddl', '', NULL, NULL, NULL, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(CRC32(CONCAT_WS('#', `id`, convert(`name` using utf8mb4), `tel`, CONCAT(ISNULL(`tel`)))) AS UNSIGNED)), , )), ) AS crc FROM `zlm`.`test_ddl` /*checksum table*/'
...
Last_SQL_Errno:
Last_SQL_Error: Error 'Table 'zlm.test_ddl' doesn't exist' on query. Default database: 'zlm'. Query: 'REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT 'zlm', 'test_ddl', '', NULL, NULL, NULL, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(CRC32(CONCAT_WS('#', `id`, convert(`name` using utf8mb4), `tel`, CONCAT(ISNULL(`tel`)))) AS UNSIGNED)), , )), ) AS crc FROM `zlm`.`test_ddl` /*checksum table*/'
3.Use parameter "--no-check-slave-tables" when execute pt-table-checksum.
[root@zlm2 :: ~]
#pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --no-check-slave-tables -h192.168.1. -P3306 -urepl --ask-pass
Enter MySQL password:
Checking if all tables can be checksummed ...
Starting checksum ...
TS ERRORS DIFFS ROWS DIFF_ROWS CHUNKS SKIPPED TIME TABLE
-22T04:: 0.017 mysql.columns_priv
-22T04:: 0.015 mysql.db
-22T04:: 0.013 mysql.engine_cost
-22T04:: 0.011 mysql.event
-22T04:: 0.010 mysql.func
-22T04:: 0.011 mysql.help_category
-22T04:: 0.015 mysql.help_keyword
-22T04:: 0.012 mysql.help_relation
-22T04:: 0.015 mysql.help_topic
-22T04:: 0.013 mysql.ndb_binlog_index
-22T04:: 0.012 mysql.plugin
-22T04:: 0.012 mysql.proc
-22T04:: 0.012 mysql.procs_priv
-22T04:: 0.015 mysql.proxies_priv
-22T04:: 0.013 mysql.server_cost
-22T04:: 0.014 mysql.servers
-22T04:: 0.013 mysql.tables_priv
-22T04:: 0.012 mysql.time_zone
-22T04:: 0.013 mysql.time_zone_leap_second
-22T04:: 0.013 mysql.time_zone_name
-22T04:: 0.011 mysql.time_zone_transition
-22T04:: 0.012 mysql.time_zone_transition_type
-22T04:: 0.012 mysql.user
-22T04:: 0.011 sys.sys_config
-22T04:: Error checksumming table zlm.test_ddl: Error getting row count estimate of table zlm.test_ddl on replica zlm3: DBD::mysql::db selectrow_hashref failed: Table 'zlm.test_ddl' doesn't exist [for Statement "EXPLAIN SELECT * FROM `zlm`.`test_ddl` WHERE 1=1"] at /usr/bin/pt-table-checksum line 6823. -22T04:: 0.001 zlm.test_ddl
-22T04:: 0.013 zlm.test_ddl_no_pk
-22T04:: 0.011 zlm.test_innodb
-22T04:: 0.014 zlm.test_myisam ###If slave has already down,then it will wait until you handle the issue and restart slave.###
[root@zlm2 :: ~]
#pt-table-checksum --no-check-binlog-format --replicate=zlm.checksums --no-check-slave-tables -h192.168.1. -P3306 -urepl --ask-pass
Enter MySQL password:
Checking if all tables can be checksummed ...
Starting checksum ...
Replica zlm3 is stopped. Waiting.
Replica zlm3 is stopped. Waiting.
Replica zlm3 is stopped. Waiting.
Summary
- pt-table-checksum is a light tool with less influence of performance.
- pt-table-checksum use CRC arlgorism instead of MD5 and SHA1 to reduce consumption of CPU.
- Parameter "--no-check-binlog-format" is necessary while replications are using "row" binlog format.
- You can implement pt-table-checksum tool on either master or slave even other third server which is not belongs to replicaitons at all.
Percona-Tookit工具包之pt-table-checksum的更多相关文章
- NXP ARM Vector Table CheckSum
Signature Creator for NXP Cortex-M Devices Algorithm for creating the checksum The reserved Cortex-M ...
- DROP TABLE 恢复【一】
当DROP TABLE指令敲下的时候,你很爽,你有考虑过后果么?如果该表真的没用,你DROP到无所谓,如果还有用的,这时你肯定吓惨了吧,如果你有备份,那么恭喜你,逃过一劫,如果没有备份呢?这时就该绝望 ...
- IAR EWARM Checksum Technical Note
IELFTOOL Checksum - Basic actions EW targets: ARM, RH850, RX, SH, STM8 EW component: General issues ...
- Mysql: pt-table-checksum 和 pt-table-sync 检查主从一致性，实验过程
一.安装 percona 包 1.安装仓库的包 https://www.percona.com/doc/percona-repo-config/yum-repo.html sudo yum insta ...
- pt-table-checksum
pt-table-checksum是percona公司提供的一个用于在线比对主从数据一致性的工具. 实现原理 将一张大表分成多个chunk,每次针对一个chunk进行校验,同时将校验的结果通过REPL ...
- [知识库分享系列] 二、.NET（ASP.NET）
最近时间又有了新的想法,当我用新的眼光在整理一些很老的知识库时,发现很多东西都已经过时,或者是很基础很零碎的知识点.如果分享出去大家不看倒好,更担心的是会误人子弟,但为了保证此系列的完整,还是选择分享 ...
- pt-table-checksum解读
pt-table-checksum是目前可以说是最好的查看主从一致性的工具 先来个使用例子,有助快速上手使用 在主库执行: mysql>GRANT SELECT, PROCESS, SUPER, ...
- mysql数据一致性检查及修复
percona-toolkit-2.2.20-1.noarchmysql 5.6.29-logmaster:192.168.166.129slave:192.168.166.131 一.创建数据库校验 ...
- MySQL 错误集-汇总
Q&A: MySQl报错之@@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_MODE = ON 导入的时候加入-f参数即可 原因分 ...
随机推荐
- Vue element-ui：滚动条 分页 禁用选项
1.滚动条设置: <el-scrollbar style="height:100%;">.......</el-scrollbar> 默认会同时出现水平和垂 ...
- SharePoint 2013 - Workflow Manager
1. Workflow Manager可以与SharePoint 安装在同一台机器上,只是不建议这么做:由于Workflow Manager 需要使用数据库,我个人将其安装在 SQL Server机器 ...
- 任务五：零基础HTML及CSS编码练习加强版
任务目的 针对设计稿样式进行合理的HTML架构,包括以下但不限于: 掌握常用HTML标签的含义.用法 能够基于设计稿来合理规划HTML文档结构 理解语义化,合理地使用HTML标签来构建页面 掌握基本的 ...
- JMeter 配置元件之-HTTP Cookie管理器-实现 Cookie 登录
下面以购物App常用的收藏功能为例,讲述如何在 JMeter 中使用 cookie 进行登录态操作.具体的操作步骤如下所示: 第一步.录制查看收藏列表的 HTTP请求: 录制查看收藏列表的HTTP请求 ...
- Azure 9 月新发布
亲爱的小伙伴们, 我们很高兴向您宣布以下新功能与相关调整,欢迎关注与使用. 1. SQL 数据库弹性池 2. 存储指标更新 3. SQL 数据库 P15 4. Azure 高级存储 5. Wosig ...
- 大数据的正确用法你get到了吗？
Azure 镜像市场已于2016年9月21日正式上线,在这个统一的集成平台中,客户可以轻松地浏览.搜索和选择一系列来自第三方的应用和解决方案,并可以将其快速一键部署到 Azure 实例当中. 在移动为 ...
- Struts2学习-拦截器2续
定义拦截器有2种办法:1.实现Interceptor接口2.集成AbstractInterceptor抽象类 一.方法1 ..... <struts> <package name=& ...
- Linux下实现普通用户免密登录并执行root权限
需求:服务器的root和密码登录都禁用,只开放普通用户登录,这时需要给普通用户配置秘钥文件,实现无密码登录 需要注意的是使用什么用户,就把秘钥文件拷入到该用户的家目录下,如果是root用户,就直接拷贝 ...
- Wi-Fi
AP就是一个无线的交换机,提供无线信号发射接收的功能 Wi-Fi是一种可以将个人电脑.手持设备(如PDA.手机)等终端以无线方式互相连接的技术 两个不一样的东西,无法比较的 你说的应该是无线路由器和无 ...
- webpack前言：前端模块系统的演进
前端开发和其他开发工作的主要区别,首先是前端是基于多语言.多层次的编码和组织工作,其次前端产品的交付是基于浏览器,这些资源是通过增量加载的方式运行到浏览器端,如何在开发环境组织好这些碎片化的代码和资源 ...