pt-online-schema-change的实现原理

pt-online-schema-change用于MySQL的在线DDL。

下面结合官方文档和general log来分析其实现原理。

测试表

mysql> show create table t2\G

*************************** 1. row ***************************

       Table: t2

Create Table: CREATE TABLE `t2` (

  `id` int(11) NOT NULL AUTO_INCREMENT,

  PRIMARY KEY (`id`)

) ENGINE=InnoDB AUTO_INCREMENT=1005764 DEFAULT CHARSET=utf8

1 row in set (0.19 sec)

该表中只有1列，id，自增主键。

其中，表中已经存在一部分数据

mysql> select count(*) from t2;

+----------+

| count(*) |

+----------+

|  1005763 |

+----------+

1 row in set (0.31 sec)

利用pt-online-schema-change对该表新增一列

# pt-online-schema-change --execute --alter "ADD COLUMN c1 DATETIME" D=test,t=t2

Found 2 slaves:

  test

  hbase

Will check slave lag on:

  test

  hbase

Operation, tries, wait:

  analyze_table, 10, 1

  copy_rows, 10, 0.25

  create_triggers, 10, 1

  drop_triggers, 10, 1

  swap_tables, 10, 1

  update_foreign_keys, 10, 1

Altering `test`.`t2`...

Creating new table...

CREATE TABLE `test`.`_t2_new` (

  `id` int(11) NOT NULL AUTO_INCREMENT,

  PRIMARY KEY (`id`)

) ENGINE=InnoDB AUTO_INCREMENT=1005764 DEFAULT CHARSET=utf8

Created new table test._t2_new OK.

Waiting forever for new table `test`.`_t2_new` to replicate to test...

Altering new table...

ALTER TABLE `test`.`_t2_new` ADD COLUMN c1 DATETIME

Altered `test`.`_t2_new` OK.

2016-11-21T12:49:18 Creating triggers...

CREATE TRIGGER `pt_osc_test_t2_del` AFTER DELETE ON `test`.`t2` FOR EACH ROW DELETE IGNORE FROM `test`.`_t2_new` WHERE `test`.`_t2_ne

w`.`id` <=> OLD.`id`CREATE TRIGGER `pt_osc_test_t2_upd` AFTER UPDATE ON `test`.`t2` FOR EACH ROW REPLACE INTO `test`.`_t2_new` (`id`) VALUES (NEW.`id`)

CREATE TRIGGER `pt_osc_test_t2_ins` AFTER INSERT ON `test`.`t2` FOR EACH ROW REPLACE INTO `test`.`_t2_new` (`id`) VALUES (NEW.`id`)

2016-11-21T12:49:18 Created triggers OK.

2016-11-21T12:49:18 Copying approximately 1005075 rows...

INSERT LOW_PRIORITY IGNORE INTO `test`.`_t2_new` (`id`) SELECT `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= ?)) AND (

(`id` <= ?)) LOCK IN SHARE MODE /*pt-online-schema-change 2352 copy nibble*/SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= ?)) ORDER BY `id` LIMIT ?, 2 /*next chun

k boundary*/Copying `test`.`t2`:  40% 00:44 remain

Copying `test`.`t2`:  82% 00:12 remain

2016-11-21T12:50:31 Copied rows OK.

2016-11-21T12:50:31 Analyzing new table...

2016-11-21T12:50:32 Swapping tables...

RENAME TABLE `test`.`t2` TO `test`.`_t2_old`, `test`.`_t2_new` TO `test`.`t2`

2016-11-21T12:50:35 Swapped original and new tables OK.

2016-11-21T12:50:35 Dropping old table...

DROP TABLE IF EXISTS `test`.`_t2_old`

2016-11-21T12:50:36 Dropped old table `test`.`_t2_old` OK.

2016-11-21T12:50:36 Dropping triggers...

DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_del`;

DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_upd`;

DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_ins`;

2016-11-21T12:50:36 Dropped triggers OK.

Successfully altered `test`.`t2`.

查看general log中的输出

161017 11:22:56     1052 Connect    root@localhost on test

         1052 Query    set autocommit=1

         1052 Query    SHOW VARIABLES LIKE 'innodb\_lock_wait_timeout'

         1052 Query    SET SESSION innodb_lock_wait_timeout=1

         1052 Query    SHOW VARIABLES LIKE 'lock\_wait_timeout'

         1052 Query    SET SESSION lock_wait_timeout=60

         1052 Query    SHOW VARIABLES LIKE 'wait\_timeout'

         1052 Query    SET SESSION wait_timeout=10000

         1052 Query    SELECT @@SQL_MODE

         1052 Query    SET @@SQL_QUOTE_SHOW_CREATE = 1/*!40101, @@SQL_MODE='NO_AUTO_VALUE_ON_ZERO,STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION'*/

         1052 Query    SELECT @@server_id /*!50038 , @@hostname*/

         1053 Connect    root@localhost on test

         1053 Query    set autocommit=1

         1053 Query    SHOW VARIABLES LIKE 'innodb\_lock_wait_timeout'

         1053 Query    SET SESSION innodb_lock_wait_timeout=1

         1053 Query    SHOW VARIABLES LIKE 'lock\_wait_timeout'

         1053 Query    SET SESSION lock_wait_timeout=60

         1053 Query    SHOW VARIABLES LIKE 'wait\_timeout'

         1053 Query    SET SESSION wait_timeout=10000

         1053 Query    SELECT @@SQL_MODE

         1053 Query    SET @@SQL_QUOTE_SHOW_CREATE = 1/*!40101, @@SQL_MODE='NO_AUTO_VALUE_ON_ZERO,STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION'*/

         1053 Query    SELECT @@server_id /*!50038 , @@hostname*/

上述主要是设置会话的变量信息，包括innodb_lock_wait_timeout，wait_timeout和SQL_QUOTE_SHOW_CREATE。

         1052 Query    SHOW VARIABLES LIKE 'wsrep_on'

         1052 Query    SHOW VARIABLES LIKE 'version%'

         1052 Query    SHOW ENGINES

         1052 Query    SHOW VARIABLES LIKE 'innodb_version'

         1052 Query    SHOW VARIABLES LIKE 'innodb_stats_persistent'

         1052 Query    SELECT @@SERVER_ID

         1052 Query    SHOW GRANTS FOR CURRENT_USER()

         1052 Query    SHOW FULL PROCESSLIST

         1052 Query    SHOW SLAVE HOSTS

         1052 Query    SHOW GLOBAL STATUS LIKE 'Threads_running'

         1052 Query    SHOW GLOBAL STATUS LIKE 'Threads_running'

         1052 Query    SELECT CONCAT(@@hostname, @@port)

         1052 Query    SHOW TABLES FROM `test` LIKE 't2'

         1052 Query    SHOW TRIGGERS FROM `test` LIKE 't2'

         1052 Query    /*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, @@SQL_MODE := '', @OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, @@SQL_QUOTE_SHOW_CREATE := 1 */

         1052 Query    USE `test`

         1052 Query    SHOW CREATE TABLE `test`.`t2`

         1052 Query    /*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */

         1052 Query    EXPLAIN SELECT * FROM `test`.`t2` WHERE 1=1

         1052 Query    SELECT table_schema, table_name FROM information_schema.key_column_usage WHERE referenced_table_schema='test' AND referenced_table_name='t2'

         1052 Query    SHOW VARIABLES LIKE 'wsrep_on'

         1052 Query    /*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, @@SQL_MODE := '', @OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, @@SQL_QUOTE_SHOW_CREATE := 1 */

解释：

1. 查看参数变量，当前用户的权限，slave的信息，会话变量

2. 确认t2是否存在，t2上是否有触发器

3. 查看执行计划

4. 查看是否t2表是否被其它表外键关联。

           39 Query    USE `test`

           39 Query    SHOW CREATE TABLE `test`.`t2`

           39 Query    /*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */

           39 Query    CREATE TABLE `test`.`_t2_new` (

  `id` int(11) NOT NULL AUTO_INCREMENT,

  PRIMARY KEY (`id`)

) ENGINE=InnoDB AUTO_INCREMENT=1005764 DEFAULT CHARSET=utf8

161121 12:49:18       39 Query    ALTER TABLE `test`.`_t2_new` ADD COLUMN c1 DATETIME

           39 Query    /*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, @@SQL_MODE := '', @OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, @@SQL_QUOTE_SHOW_CREATE := 1 */

           39 Query    USE `test`

           39 Query    SHOW CREATE TABLE `test`.`_t2_new`

           39 Query    /*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */

           39 Query    CREATE TRIGGER `pt_osc_test_t2_del` AFTER DELETE ON `test`.`t2` FOR EACH ROW DELETE IGNORE FROM `test`.`_t2_new` WHERE `test`.`_t2_new`.`id` <=> OLD.`id`

           39 Query    CREATE TRIGGER `pt_osc_test_t2_upd` AFTER UPDATE ON `test`.`t2` FOR EACH ROW REPLACE INTO `test`.`_t2_new` (`id`) VALUES (NEW.`id`)

           39 Query    CREATE TRIGGER `pt_osc_test_t2_ins` AFTER INSERT ON `test`.`t2` FOR EACH ROW REPLACE INTO `test`.`_t2_new` (`id`) VALUES (NEW.`id`)

解释：

1. 根据目标表结构创建一张新表。

2. 对新表添加字段，可以看出pt-online-shema-change对表结构进行变更依赖的还是MySQL自身的Online DDL。

3. 针对目标表创建三个触发器，DELETE，UPDATE和INSERT，因为REPLACE操作只有在主键或唯一索引存在的情况下才有意义，这也就解释了为什么目标表上要有主键或唯一索引。

           39 Query    EXPLAIN SELECT * FROM `test`.`t2` WHERE 1=1

           39 Query    SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) ORDER BY `id` LIMIT 1 /*first lower boundary*/

           39 Query    SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX (`PRIMARY`) WHERE `id` IS NOT NULL ORDER BY `id` LIMIT 1 /*key_len*/

           39 Query    EXPLAIN SELECT /*!40001 SQL_NO_CACHE */ * FROM `test`.`t2` FORCE INDEX (`PRIMARY`) WHERE `id` >= '' /*key_len*/

           39 Query    EXPLAIN SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '')) ORDER BY `id` LIMIT 999, 2 /*next chunk boundary*/

           39 Query    SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '')) ORDER BY `id` LIMIT 999, 2 /*next chunk boundary*/

           39 Query    EXPLAIN SELECT `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '')) AND ((`id` <= '')) LOCK IN SHARE MODE /*explain pt-online-schema-change 2352 copy nibble*/

           39 Query    INSERT LOW_PRIORITY IGNORE INTO `test`.`_t2_new` (`id`) SELECT `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '')) AND ((`id` <= '')) LOCK IN SHARE MODE /*pt-online-schema-change 2352 copy nibble*/

           39 Query    SHOW WARNINGS

           39 Query    SELECT @@SERVER_ID

           39 Query    SHOW GRANTS FOR CURRENT_USER()

           39 Query    SHOW FULL PROCESSLIST

           39 Query    SELECT @@SERVER_ID

           39 Query    SHOW GRANTS FOR CURRENT_USER()

           39 Query    SHOW FULL PROCESSLIST

161121 12:49:20       39 Query    SELECT 'pt-online-schema-change keepalive'

161121 12:49:21       39 Query    SELECT @@SERVER_ID

           39 Query    SHOW GRANTS FOR CURRENT_USER()

           39 Query    SHOW FULL PROCESSLIST

           39 Query    SHOW GLOBAL STATUS LIKE 'Threads_running'

           39 Query    EXPLAIN SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '')) ORDER BY `id` LIMIT 28516, 2 /*next chunk boundary*/

           39 Query    SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '')) ORDER BY `id` LIMIT 28516, 2 /*next chunk boundary*/

           39 Query    EXPLAIN SELECT `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '')) AND ((`id` <= '')) LOCK IN SHARE MODE /*explain pt-online-schema-change 2352 copy nibble*/

           39 Query    INSERT LOW_PRIORITY IGNORE INTO `test`.`_t2_new` (`id`) SELECT `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '')) AND ((`id` <= '')) LOCK IN SHARE MODE /*pt-online-schema-change 2352 copy nibble*/

解释：

上述输出只包含两个chunk的选择。其它chunk的选择基本相同。

1. SHOW GLOBAL STATUS LIKE 'Threads_running'用于监控当前的系统负载。

2. 可以看出pt-online-schema-change是以chunk为单位进行目标表数据的拷贝。

3. 在拷贝的过程中，对目标表的相关记录加了共享锁，此时，会堵塞客户端对这些记录的DML操作。

           39 Query    ANALYZE TABLE `test`.`_t2_new` /* pt-online-schema-change */

161121 12:50:32       39 Query    RENAME TABLE `test`.`t2` TO `test`.`_t2_old`, `test`.`_t2_new` TO `test`.`t2`

161121 12:50:35       39 Query    DROP TABLE IF EXISTS `test`.`_t2_old`

161121 12:50:36       39 Query    DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_del`

           39 Query    DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_upd`

           39 Query    DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_ins`

           39 Query    SHOW TABLES FROM `test` LIKE '\_t2\_new'

161121 12:50:37       40 Quit

           39 Quit

解释：

1. 在完成数据的拷贝后，会对新表执行ANALYZE操作，这样，可及时更新新表的统计信息。

官档的解释如下：

This circumvents a potentially serious issue related to InnoDB optimizer statistics. If the table being alerted is

busy and the tool completes quickly, the new table will not have optimizer statistics after being swapped. This

can cause fast, index-using queries to do full table scans until optimizer statistics are updated (usually after 10

seconds). If the table is large and the server very busy, this can cause an outage.

2. 对目标表和新表进行RENAME操作。

3. 删除原来的目标表

4. 删除触发器。

pt-online-schema-change的实现原理的更多相关文章

schema change + ogg 变更手册
Check OGG until no data queuing in replication process:testRO:a)login test5 –l oggmgrb)oggc)#ggsci ...
Online Schema Change for MySQL
It is great to be able to build small utilities on top of an excellent RDBMS. Thank you MySQL. This ...
AppBoxFuture(四). 随需而变-Online Schema Change
需求变更是信息化过程中的家常便饭,而在变更过程中如何尽可能小的影响在线业务是比较头疼的事情.举个车联网监控的例子:原终端设备上传车辆的经纬度数据,新的终端设备支持同时上传速度数据,而旧的车辆状态表 ...
Online, Asynchronous Schema Change in F1
F1: A Distributed SQL Database That Scales http://disksing.com/understanding-f1-schema-change ma ...
MySQL OSC(在线更改表结构)原理
1 OSC介绍在我们的数据库操作中,更改表结构是一个常见的操作,而当我们的表数据量非常大时,我们更改表结构的时间是非常的长,并且在跟改期间,会生成一个互斥锁,阻塞对整个表的所有操作,这样,对于我们 ...
Online Schema Upgrade in MySQL Galera Cluster using TOI Method
http://severalnines.com/blog/online-schema-upgrade-mysql-galera-cluster-using-toi-method As a fo ...
OSC的原理
OSC是Online Schema Change简写,即在线架构改变.其实现步骤: 1. init,即初始化阶段,会对创建的表做一些验证工作,如检查表是否有主键,是否存在触发器或者外键等.2. cre ...
Schema 与数据类型优化
这是<高性能 MySQL(第三版)>第四章<Schema 与数据类型优化>的读书笔记. 1. 选择优化的数据类型数据类型的选择原则: 越小越好:选择满足需求的最小类型.注意, ...
MongoDB 变更流（Change Stream）介绍
1. 什么是Change Stream Change Stream 是MongoDB用于实现变更追踪的解决方案,类似于关系数据库的触发器,但原理不完全相同: | | Change Stream | 触 ...
iDB是如何运转的一
郑昀创建于2015/12/2 最后更新于2015/12/4 关键词:数据库,MySQL,自动化运维,DDL,DML,SQL审核,备份,回滚,Inception,osc 提纲: 普通DBA和文艺DBA ...

随机推荐

如何用SQL脚本在SQL Server Replication中创建合并复制，以及怎么创建分区合并复制
假设我们要创建合并复制的发布端数据库是EFDemo其中有四张表,订阅端数据库是EFDemoSubscription,如下图所示: 首先创建发布端快照代理Sql agent job:"EFDe ...
innodb_locks_unsafe_for_binlog分析
mysql数据库中默认的隔离级别为repeat-read. innodb默认使用了next-gap算法,这种算法结合了index-row锁和gap锁.正因为这样的锁算法,innodb在可重复读这样的默 ...
[Spark Streaming_1] Spark Streaming 概述
0. 说明 Spark Streaming 介绍 && 在 IDEA 中编写 Spark Streaming 程序 1. Spark Streaming 介绍 Spark Stream ...
C++第六次作业
前言拿到作业的时候,整个人都不好了,虽然之前和同学说以后一起写游戏,画界面,然而现在的自己对界面的知识一窍不通,虽然同学分享了一些资料,但是通过这次作业,发现自己火候还是不够-- 问题描述及仓库地址 ...
服务器上u盘装机centos7.2
说明: 截止目前CentOS 7.x最新版本为CentOS 7.2.1511,下面介绍CentOS 7.2.1511的具体安装配置过程服务器相关设置如下: 操作系统:CentOS 7.2.1511 ...
ModuleNotFoundError No module named urllib2
ModuleNotFoundError No module named urllib2?那么在进行编辑的来进行代码上开发,那就会出现的来代码的上错误,也是版本的上差异导致的问题. 工具/原料 ...
Actor模型---SwiftActors
actor是一个无线程区别的内存访问对象:actor背后有线程支持:actor的事件处理依赖与这个线程(队列.池). actor是一种面向对象的线程(池)模型,强调对事件的响应:在iOS中相当于一种通 ...
构造方法、 This关键字、static、封装
1.1 构造方法构造方法是一种特殊的方法,专门用于构造/实例化对象,形式: [修饰符] 类名(){ } 构造方法根据是否有参数分为无参构造和有参构. 1.1.1 无参构造无参构造方法就是构造方法没 ...
android 解决小米手机上选择照片路径为null情况
昨天测试帅哥说他手机选择图库崩溃了,这是一个上传头像的功能,相信很多应用都有这个功能,于是我就把手机拿过来打log看了下返回的路径为null,在网上搜索了下解决方案,现在把解决方案记录下: 这是在o ...
javascript实现拖曳与拖放图片
javascript实现拖曳与拖放图片其实对于drag和drop拖曳与拖放事件IE很早以前就支持这个操作了,我们先来看看HTML5中新增的拖放API. 在HTML5中想要实现拖放操作,至少要做以下操 ...

pt-online-schema-change的实现原理

pt-online-schema-change的实现原理的更多相关文章

随机推荐

热门专题