Want to archive tables? Use Percona Toolkit’s pt-archiver--转载
原文地址:https://www.percona.com/blog/2013/08/12/want-to-archive-tables-use-pt-archiver/
Percona Toolkit’s pt-archiver is one of the best utilities to archive the records from large tables to another tables or files. One interesting thing is that pt-archiver is a read-write tool. It deletes data from the source by default, so after archiving you don’t need to delete it separately.
As it is done by default, you should take care before actually running it on then production server. You can test your archiving jobs with the — dry-run OR you can use the –no-delete option if you’re not sure about. The purpose of this script is mainly to archive old data from the table without impacting OLTP queries and insert the data into another table on the same/different server OR into a file in a format which is suitable for LOAD DATA INFILE.
How does pt-archiver select records to archive?
Pt-archiver uses the index to select records from the table. The index is used to optimize repeated accesses to the table. Pt-archiver remembers the last row it retrieves from each SELECT statement, and uses it to construct a WHERE clause. It does this using the columns in the specified index that should allow MySQL to start the next SELECT where the last one ended – rather than potentially scanning from the beginning of the table with each successive SELECT.
If you want to run pt-archiver with a specific index you can use the “-i” option in –source DSN options. The “-i” option tells pt-archiver which index it should scan to archive. This appears in a FORCE INDEX orUSE INDEX hint in the SELECT statements that are used to fetch rows to archive. If you don’t specify anything, pt-archiver will auto-discover a good index, preferring a PRIMARY KEY if one exists. Most of the time, without “-i” option, pt-archiver works well.
How to run pt-archiver?
For archive records into normal file, you can run something like
Shell
1
|
pt-archiver --source h=localhost,D=nil,t=test --file '/home/nilnandan/%Y-%m-%d-tabname' --where "name='nil'" --limit-1000
|
From archive records from one table to another table on same server or different, you can run something like
Shell
1
|
pt-archiver --source h=localhost,D=nil,t=test --dest h=fedora.vm --where "name='nil'" --limit-1000
|
Please check this before you use default file option (-F) in –source https://www.percona.com/doc/percona-toolkit/2.1/pt-archiver.html#cmdoption-pt-archiver–dest
Archiving in a replication environment:
In the replication environment it’s really important that the slave should not lag for a long time. So for that, there are two options which we can use while archiving to control the slave lag on slave server.
–check-slave-lag : Pause archiving until the specified DSN’s slave lag is less than –max-lag. In this option, you can give slave details to connect slave lag. (i.e –check-slave-lag h=localhost,S=/tmp/mysql_sandbox29784.sock)
–max-lag : Pause archiving if the slave given by –check-slave-lag lags.
This options causes pt-archiver to look at the slave every time when it’s about to fetch another row. If the slave’s lag is greater than the option’s value, or if the slave isn’t running (so its lag is NULL), pt-archiver sleeps for –check-interval seconds and then looks at the lag again. It repeats until the slave is caught up, then proceeds to fetch and archive the row.
Some useful options for pt-archiver:
–for-update/-share-lock : Adds the FOR UPDATE/LOCK IN SHARE MODE modifier to SELECT statements.
–no-delete : Do not delete archived rows.
–plugin : Perl module name to use as a generic plugin.
–progress : Print progress information every X rows.
–statistics : Collect and print timing statistics.
–where : WHERE clause to limit which rows to archive (required).
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
nilnandan@nil:~$ pt-archiver --source h=localhost,D=nil,t=test,S=/tmp/mysql_sandbox29783.sock --file '/home/nilnandan/%Y-%m-%d-tabname' --where "name='nilnandan'" --limit=50000 --progress=50000 --txn-size=50000 --statistics --bulk-delete --max-lag=1 --check-interval=15 --check-slave-lag h=localhost,S=/tmp/mysql_sandbox29784.sock
TIME ELAPSED COUNT
2013-08-08T10:08:39 0 0
2013-08-08T10:09:25 46 50000
2013-08-08T10:10:32 113 100000
2013-08-08T10:11:41 182 148576
Started at 2013-08-08T10:08:39, ended at 2013-08-08T10:11:59
Source: D=nil,S=/tmp/mysql_sandbox29783.sock,h=localhost,t=test
SELECT 148576
INSERT 0
DELETE 148576
Action Count Time Pct
print_file 148576 18.2674 9.12
bulk_deleting 3 8.9535 4.47
select 4 2.9204 1.46
commit 3 0.0005 0.00
other 0 170.0719 84.95
nilnandan@nil:~$
|
Percona Toolkit’s pt-archiver works with Percona XtraDB Cluster (PXC) 5.5.28-23.7 and newer, but there are three limitations you should consider before archiving on a cluster. You can get more informationhere.
pt-archiver is extensible via a plugin mechanism. You can inject your own code to add advanced archiving logic that could be useful for archiving dependent data, applying complex business rules, or building a data warehouse during the archiving process. Follow this URL for more info on that.
Bugs related to pt-archiver: https://bugs.launchpad.net/percona-toolkit/+bugs?field.tag=pt-archiver
More details about pt-archiver: https://www.percona.com/doc/percona-toolkit/2.2/pt-archiver.html
Want to archive tables? Use Percona Toolkit’s pt-archiver--转载的更多相关文章
- Percona Toolkit mysql辅助利器
1 PT介绍 Percona Toolkit简称pt工具—PT-Tools,是Percona公司开发用于管理MySQL的工具,功能包括检查主从复制的数据一致性.检查重复索引.定位IO占用高的表文件.在 ...
- Percona Toolkit工具使用
Percona Toolkit简称pt工具-PT-Tools,是Percona公司开发用于管理MySQL的工具,功能包括检查主从复制的数据一致性.检查重复索引.定位IO占用高的表文件.在线DDL等 下 ...
- Centos 安装Percona Toolkit工具集
1.下载 下载地址: https://www.percona.com/downloads/percona-toolkit/LATEST/ [root@bogon ~]# wget https:// ...
- RDS for MySQL 如何使用 Percona Toolkit
Percona Toolkit 包含多种用于 MySQL 数据库管理的工具. 下面介绍常用的 pt-online-schema-change 和 pt-archiver 搭配 RDS MySQL ...
- Percona Toolkit工具集介绍
部署mysql工具是一个非常重要的部分,所以工具的可靠性和很好的设计非常重要.percona toolkit是一个有30多个mysql工具的工具箱.兼容mysql,percona server,mar ...
- Percona Toolkit工具连接MySQL 8报错的解决方案
使用Percona Toolkit的工具连接MySQL 8.x数据库时,会遇到类似"failed: Plugin caching_sha2_password could not be loa ...
- Percona Toolkit 使用
安装 percona-toolkit perl Makefile.PL make make test make install 默认安装到 /usr/local/bin 目录下 可能需要 DBI-1. ...
- 使用Percona Toolkit解决Mysql主从不同步问题【备忘】
由于各种原因,mysql主从架构经常会出现数据不一致的情况出现,大致归结为如下几类 1:备库写数据 2:执行non-deterministic query 3:回滚掺杂事务表和非事务表的事务 4:bi ...
- Percona Toolkit 2.2.19 is now available
New Features: 1221372: pt-online-schema-change now aborts with an error if the server is a slave, be ...
随机推荐
- 紫书 习题 10-20 UVa 1648 (推公式)
设一次上去a层,一次下去b层,有x次上去,有(n-x)次下去 则ax - (n-x)b >= 1 x >= (nb+1) / (a+b) 如果可以整除, x = (nb+1) / (a+b ...
- 【Uva 10618】Tango Tango Insurrection
[Link]: [Description] 玩跳舞机. 有一定的约束. 归纳起来就是以下三点 1.两只脚不能同时踩一个位置 2.如果左脚踩在了右键上,那么下一次移动的一定要是左脚 3.如果右脚踩在了左 ...
- WPF 布局控件 之 DockPanel
DockPanel为容器控件 主要了解其Dock属性和LastChildFill属性的使用 一.LastChildFill="True" 时 代码: <DockPanel L ...
- 53.遇到SyntaxError: Unexpected token
转自:https://segmentfault.com/q/1010000002649920/a-1020000002655984 不知道你自己现在已经发现问题没, Unexpected token ...
- pip的认识
一.pip与python的关系:pip并不是一种语言,而是一个Python包管理工具,主要是用于安装 PyPI 上的软件包.安装好pip之后,使用pip install 命令即可方便的安装python ...
- OpenCV —— ROI
通过 cvResetImageRoI 函数释放ROI是非常重要的,否则其他操作将默认在ROI区域中进行 通过巧妙的使用widthStep,可以达到同ROI一样的效果 —— 如果想设置和保持一副图像的多 ...
- dig---域名查询
dig命令是常用的域名查询工具,可以用来测试域名系统工作是否正常. QUESTION SECTION 这部分是提问,显示你要查询的域名 ANSWER SECTION 即答案,显示查询到的域名对应的IP ...
- echo---打印变量或输出字符串
cho命令用于在shell中打印shell变量的值,或者直接输出指定的字符串.linux的echo命令,在shell编程中极为常用, 在终端下打印变量value的时候也是常常用到的,因此有必要了解下e ...
- golang-小试牛刀
首先说下,我为什么选择go吧.之前一直做的都是.net平台下的开发,常用的服务端语言就是C#.刚接触C#的时候,就喜欢上了这种高级语言,它优雅.易上手.开发周期短,很多高级特性以及自带的托管内存管理G ...
- 漫话Unity(二)
三.Unity编辑器介绍 Unity是一个商业级的3d游戏引擎.一个引擎的专业程度事实上并非体如今它多么牛b 的次世代效果,说实话那些效果即便你会用也不敢用.由于没有哪个手机是次世代的. 游戏引擎的专 ...