Archive MySQL Data In Chunks Using Stored Procedure
sqladminon September 26, 2018
In a DBA’s day to day activities, we are doing Archive operation on our transnational database servers to improve your queries and control the Disk space. The archive is a most expensive operation since its involved a huge number of Read and Write will be performed. So its mandatory to run the archive queries in chunks. The archive is depended on business use. Many of us need a copy of the data on an archive database to refer later. To perform the archive we can just simply run the delete query with the limit. But we need to run the query again and again until the matched rows count is 0. We can create a procedure to do this in a while loop. I have created one such procedure to archive many tables.

Why Archive is an expensive operation?
Generally how we are arching the data is delete from table_name where column_name <= some_value; If you are running on a table which needs to be deleted around 15million records, then you need the undo log file to hold all of these records. There will be a heavy IO happening in the Disk. And lt’ll lock the rows and some other locks will be held until the Archive complete. Replication may delay because of this.
When Archive is going to mess up the production?
- Running archive commands on a heavy traffic time.
- Archive without a proper where clause.
- Delete data without limit.
- Performing archive contrition on a not indexed column.
- Continuously run the delete query in chunks on a replication environment. {without sleep(1 or few seconds}.
How to perform the archive properly?
- To do this, the first condition is use limit in the delete.
- Create an index on the where clause.
- At least do sleep 1sec for each chuck which will be good for a replication infra.
- Set autocommit=1
- Optional: Set transaction isolation to Read Committed.
- Do not mention the number of loops without knowing the actual loop counts to process the complete delete.
My approach to this:
Inspired by Rick James’s Blog, I have prepared a single stored procedure to perform archive on multiple tables. We just need to pass the table name, date column and then date to archive. I have tested with datetime and Primary key column.
Archive a single table:
The below procedure will perform delete on table test and remove older than 10 days records.
use sqladmin; DROP PROCEDURE
IF EXISTS archive;
delimiter //
CREATE PROCEDURE
archive()
begin
DECLARE rows INT;
DECLARE rows_deleted INT;
SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET rows = 1;
SET rows_deleted = 10000;
WHILE rows > 0
do
SET autocommit=1;
DELETE
FROM test
WHERE dop < DATE(Date_sub(Now(), INTERVAL 10 day))
LIMIT 10000;
SET rows = row_count();
select sleep(1);
commit;
END WHILE;
END //
delimiter ;
Archive multiple tables:
This procedure will help you to archive multiple tables, you just need to pass the table name, column name and the date for the archive. I love to use this
use sqladmin; DROP PROCEDURE
IF EXISTS sqladmin_archive;
delimiter //
CREATE PROCEDURE
sqladmin_archive(IN archive_dbname varchar(100), IN archive_table varchar(100), IN archive_column varchar(100), IN archive_date varchar(100)) begin
DECLARE rows INT;
DECLARE rows_deleted INT;
SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET rows = 1;
SET rows_deleted = 10000;
WHILE rows > 0
do
SET autocommit=1;
SET @query =CONCAT('DELETE FROM ',archive_dbname,'.',archive_table,' WHERE ',archive_column,' <= "',archive_date ,'" LIMIT 10000;');
PREPARE arcive_stmt FROM @query;
EXECUTE arcive_stmt;
SET rows = row_count();
SET rows = row_count();
select sleep(1);
commit;
DEALLOCATE PREPARE arcive_stmt;
END WHILE;
END //
delimiter ; -- Execute this procedure
CALL sqladmin_archive ('mydb','test_table','created_at','2018-09-12');
Take dump before archive with where clause:
This script is my favorite one, but this depends on the above stored procedure. This shell script will take the dump of the table with where clause of the date that we want to archive. You can customize this as per your requirement.
#!/bin/bash # pass variables
archive_dbname=$1
archive_table=$2
archive_column=$3
days_to_archive=$4
archive_date="'"`date +'%Y-%m-%d' --date="-$days_to_archive day"`"'"
where_clause=$archive_column'<='$archive_date
dump_file=$archive_table_`date +'%Y-%m-%d' --date="-$days_to_archive day"`".sql" # Dump the table
echo "DUMP Starting for the table $archive_table ....."
mysqldump -u root $archive_dbname $archive_table --where=$where_clause > $dump_file
echo "DUMP Done......" # Archive the data
echo "Deleting the data on the table $archive_table ....."
mysql -u root sqladmin -e"CALL sqladmin_archive('$archive_dbname','$archive_table','$archive_column',$archive_date);"
echo "Deleting is Done ....."
Example Archive:
This example, Im going to archive a table called test. The column started_at contains the timestamp value. I want to remove older than 15 days data in the table. This table is located in the database name called sqladmin.
./archive_script.sh sqladmin test started_at 15
Archive MySQL Data In Chunks Using Stored Procedure的更多相关文章
- JDBC连接执行 MySQL 存储过程报权限错误:User does not have access to metadata required to determine stored procedure parameter types. If rights can not be granted,
国内私募机构九鼎控股打造APP,来就送 20元现金领取地址:http://jdb.jiudingcapital.com/phone.html 内部邀请码:C8E245J (不写邀请码,没有现金送) 国 ...
- Net连接mysql的公共Helper类MySqlHelper.cs带MySql.Data.dll下载
MySqlHelper.cs代码如下: using System; using System.Collections.Generic; using System.Linq; using System. ...
- How To Call Stored Procedure In Hibernate
How To Call Stored Procedure In Hibernate In this tutorial, you will learn how to call a store proce ...
- [转]Mapping Stored Procedure Parameters in SSIS OLE DB Source Editor
本文转自:http://geekswithblogs.net/stun/archive/2009/03/05/mapping-stored-procedure-parameters-in-ssis-o ...
- 关于Linux和Windows下部署mysql.data.dll的注册问题
mysql ado.net connector下载地址: http://dev.mysql.com/downloads/connector/net/ 选择版本: Generally Available ...
- Stored Procedure 里的 WITH RECOMPILE 到底是干麻的?
在 SQL Server 创建或修改「存储过程(stored procedure)」时,可加上 WITH RECOMPILE 选项,但多数文档或书籍都写得语焉不详,或只解释为「每次执行此存储过程时,都 ...
- [转]Dynamic SQL & Stored Procedure Usage in T-SQL
转自:http://www.sqlusa.com/bestpractices/training/scripts/dynamicsql/ Dynamic SQL & Stored Procedu ...
- Retrieving Out Params From a Stored Procedure With Python
http://www.rodneyoliver.com/blog/2013/08/08/retrieving-out-params-from-a-stored-procedure-with-pytho ...
- [转]Easy Stored Procedure Output Oracle Select
本文转自:http://www.oraclealchemist.com/oracle/easy-stored-procedure-output/ I answered a question on a ...
随机推荐
- Vue笔记:VS Code 常用快捷键
VS Code 常用快捷键 1.注释: 单行注释:ctrl+/, 注释后再按取消 取消单行注释:alt+shift+A 注释后再按取消 2.移动行 向上移动一行:alt+up 向下移动一行:alt+d ...
- 百度&高德地图小区景点边界轮廓实现
经常的我们在使用地图功能时,会发现在选择一个小区或者一个热门景点的时候,地图上面会给出其边界轮廓,能够方便我们知道其范围大小,有时候在我们使用地图组件的时候,也会面临着类似的需求.比如在地图上面标识出 ...
- mybatis插入数据库 返回主键
传递参数为对象TaskEntity, 返回主键结果为Integer 与 主键 task_id 的类型一致即可 <insert id="addTask" parameterTy ...
- shell命令——if
if中[ ]实际上调用的是test的一种快捷方法.bash的数值和字符串比较运算符: 注意=两边的空格 字符串 数值 为真,如果 x = y x -eq y x != y x -ne y x ...
- PowerBuilder编程新思维2:嵌入(Thread多线程)
PowerBuilder编程新思维2:嵌入(Thread多线程) 在PB中使用多线程,在网上有大量的文章介绍.不过深入研究并试着给出更易用的模型的,目前还只有"路人甲cw"的一篇& ...
- Java 并发编程常识 —— by 梁飞
参考 :梁飞 并发编程常识
- [转]DevOps的三大原则
本文转自:https://blog.csdn.net/difffate/article/details/77542768 DevOps的出现有其必然性.在软件开发生命周期中,遇到了两次瓶颈. 第一次瓶 ...
- Linux-mkdosfs格式化磁盘命令(15)
名称:mkdosfs 使用:mkdosfs [块设备名称] 说明: 将一个块设备格式化为DOS磁盘类型 例: mkdosfs /dev/memblock //将memblock块设备格式化为dos磁盘 ...
- Java基础——线程
一. 进程 是指一个内存中运行的应用程序,每个进程都有自己独立的一块内存空间,一个进程中可以启动多个线程. 比如在Windows系统中,一个运行的exe就是一个进程. 二.线程 是指进程中的一个执行流 ...
- 非常可乐(杭电hdu1495)bfs
非常可乐 Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 32768/32768 K (Java/Others) Total Sub ...