Archive MySQL Data In Chunks Using Stored Procedure
sqladminon September 26, 2018
In a DBA’s day to day activities, we are doing Archive operation on our transnational database servers to improve your queries and control the Disk space. The archive is a most expensive operation since its involved a huge number of Read and Write will be performed. So its mandatory to run the archive queries in chunks. The archive is depended on business use. Many of us need a copy of the data on an archive database to refer later. To perform the archive we can just simply run the delete query with the limit. But we need to run the query again and again until the matched rows count is 0. We can create a procedure to do this in a while loop. I have created one such procedure to archive many tables.

Why Archive is an expensive operation?
Generally how we are arching the data is delete from table_name where column_name <= some_value; If you are running on a table which needs to be deleted around 15million records, then you need the undo log file to hold all of these records. There will be a heavy IO happening in the Disk. And lt’ll lock the rows and some other locks will be held until the Archive complete. Replication may delay because of this.
When Archive is going to mess up the production?
- Running archive commands on a heavy traffic time.
- Archive without a proper where clause.
- Delete data without limit.
- Performing archive contrition on a not indexed column.
- Continuously run the delete query in chunks on a replication environment. {without sleep(1 or few seconds}.
How to perform the archive properly?
- To do this, the first condition is use limit in the delete.
- Create an index on the where clause.
- At least do sleep 1sec for each chuck which will be good for a replication infra.
- Set autocommit=1
- Optional: Set transaction isolation to Read Committed.
- Do not mention the number of loops without knowing the actual loop counts to process the complete delete.
My approach to this:
Inspired by Rick James’s Blog, I have prepared a single stored procedure to perform archive on multiple tables. We just need to pass the table name, date column and then date to archive. I have tested with datetime and Primary key column.
Archive a single table:
The below procedure will perform delete on table test and remove older than 10 days records.
use sqladmin; DROP PROCEDURE
IF EXISTS archive;
delimiter //
CREATE PROCEDURE
archive()
begin
DECLARE rows INT;
DECLARE rows_deleted INT;
SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET rows = 1;
SET rows_deleted = 10000;
WHILE rows > 0
do
SET autocommit=1;
DELETE
FROM test
WHERE dop < DATE(Date_sub(Now(), INTERVAL 10 day))
LIMIT 10000;
SET rows = row_count();
select sleep(1);
commit;
END WHILE;
END //
delimiter ;
Archive multiple tables:
This procedure will help you to archive multiple tables, you just need to pass the table name, column name and the date for the archive. I love to use this
use sqladmin; DROP PROCEDURE
IF EXISTS sqladmin_archive;
delimiter //
CREATE PROCEDURE
sqladmin_archive(IN archive_dbname varchar(100), IN archive_table varchar(100), IN archive_column varchar(100), IN archive_date varchar(100)) begin
DECLARE rows INT;
DECLARE rows_deleted INT;
SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET rows = 1;
SET rows_deleted = 10000;
WHILE rows > 0
do
SET autocommit=1;
SET @query =CONCAT('DELETE FROM ',archive_dbname,'.',archive_table,' WHERE ',archive_column,' <= "',archive_date ,'" LIMIT 10000;');
PREPARE arcive_stmt FROM @query;
EXECUTE arcive_stmt;
SET rows = row_count();
SET rows = row_count();
select sleep(1);
commit;
DEALLOCATE PREPARE arcive_stmt;
END WHILE;
END //
delimiter ; -- Execute this procedure
CALL sqladmin_archive ('mydb','test_table','created_at','2018-09-12');
Take dump before archive with where clause:
This script is my favorite one, but this depends on the above stored procedure. This shell script will take the dump of the table with where clause of the date that we want to archive. You can customize this as per your requirement.
#!/bin/bash # pass variables
archive_dbname=$1
archive_table=$2
archive_column=$3
days_to_archive=$4
archive_date="'"`date +'%Y-%m-%d' --date="-$days_to_archive day"`"'"
where_clause=$archive_column'<='$archive_date
dump_file=$archive_table_`date +'%Y-%m-%d' --date="-$days_to_archive day"`".sql" # Dump the table
echo "DUMP Starting for the table $archive_table ....."
mysqldump -u root $archive_dbname $archive_table --where=$where_clause > $dump_file
echo "DUMP Done......" # Archive the data
echo "Deleting the data on the table $archive_table ....."
mysql -u root sqladmin -e"CALL sqladmin_archive('$archive_dbname','$archive_table','$archive_column',$archive_date);"
echo "Deleting is Done ....."
Example Archive:
This example, Im going to archive a table called test. The column started_at contains the timestamp value. I want to remove older than 15 days data in the table. This table is located in the database name called sqladmin.
./archive_script.sh sqladmin test started_at 15
Archive MySQL Data In Chunks Using Stored Procedure的更多相关文章
- JDBC连接执行 MySQL 存储过程报权限错误:User does not have access to metadata required to determine stored procedure parameter types. If rights can not be granted,
国内私募机构九鼎控股打造APP,来就送 20元现金领取地址:http://jdb.jiudingcapital.com/phone.html 内部邀请码:C8E245J (不写邀请码,没有现金送) 国 ...
- Net连接mysql的公共Helper类MySqlHelper.cs带MySql.Data.dll下载
MySqlHelper.cs代码如下: using System; using System.Collections.Generic; using System.Linq; using System. ...
- How To Call Stored Procedure In Hibernate
How To Call Stored Procedure In Hibernate In this tutorial, you will learn how to call a store proce ...
- [转]Mapping Stored Procedure Parameters in SSIS OLE DB Source Editor
本文转自:http://geekswithblogs.net/stun/archive/2009/03/05/mapping-stored-procedure-parameters-in-ssis-o ...
- 关于Linux和Windows下部署mysql.data.dll的注册问题
mysql ado.net connector下载地址: http://dev.mysql.com/downloads/connector/net/ 选择版本: Generally Available ...
- Stored Procedure 里的 WITH RECOMPILE 到底是干麻的?
在 SQL Server 创建或修改「存储过程(stored procedure)」时,可加上 WITH RECOMPILE 选项,但多数文档或书籍都写得语焉不详,或只解释为「每次执行此存储过程时,都 ...
- [转]Dynamic SQL & Stored Procedure Usage in T-SQL
转自:http://www.sqlusa.com/bestpractices/training/scripts/dynamicsql/ Dynamic SQL & Stored Procedu ...
- Retrieving Out Params From a Stored Procedure With Python
http://www.rodneyoliver.com/blog/2013/08/08/retrieving-out-params-from-a-stored-procedure-with-pytho ...
- [转]Easy Stored Procedure Output Oracle Select
本文转自:http://www.oraclealchemist.com/oracle/easy-stored-procedure-output/ I answered a question on a ...
随机推荐
- 第三方登录:QQ登录实现(OAuth2.0)
一.创建应用 1.在 QQ互联 创建应用 地址:https://connect.qq.com/manage.html#/ 然后进行实名认证,创建应用,审核通过 然后点击查看,可以获得 APP ID 和 ...
- j2ee高级开发技术课程第三周
一.分析Filter例子(轻量级javaee企业应用实战p132) // 执行过滤的核心方法 public void doFilter(ServletRequest request, ServletR ...
- Windows x86 下的 静态代码混淆
0x00 前言 静态反汇编之王,毫无疑问就是Ida pro,大大降低了反汇编工作的门槛,尤其是出色的“F5插件”Hex-Rays可以将汇编代码还原成类似于C语言的伪代码,大大提高了可读性.但个人觉得 ...
- INTEST/EXTEST SCAN
INTEST scan指的是对IP 内部的scan cell的扫描测试,针对IP内部的flip-flop进行shift/capture的操作.和INTEST SCAN 对应的就是EXTEST SCAN ...
- [转]Magento 2中文文档教程 - 配置和运行cron(定时任务)
本文转自:https://blog.csdn.net/xz_src/article/details/72793476 cron(定时任务)概述 Magento 2 有许多功能需要用到cron(定时任务 ...
- 检测SQLServer复制订阅进度
--检测复制订阅进度 --SQLSERVER2012 --创建链接服务器ReadServer,WriteServer,它们分别链接发布服务器.订阅服务器 --修改数据库名称 --在分发服务器执行 US ...
- [日常] Go语言圣经--接口约定习题2
练习 7.3: 为在gopl.io/ch4/treesort (§4.4)的*tree类型实现一个String方法去展示tree类型的值序列. package main import( "f ...
- [日常] go语言圣经-声明,变量,赋值,类型,包和文件习题
go语言圣经-声明1.四种类型的声明语句:var.const.type和func,分别对应变量.常量.类型和函数实体对象的声明2.包一级声明语句声明的名字可在整个包对应的每个源文件中访问,局部声明的名 ...
- MySQL的相关应用
本文主要介绍数据库中MySQL的基础知识,包括数据插入.数据更新.数据删除.数据查询.函数.CASE语句.表连接.子查询. 一.数据插入(insert 语句) 首先我创建一个表,如下: 接着,进行数据 ...
- java8 集合神操作
public class StreamUtils { private static final List<Integer> listInteger = Lists.newArrayList ...