MySQL优化--IO调度算法优化
之前已经在微信公众号分享了数据库优化的方法，链接为https://mp.weixin.qq.com/s/6Atzk9UKPJRxxAs0nsKBXg 。 其中操作系统部分介绍了IO调度算法的优化，本文将通过压力测试的方式来对比不同的调度算法下磁盘IO的表现。
1 准备工作
1.1 安装sysbench
本次采用sysbench进行压测，先安装sysbench，步骤如下：
curl -s https://packagecloud.io/install/repositories/akopytov/sysbench/script.rpm.sh | sudo bash
yum -y install sysbench
sysbench --version
1.2 准备测试文件
生成后续需要使用的测试文件，block大小为16k（MySQL DBA都懂的，哈哈），创建4个文件，合计20G
[root@mha1 ~]# sysbench fileio --file-num= --file-block-size= --file-total-size=20G prepare
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) files, 5242880Kb each, 20480Mb total
Creating files for the test...
Extra file open flags: (none)
Creating file test_file.
Creating file test_file.
Creating file test_file.
Creating file test_file.
bytes written in 47.94 seconds (427.24 MiB/sec).
1.3 准备测试表
因为也要进行数据库读写方面的测试，因此需要先创建相关表及数据
[root@mha1 ~]# sysbench --db-driver=mysql --time= --threads= --report-interval= --mysql-host=127.0.0.1 --mysql-port= --mysql-user=root --mysql-password= --mysql-db=testdb --tables= --table_size= oltp_insert --db-ps-mode=disable prepare
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) Initializing worker threads... Creating table 'sbtest3'...
Creating table 'sbtest2'...
Creating table 'sbtest5'...
Creating table 'sbtest6'...
Creating table 'sbtest8'...
Creating table 'sbtest7'...
Creating table 'sbtest4'...
Creating table 'sbtest1'...
Creating table 'sbtest10'...
Creating table 'sbtest9'...
Inserting records into 'sbtest4'
Inserting records into 'sbtest7'
Inserting records into 'sbtest5'
Inserting records into 'sbtest3'
Inserting records into 'sbtest1'
Inserting records into 'sbtest2'
Inserting records into 'sbtest8'
Inserting records into 'sbtest6'
Inserting records into 'sbtest10'
Inserting records into 'sbtest9'
Creating a secondary index on 'sbtest7'...
Creating a secondary index on 'sbtest10'...
Creating a secondary index on 'sbtest8'...
Creating a secondary index on 'sbtest5'...
Creating a secondary index on 'sbtest2'...
Creating a secondary index on 'sbtest9'...
Creating a secondary index on 'sbtest1'...
Creating table 'sbtest17'...
Creating a secondary index on 'sbtest3'...
Inserting records into 'sbtest17'
Creating a secondary index on 'sbtest4'...
Creating a secondary index on 'sbtest6'...
Creating table 'sbtest20'...
Inserting records into 'sbtest20'
Creating table 'sbtest18'...
Inserting records into 'sbtest18'
Creating table 'sbtest15'...
Inserting records into 'sbtest15'
Creating table 'sbtest19'...
Inserting records into 'sbtest19'
Creating table 'sbtest14'...
Inserting records into 'sbtest14'
Creating table 'sbtest11'...
Inserting records into 'sbtest11'
Creating table 'sbtest13'...
Creating table 'sbtest12'...
Inserting records into 'sbtest13'
Inserting records into 'sbtest12'
Creating table 'sbtest16'...
Inserting records into 'sbtest16'
Creating a secondary index on 'sbtest17'...
Creating a secondary index on 'sbtest20'...
Creating a secondary index on 'sbtest18'...
Creating a secondary index on 'sbtest19'...
Creating a secondary index on 'sbtest15'...
Creating a secondary index on 'sbtest11'...
Creating a secondary index on 'sbtest12'...
Creating a secondary index on 'sbtest13'...
Creating a secondary index on 'sbtest14'...
Creating a secondary index on 'sbtest16'...
2 查看支持的调度算法
本次磁盘为SSD硬盘，操作系统版本文Centos7.8 。下面将调度算法修改为三种不同的值来进行随机读与随机写的压力测试
本系统为Centos7.8,需要查看支持的IO调度算法，然后再进行修改测试。
[root@mha1 ~]# dmesg | grep -i scheduler
[ 4.885816] io scheduler noop registered
[ 4.885820] io scheduler deadline registered (default)
[ 4.885867] io scheduler cfq registered
[ 4.885870] io scheduler mq-deadline registered
[ 4.885872] io scheduler kyber registered
可见，再本系统中，默认的调度算法为 deadline。
也可以通过如下命令查看当前的调度算法，其中中括号里代表当前使用的调度算法。
[root@mha1 ~]# cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
3 deadline算法
Deadline在机械盘的情况下对数据库环境(ORACLE RAC,MySQL等)是最好的选择。下面将进行随机写与随机读的压力测试
3.1 随机写
[root@mha1 ~]# sysbench fileio \
> --time= \
> --threads= \
> --file-total-size=20G \
> --file-test-mode=rndwr \
> --file-num= \
> --file-extra-flags=direct \
> --file-fsync-freq= \
> --file-block-size= \
> run
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) Running the test with following options:
Number of threads:
Initializing random number generator from current time Extra file open flags: directio
files, 5GiB each
20GiB total file size
Block size 16KiB
Number of IO requests:
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random write test
Initializing worker threads... Threads started! File operations:
reads/s: 0.00
writes/s: 6935.37
fsyncs/s: 0.53 Throughput:
read, MiB/s: 0.00
written, MiB/s: 108.37 General statistics:
total time: .0138s
total number of events: Latency (ms):
min: 0.10
avg: 3.46
max: 107.39
95th percentile: 14.73
sum: 4317610.93 Threads fairness:
events (avg/stddev): 52020.1667/426.95
execution time (avg/stddev): 179.9005/0.01
随机写的iops为6935.37，磁盘写入速度是108.37MiB/s
3.2 随机读
[root@mha1 ~]# sysbench fileio \
> --time= \
> --threads= \
> --file-total-size=20G \
> --file-test-mode=rndrd \
> --file-num= \
> --file-extra-flags=direct \
> --file-fsync-freq= \
> --file-block-size= \
> run
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) Running the test with following options:
Number of threads:
Initializing random number generator from current time Extra file open flags: directio
files, 5GiB each
20GiB total file size
Block size 16KiB
Number of IO requests:
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Initializing worker threads... Threads started! File operations:
reads/s: 7956.88
writes/s: 0.00
fsyncs/s: 0.00 Throughput:
read, MiB/s: 124.33
written, MiB/s: 0.00 General statistics:
total time: .0075s
total number of events: Latency (ms):
min: 0.10
avg: 3.01
max: 322.24
95th percentile: 5.47
sum: 4309094.67 Threads fairness:
events (avg/stddev): 59679.7083/2688.56
execution time (avg/stddev): 179.5456/0.18
随机读的iops为7956.88，磁盘读取速度是124.33MiB/s
3.3 测试数据库写
sysbench --db-driver=mysql --time= --threads= --report-interval= --mysql-host=127.0.0.1 --mysql-port= --mysql-user=root --mysql-password= --mysql-db=testdb --tables= --table_size= oltp_write_only --db-ps-mode=disable run SQL statistics:
queries performed:
read:
write:
other:
total:
transactions: (491.43 per sec.)
queries: (2948.62 per sec.)
ignored errors: (0.01 per sec.)
reconnects: (0.00 per sec.) General statistics:
total time: .1050s
total number of events: Latency (ms):
min: 2.58
avg: 16.27
max: 2608.34
95th percentile: 35.59
sum: 2399415.58 Threads fairness:
events (avg/stddev): 18435.3750/90.33
execution time (avg/stddev): 299.9269/0.04
可见，随机写入的TPS为491.43 ,查询次数为2948.62
3.4 测试数据库读
sysbench --db-driver=mysql --time= --threads= --report-interval= --mysql-host=127.0.0.1 --mysql-port= --mysql-user=root --mysql-password= --mysql-db=testdb --tables= --table_size= oltp_read_only --db-ps-mode=disable run SQL statistics:
queries performed:
read:
write:
other:
total:
transactions: (393.13 per sec.)
queries: (6290.13 per sec.)
ignored errors: (0.00 per sec.)
reconnects: (0.00 per sec.) General statistics:
total time: .0949s
total number of events: Latency (ms):
min: 3.08
avg: 20.34
max: 170.48
95th percentile: 29.19
sum: 2399636.31 Threads fairness:
events (avg/stddev): 14747.2500/1513.84
execution time (avg/stddev): 299.9545/0.04
可见，随机读时的TPS为393.13 ,查询次数为6290.13
4 noop算法
4.1 修改为noop算法
noop称为电梯调度算法，是基于FIFO队列实现的。所有的请求都是先进先出的，因为SSD的随机读、随机写速度快，因此该算法适合SSD硬盘。
[root@mha1 ~]# echo 'noop' >/sys/block/sda/queue/scheduler
[root@mha1 ~]# cat /sys/block/sda/queue/scheduler
[noop] deadline cfq
4.2 随机写
[root@mha1 ~]# sysbench fileio \
> --time= \
> --threads= \
> --file-total-size=20G \
> --file-test-mode=rndwr \
> --file-num= \
> --file-extra-flags=direct \
> --file-fsync-freq= \
> --file-block-size= \
> run
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) Running the test with following options:
Number of threads:
Initializing random number generator from current time Extra file open flags: directio
files, 5GiB each
20GiB total file size
Block size 16KiB
Number of IO requests:
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random write test
Initializing worker threads... Threads started! File operations:
reads/s: 0.00
writes/s: 7057.60
fsyncs/s: 0.53 Throughput:
read, MiB/s: 0.00
written, MiB/s: 110.27 General statistics:
total time: .0136s
total number of events: Latency (ms):
min: 0.10
avg: 3.40
max: 240.39
95th percentile: 14.46
sum: 4317435.99 Threads fairness:
events (avg/stddev): 52936.7083/487.57
execution time (avg/stddev): 179.8932/0.02
随机写的iops为7057.60，磁盘写入速度是110.27MiB/s
4.3 随机读
[root@mha1 ~]# sysbench fileio \
> --time= \
> --threads= \
> --file-total-size=20G \
> --file-test-mode=rndrd \
> --file-num= \
> --file-extra-flags=direct \
> --file-fsync-freq= \
> --file-block-size= \
> run
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) Running the test with following options:
Number of threads:
Initializing random number generator from current time Extra file open flags: directio
files, 5GiB each
20GiB total file size
Block size 16KiB
Number of IO requests:
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Initializing worker threads... Threads started! File operations:
reads/s: 8399.89
writes/s: 0.00
fsyncs/s: 0.00 Throughput:
read, MiB/s: 131.25
written, MiB/s: 0.00 General statistics:
total time: .0100s
total number of events: Latency (ms):
min: 0.10
avg: 2.85
max: 315.77
95th percentile: 5.00
sum: 4312384.33 Threads fairness:
events (avg/stddev): 63003.3750/10086.77
execution time (avg/stddev): 179.6827/0.12
随机读的iops为8399.89，磁盘读取速度是131.25MiB/s
4.4 数据库写入
sysbench --db-driver=mysql --time= --threads= --report-interval= --mysql-host=127.0.0.1 --mysql-port= --mysql-user=root --mysql-password= --mysql-db=testdb --tables= --table_size= oltp_write_only --db-ps-mode=disable run SQL statistics:
queries performed:
read:
write:
other:
total:
transactions: (544.38 per sec.)
queries: (3266.28 per sec.)
ignored errors: (0.01 per sec.)
reconnects: (0.00 per sec.) General statistics:
total time: .0903s
total number of events: Latency (ms):
min: 2.62
avg: 14.69
max: 220.12
95th percentile: 32.53
sum: 2399040.57 Threads fairness:
events (avg/stddev): 20420.5000/112.69
execution time (avg/stddev): 299.8801/0.04
可见，随机写入的TPS为 544.38 ,查询次数为3266.28
4.5 数据库只读
sysbench --db-driver=mysql --time= --threads= --report-interval= --mysql-host=127.0.0.1 --mysql-port= --mysql-user=root --mysql-password= --mysql-db=testdb --tables= --table_size= oltp_read_only --db-ps-mode=disable run
SQL statistics:
queries performed:
read:
write:
other:
total:
transactions: (379.97 per sec.)
queries: (6079.59 per sec.)
ignored errors: (0.00 per sec.)
reconnects: (0.00 per sec.) General statistics:
total time: .0869s
total number of events: Latency (ms):
min: 3.08
avg: 21.04
max: 321.03
95th percentile: 31.37
sum: 2399600.56 Threads fairness:
events (avg/stddev): 14253.2500/1475.71
execution time (avg/stddev): 299.9501/0.02
可见，只读时的TPS为 379.97,查询次数为6079.59
5 cfq算法
5.1 修改为cfq算法
cfq称为绝对公平调度算法，为每个进程及线程单独创建一个队列来管理IO请求，起到每个进程和线程均匀分布IO的效果。此算法适用于通用服务器，centos6中为默认的IO调度算法。
[root@mha1 ~]# echo 'cfq' >/sys/block/sda/queue/scheduler
[root@mha1 ~]# cat /sys/block/sda/queue/scheduler
noop deadline [cfq]
5.2 随机写
[root@mha1 ~]# sysbench fileio \
> --time= \
> --threads= \
> --file-total-size=20G \
> --file-test-mode=rndwr \
> --file-num= \
> --file-extra-flags=direct \
> --file-fsync-freq= \
> --file-block-size= \
> run
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) Running the test with following options:
Number of threads:
Initializing random number generator from current time Extra file open flags: directio
files, 5GiB each
20GiB total file size
Block size 16KiB
Number of IO requests:
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random write test
Initializing worker threads... Threads started! File operations:
reads/s: 0.00
writes/s: 6614.37
fsyncs/s: 0.53 Throughput:
read, MiB/s: 0.00
written, MiB/s: 103.35 General statistics:
total time: .0118s
total number of events: Latency (ms):
min: 0.10
avg: 3.63
max: 348.78
95th percentile: 15.27
sum: 4317092.54 Threads fairness:
events (avg/stddev): 49611.5417/517.80
execution time (avg/stddev): 179.8789/0.03
随机写的iops为6614.37，磁盘写入速度是103.35MiB/s
5.3 随机读
[root@mha1 ~]# sysbench fileio \
> --time= \
> --threads= \
> --file-total-size=20G \
> --file-test-mode=rndrd \
> --file-num= \
> --file-extra-flags=direct \
> --file-fsync-freq= \
> --file-block-size= \
> run
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) Running the test with following options:
Number of threads:
Initializing random number generator from current time Extra file open flags: directio
files, 5GiB each
20GiB total file size
Block size 16KiB
Number of IO requests:
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Initializing worker threads... Threads started! File operations:
reads/s: 7481.39
writes/s: 0.00
fsyncs/s: 0.00 Throughput:
read, MiB/s: 116.90
written, MiB/s: 0.00 General statistics:
total time: .0086s
total number of events: Latency (ms):
min: 0.10
avg: 3.20
max: 374.49
95th percentile: 5.77
sum: 4312382.07 Threads fairness:
events (avg/stddev): 56113.7917/3058.00
execution time (avg/stddev): 179.6826/0.17
随机读的iops为7481.39，磁盘读取速度是116.90MiB/s
5.4 数据库写
sysbench --db-driver=mysql --time= --threads= --report-interval= --mysql-host=127.0.0.1 --mysql-port= --mysql-user=root --mysql-password= --mysql-db=testdb --tables= --table_size= oltp_write_only --db-ps-mode=disable run
SQL statistics:
queries performed:
read:
write:
other:
total:
transactions: (498.54 per sec.)
queries: (2991.25 per sec.)
ignored errors: (0.01 per sec.)
reconnects: (0.00 per sec.) General statistics:
total time: .2552s
total number of events: Latency (ms):
min: 2.55
avg: 16.02
max: 779.62
95th percentile: 35.59
sum: 2397311.08 Threads fairness:
events (avg/stddev): 18711.3750/132.24
execution time (avg/stddev): 299.6639/0.38
可见，随机写入的TPS为498.54 ,查询次数为2991.25
5.5 数据库读
sysbench --db-driver=mysql --time= --threads= --report-interval= --mysql-host=127.0.0.1 --mysql-port= --mysql-user=root --mysql-password= --mysql-db=testdb --tables= --table_size= oltp_read_only --db-ps-mode=disable run
SQL statistics:
queries performed:
read:
write:
other:
total:
transactions: (344.66 per sec.)
queries: (5514.58 per sec.)
ignored errors: (0.00 per sec.)
reconnects: (0.00 per sec.) General statistics:
total time: .1562s
total number of events: Latency (ms):
min: 3.11
avg: 23.19
max: 222.31
95th percentile: 38.25
sum: 2399486.55 Threads fairness:
events (avg/stddev): 12931.6250/1278.72
execution time (avg/stddev): 299.9358/0.01
可见，只读时的TPS为 344.66,查询次数为5514.58
6 小结
根据测试结果对比一下三种调度算法的读写速度
| 算法 | IOPS | 磁盘写速度 | IOPS | 磁盘读速度 | oltp_write_only | oltp_read_only |
| deadline | 6935.37 | 118.37MiB/s | 7956.88 | 124.33MiB/s | TPS为491.43 ,查询次数为2948.62 | TPS为393.13 ,查询次数为6290.13 |
| noop | 7057.60 | 110.27MiB/s | 8399.89 | 131.25MiB/s | TPS为 544.38 ,查询次数为3266.28 | TPS为 379.97,查询次数为6079.59 |
| cfq | 6614.37 | 103.35MiB/s | 7481.39 | 116.90MiB/s | TPS为498.54 ,查询次数为2991.25 | TPS为 344.66,查询次数为5514.58 |
因为本次测试环境为SSD硬盘，因此，在此情况下建议选择noop磁盘IO调度算法，此结论也符合我们的预期。
特别注意：磁盘IO的调度算法还需要根据磁盘情况、数据库类型、数据库架构、业务场景（OLTP、OLAP等）等各种场景进行区分，不同的场景调度算法也要调整，不可一概而论。如果不确定的话，建议进行压测来判断，选择符合对应场景下最合适的算法。
想了解更多内容或参与技术交流可以关注微信公众号【数据库干货铺】或进技术交流群沟通。
MySQL优化--IO调度算法优化的更多相关文章
- MySQL数据库在IO性能优化方面的设置选择(硬件)
提起MySQL数据库在硬件方面的优化无非是CPU.内存和IO.下面我们着重梳理一下关于磁盘I/O方面的优化. 1.磁盘冗余阵列RAID RAID(Redundant Array of Inexpens ...
- 修改numa和io调度优化mysql性能
一.NUMA设置单机单实例,建议关闭NUMA,关闭的方法有三种:1.硬件层,在BIOS中设置关闭:2.OS内核,启动时设置numa=off:3.可以用numactl命令将内存分配策略修改为interl ...
- MySQL服务器 IO 100%的分析与优化方案
前言 压力测试过程中,如果因为资源使用瓶颈等问题引发最直接性能问题是业务交易响应时间偏大,TPS逐渐降低等.而问题定位分析通常情况下,最优先排查的是监控服务器资源利用率,例如先用TOP 或者nmon等 ...
- 美图秀秀DBA谈MySQL运维及优化
美图秀秀DBA谈MySQL运维及优化 https://mp.weixin.qq.com/s?__biz=MzI4NTA1MDEwNg==&mid=401797597&idx=2& ...
- 性能调优之MySQL篇四：MySQL配置定位以及优化
一.CPU最大性能模式 cpu利用特点 5.1 最高可用4个核 5.5 最高可用24核 5.6 最高可用64核心 一次query对应一个逻辑CPU 你仔细检查的话,有些服务器上会有的一个有趣的现象:你 ...
- [MySQL Reference Manual] 8 优化
8.优化 8.优化 8.1 优化概述 8.2 优化SQL语句 8.2.1 优化SELECT语句 8.2.1.1 SELECT语句的速度 8.2.1.2 WHERE子句优化 8.2.1.3 Range优 ...
- MySql学习(六) —— 数据库优化理论(二) —— 查询优化技术
逻辑查询优化包括的技术 1)子查询优化 2)视图重写 3)等价谓词重写 4)条件简化 5)外连接消除 6)嵌套连接消除 7)连接消除 8)语义优化 9)非SPJ优化 一.子查询优化 1. ...
- MySql学习(五) —— 数据库优化理论篇(一)
一.数据库管理系统 数据库管理系统(Database Management System, DBMS) 衡量是否是数据库的标准: ACID:是指在数据库管理系统(DBMS)中事务所具有的四个特性: 1 ...
- mysql内存使用以及优化中需要的几点注意
1.从内存中读取数据是微秒级别的.而从磁盘读则是毫秒级别的.二者相差一个数量级.所以想优化数据库,第一个要做到的就是优化io. 2.key_buffer_size[global]设置的内存区域大小缓存 ...
随机推荐
- Django学习路8_学生表和班级表级联并相互查询信息
在创建好的 app3 目录下的 models.py 中 创建对 数据表(学生和班级)的描述 分析: 学生和班级是一对多的关系 班级表的设计 class Grade(models.Model): # 设 ...
- PHP substr_count() 函数
实例 计算 "world" 在字符串中出现的次数: <?php高佣联盟 www.cgewang.comecho substr_count("Hello world. ...
- 6.28 NOI模拟赛 好题 状压dp 随机化
算是一道比较新颖的题目 尽管好像是两年前的省选模拟赛题目.. 对于20%的分数 可以进行爆搜,对于另外20%的数据 因为k很小所以考虑上状压dp. 观察最后答案是一个连通块 从而可以发现这个连通块必然 ...
- Prometheus的伴侣：Grafana在centos下的搭建
Grafana 是一款采用 go 语言编写的开源应用,主要用于监控指标数据的可视化展现,是当前最流行的时序数据展示工具,目前已经支持绝大部分常用的时序数据库.Grafana常常搭配用作Promethe ...
- python range函数的用法
range 函数是Python内置函数.可创建一个整数列表,一般用在 for 循环中. 函数语法:range(start, stop[, step]) start: 计数从 start 开始.默认是从 ...
- Python的10个神奇的技巧
尽管从表面上看,Python似乎是任何人都可以学习的一种简单语言,但确实如此,许多人可能惊讶地知道一个人可以熟练掌握该语言. Python是其中的一门很容易学习的东西,但可能很难掌握. 在Python ...
- 编程与算法（一）、C语言实现二分法（方程近似解）
一.二分法 假设有这样一个函数f(x) 函数与x轴有一个交点(也就是f(a)*f(b)<0,a<b),现在我们要求这个点的x值,也就是方程f(x)=0的一个实根 直接解显然不合适,那么接下 ...
- Spring IOC 原理深层解析
1 Spring IOC概念认识 1.1 区别IOC与DI 首先我们要知道IOC(Inverse of Control:控制反转)是一种设计思想,就是 将原本在程序中手动创建对象的控制权,交由Spri ...
- Python实现迪杰斯特拉算法
首先我采用邻接矩阵法来表示图(有向图无向图皆可) 图的定义如下: class Graph: def __init__(self, arcs=[]): self.vexs = [] self.arcs ...
- 提前批笔试一道算法题的Java实现
题目描述 这是2021广联达校招提前批笔试算法题之一. 我们希望一个序列中的元素是各不相同的,但是理想和显示往往是有差距的.现在给出一个序列A,其中难免有相同的元素,现在提供了一种变化方式,使得经过若 ...