Measuring & Optimizing I/O Performance
By Ilya Grigorik on June 23, 2009
Measuring and optimizing IO performance
is somewhat of a black art: the tools are there, the resources and discussions are plenty, but it is also incredibly easy to get lost in the forest. I speak from recent experience. Having gone down multiple false starts with filesystem optimization, RAID tweaking,
and even app-level changes it really helped to finally step back and revisit the basics. Many man pages and discussion threads later, a few useful realizations emerged: iostat is your best friend, but it can also be incredibly deceiving; refreshing your memory
of disk latencies will go a long way; disks and filesystems are fast, but not that fast.
Monitoring IO Performance with iostat
If IO performance is suspect, iostat is your best friend. Having said that, the man pages are cryptic so don't be surprised if you find
yourself reading the source. To get started, identify the device in question and start a monitoring process:
# -k output rates in kB
# -x output extended stats
# -d monitoring single device
# sample stats every 5 seconds for device /dev/sdh
$ iostat -dxk /dev/sdi 5
Next, allocate yourself a couple of hours to understand the output or expect to find yourself down a wrong path in no time flat (been there, done that). iostat is a popular tool amongst the database crowd, so not surprisingly you'll find a lot of great discussions documenting the
use. Depending on your application you will need to focus on different metrics, but as a gentle introduction let's take a look at await, svctime and avgque:
- await - The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
- svctime - The average service time (in milliseconds) for I/O requests that were issued to the device.
- avgqu-sz - The average queue length of the requests that were issued to the device.
First off, await is a deceiving metric! Even though it claims to measure average time, it is better understood
as an aggregate function, so don't be mislead by it: avgqu-sz * svctm / (%util/100). Ideally, await should be roughly equal to your svctime, which leads us to a corollary: your average queue size is ideally
hovering around single digits. Understanding these variables alone can tell you volumes about the application generating the load.
Disk Latencies Refresher & EBS Performance
Disk access
time is determined via the sum of several variables: spin-up, seek, rotational delay, and transfer time. Assuming your disk is not is not sleeping we can discount the spin-up
time, which leaves us with seek (time for the disk arm to find the track: ~10ms), rotational
delay (time to get the right sector under the head: depends on RPM), and the actual transfer time. Hence, in the worst case we will take ~10ms to seek, 60s/7200RPM ~= 8ms in rotational delay, plus the read time. On average, for a 7.2k RPM
disk this translates into roughly ~5ms access time (~20ms in worst case) to read the first byte!
Armed with this knowledge we can now put Amazon's EBS performance in context: on average our EBS mounts show 10~30ms svctime, which all things considered is not outrageous for a SAN. This number also dips into low single digits at nights and on weekends,
which points to the fact that as with any shared resource, the performance of EBS degrades during the day.
Having said that, a 6x performance difference based on time of day is definitely not anything to sneeze at, so let's hope Amazon is on top of this!
Average queue size (avgqu-sz) is a popular metric in the DBA circles, but do be
careful with it when
running on a SAN or any multi-spindle device. Ideally, your queue size (avgqu-sz) for a single disk should be in single digits, which means that the underlying device is well matched to the IO load generated by the application. Conversely,
if the queue size is artificially low, chances are your application code can benefit from some tuning: do less disk flushing, think about caching or buffering, or in other words, double check the assumption that IO is the bottleneck!
Disks, Filesystems and Facebook Case Study: Haystack
Average access time on our disks places some hard
limits on the number of IOPs - at 5ms average, we get a very optimistic 200 req/s with no read time. Hence, if you're trying to store several hundred files a second, you might want to revisit the architecture or seriously think about switching to SSD's! Databases
such as MySQL work around this constraint by minimizing the number of file handles, caching data, and using aggressive buffering techniques. Willing to potentially loose a little bit of data with InnoDB? Set flush_log_at_trx_commit
to 2 to avoid flushing on every transaction in favor of a periodic one second flush. In similar fashion, you can tweak your MyISAM key
buffers, or even place your index and data files on different drives.
Facebook team recently released the details of their Haystack photo storage system which serves as a great case study
of working around the IO bottlenecks: over 15PB of photo storage, and ~360 new photos being uploaded every second as of April '09. To meet the requirements, they dropped the POSIX filesystem semantics and went for an append only structure with a separate in-memory
index which stores the direct inode offsets for each photo. As a result, each photo access is translated into a single IO request - a huge win. Read through it, fascinating
stuffand an illustrative example of optimizing for IO.

Ilya Grigorik is a web performance engineer and developer advocate on the Make The Web Fast team
at Google, where he spends his days and nights on making the web fast and driving adoption of performance best practices.
Follow @igrigorik
Measuring & Optimizing I/O Performance的更多相关文章
- Optimizing Item Import Performance in Oracle Product Hub/Inventory
APPLIES TO: Oracle Product Hub - Version 12.1.1 to 12.1.1 [Release 12.1] Oracle Inventory Management ...
- PatentTips - Optimizing Write Combining Performance
BACKGROUND OF THE INVENTION The use of a cache memory with a processor facilitates the reduction of ...
- [Forward]Improving Web App Performance With the Chrome DevTools Timeline and Profiles
Improving Web App Performance With the Chrome DevTools Timeline and Profiles We all want to create h ...
- Java性能提示(全)
http://www.onjava.com/pub/a/onjava/2001/05/30/optimization.htmlComparing the performance of LinkedLi ...
- Migrating Oracle on UNIX to SQL Server on Windows
Appendices Published: April 27, 2005 On This Page Appendix A: SQL Server for Oracle Professionals Ap ...
- (转) [it-ebooks]电子书列表
[it-ebooks]电子书列表 [2014]: Learning Objective-C by Developing iPhone Games || Leverage Xcode and Obj ...
- 数据库调优过程(一):SqlServer批量复制(bcp)[C#SqlBulkCopy]性能极低问题
背景 最近一段给xx做项目,这边最头疼的事情就是数据库入库瓶颈问题. 环境 服务器环境:虚拟机,分配32CPU,磁盘1.4T,4T,5T,6T几台服务器不等同(转速都是7200r),内存64G. 排查 ...
- 跨过slf4j和logback,直接晋级log4j 2
今年一直关注log4j 2,但至今还没有出正式版.等不及了,今天正式向大家介绍一下log4j的升级框架,log4j 2. log4j,相信大家都熟悉,至今对java影响最大的logging系统,至今仍 ...
- 论文笔记:Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks ICML 2017 Paper:https://arxiv.org/ ...
随机推荐
- TensorBoard:Visualizing Learning 学习笔记
为了更方便的理解.调试和优化TF程序,我们可以使用TensorBoard(可视化工具).可以使用TensorBoard查看graph,绘制图表执行过程中的定量指标.TensorBoard是完全可配置的 ...
- ThinkPhp框架:验证码功能
Think\Verify类可以支持验证码的生成和验证功能. 为了显示这个验证码功能,第一要有控制器,再就是有方法,然后是显示的页面. 一.最简单的方式生成验证码 (1)我们还是继续在那个控制器编写方法 ...
- PHP之流程的控制
在一个公司中,员工肯定会有高层.中层和普通员工的,这样如果在某个公司员工遇到某种问题,需要请假.或者提一些请求的时候,就会需要去申请,申请的时候有的是可以直接和上级申请的,有时比较重要的事情,是 ...
- 获取 metadata 过程详解 - 每天5分钟玩转 OpenStack(167)
接上节,启动 neutron router 后 instance c1 终于拿到了 metadata, 从下面 c1 的启动日志可知: c1 所认为的 metadata 服务地址是 169.254.1 ...
- flex中日期的格式化
今天我做的项目中需要把时间给拆分了,形式为:yyyy-MM-DD HH mm, 下面是我的代码实现: <?xml version="1.0" encoding="u ...
- WebService从服务端到客户端的用例
1.首先编写Wsdl(基于契约优先的方式),要注意的是命名空间(若是使用include或import)最好使用一致的,代码如下: <?xml version="1.0" en ...
- flask mega-tutorial 1.0 documentation学习记录
本文主要是记录在[用户登录]一节中出现的问题: 报错位置是在 if g.user is not None and g.user.is_authenticated(): return redirect( ...
- SVN使用小记
SVN(Subversion)是优秀的版本控制工具,之前在eclipse里面项目管理的时候,File-->Import-->SVN-->从SVN检出项目-->创建新的资源库位置 ...
- Legendary Items-微软实习生笔试第一题
题目如下: 这道题难点不仅在于正确理解题意,判断递归条件,更在于用数学方法推出解决公式.因为N最大为1百万,而内存只有256MB, 所以暴力递归肯定会超时,超空间. 不过,我才疏学浅,又没有大量时间去 ...
- 重温Javascript(二)
对象 可以想象成散列表,键值对,值可以是数据或函数 创建对象的方式 1.工厂模式 function createPerson(name, age, job){ var o = new Object() ...