(转)文件系统缓存dirty_ratio与dirty_background_ratio两个参数区别
这两天在调优数据库性能的过程中需要降低操作系统文件Cache对数据库性能的影响,故调研了一些降低文件系统缓存大小的方法,其中一种是通过修改/proc/sys/vm/dirty_background_ration以及/proc/sys/vm/dirty_ratio两个参数的大小来实现。看了不少相关博文的介绍,不过一直弄不清楚这两个参数的区别在哪里,后来看了下面的一篇英文博客才大致了解了它们的不同。
附上原文:
Better Linux Disk Caching & Performance with vm.dirty_ratio & vm.dirty_background_ratio
by BOB PLANKERS on DECEMBER 22, 2013
in BEST PRACTICES,CLOUD,SYSTEM ADMINISTRATION,VIRTUALIZATION
This is post #16 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.”
In previous posts on vm.swappiness and using RAM disks we talked about how the memory on a Linux guest is used for the OS itself (the kernel, buffers, etc.), applications, and also for file cache. File caching is an important performance improvement, and read caching is a clear win in most cases, balanced against applications using the RAM directly. Write caching is trickier. The Linux kernel stages disk writes into cache, and over time asynchronously flushes them to disk. This has a nice effect of speeding disk I/O but it is risky. When data isn’t written to disk there is an increased chance of losing it.
There is also the chance that a lot of I/O will overwhelm the cache, too. Ever written a lot of data to disk all at once, and seen large pauses on the system while it tries to deal with all that data? Those pauses are a result of the cache deciding that there’s too much data to be written asynchronously (as a non-blocking background operation, letting the application process continue), and switches to writing synchronously (blocking and making the process wait until the I/O is committed to disk). Of course, a filesystem also has to preserve write order, so when it starts writing synchronously it first has to destage the cache. Hence the long pause.
The nice thing is that these are controllable options, and based on your workloads & data you can decide how you want to set them up. Let’s take a look:
$ sysctl -a | grep dirty vm.dirty_background_ratio = 10 vm.dirty_background_bytes = 0 vm.dirty_ratio = 20 vm.dirty_bytes = 0 vm.dirty_writeback_centisecs = 500 vm.dirty_expire_centisecs = 3000
vm.dirty_background_ratio is the percentage of system memory that can be filled with “dirty” pages — memory pages that still need to be written to disk — before the pdflush/flush/kdmflush background processes kick in to write it to disk. My example is 10%, so if my virtual server has 32 GB of memory that’s 3.2 GB of data that can be sitting in RAM before something is done.
vm.dirty_ratio is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk. When the system gets to this point all new I/O blocks until dirty pages have been written to disk. This is often the source of long I/O pauses, but is a safeguard against too much data being cached unsafely in memory.
vm.dirty_background_bytes and vm.dirty_bytes are another way to specify these parameters. If you set the _bytes version the _ratio version will become 0, and vice-versa.
vm.dirty_expire_centisecs is how long something can be in cache before it needs to be written. In this case it’s 30 seconds. When the pdflush/flush/kdmflush processes kick in they will check to see how old a dirty page is, and if it’s older than this value it’ll be written asynchronously to disk. Since holding a dirty page in memory is unsafe this is also a safeguard against data loss.
vm.dirty_writeback_centisecs is how often the pdflush/flush/kdmflush processes wake up and check to see if work needs to be done.
You can also see statistics on the page cache in /proc/vmstat:
$ cat /proc/vmstat | egrep "dirty|writeback" nr_dirty 878 nr_writeback 0 nr_writeback_temp 0
In my case I have 878 dirty pages waiting to be written to disk.
Approach 1: Decreasing the Cache
As with most things in the computer world, how you adjust these depends on what you’re trying to do. In many cases we have fast disk subsystems with their own big, battery-backed NVRAM caches, so keeping things in the OS page cache is risky. Let’s try to send I/O to the array in a more timely fashion and reduce the chance our local OS will, to borrow a phrase from the service industry, be “in the weeds.” To do this we lower vm.dirty_background_ratio and vm.dirty_ratio by adding new numbers to /etc/sysctl.conf and reloading with “sysctl –p”:
vm.dirty_background_ratio = 5 vm.dirty_ratio = 10
This is a typical approach on virtual machines, as well as Linux-based hypervisors. I wouldn’t suggest setting these parameters to zero, as some background I/O is nice to decouple application performance from short periods of higher latency on your disk array & SAN (“spikes”).
Approach 2: Increasing the Cache
There are scenarios where raising the cache dramatically has positive effects on performance. These situations are where the data contained on a Linux guest isn’t critical and can be lost, and usually where an application is writing to the same files repeatedly or in repeatable bursts. In theory, by allowing more dirty pages to exist in memory you’ll rewrite the same blocks over and over in cache, and just need to do one write every so often to the actual disk. To do this we raise the parameters:
vm.dirty_background_ratio = 50 vm.dirty_ratio = 80
Sometimes folks also increase the vm.dirty_expire_centisecs parameter to allow more time in cache. Beyond the increased risk of data loss, you also run the risk of long I/O pauses if that cache gets full and needs to destage, because on large VMs there will be a lot of data in cache.
Approach 3: Both Ways
There are also scenarios where a system has to deal with infrequent, bursty traffic to slow disk (batch jobs at the top of the hour, midnight, writing to an SD card on a Raspberry Pi, etc.). In that case an approach might be to allow all that write I/O to be deposited in the cache so that the background flush operations can deal with it asynchronously over time:
vm.dirty_background_ratio = 5 vm.dirty_ratio = 80
Here the background processes will start writing right away when it hits that 5% ceiling but the system won’t force synchronous I/O until it gets to 80% full. From there you just size your system RAM and vm.dirty_ratio to be able to consume all the written data. Again, there are tradeoffs with data consistency on disk, which translates into risk to data. Buy a UPS and make sure you can destage cache before the UPS runs out of power. :)
No matter the route you choose you should always be gathering hard data to support your changes and help you determine if you are improving things or making them worse. In this case you can get data from many different places, including the application itself, /proc/vmstat, /proc/meminfo, iostat, vmstat, and many of the things in /proc/sys/vm. Good luck!
(转)文件系统缓存dirty_ratio与dirty_background_ratio两个参数区别的更多相关文章
- Linux 文件系统缓存dirty_ratio与dirty_background_ratio两个参数区别
文件系统缓存dirty_ratio与dirty_background_ratio两个参数区别 (2014-03-16 17:54:32) 转载▼ 标签: linux 文件系统缓存 cache dirt ...
- 文件系统缓存dirty_ratio与dirty_background_ratio两个参数区别
这两天在调优数据库性能的过程中需要降低操作系统文件Cache对数据库性能的影响,故调研了一些降低文件系统缓存大小的方法,其中一种是通过修改/proc/sys/vm/dirty_background_r ...
- [转载]文件系统缓存dirty_ratio与dirty_background_ra
原文地址:文件系统缓存dirty_ratio与dirty_background_ratio两个参数区别作者:vincent 这两天在调优数据库性能的过程中需要降低操作系统文件Cache对数据库性能的影 ...
- [转载]C#缓存absoluteExpiration、slidingExpiration两个参数的疑惑
看了很多资料终于搞明白cache中absoluteExpiration,slidingExpiration这两个参数的含义. absoluteExpiration:用于设置绝对过期时间,它表示只要时间 ...
- C#缓存absoluteExpiration、slidingExpiration两个参数的疑惑
看了很多资料终于搞明白cache中absoluteExpiration,slidingExpiration这两个参数的含义. absoluteExpiration:用于设置绝对过期时间,它表示只要时间 ...
- maven跳过单元测试的两个参数区别
maven在打包过程中需要执行单元测试.但有些时候单元测试已经通过只是想打包时,想跳过测试.maven提供了两个参数跳过测试:maven.test.skip=true 和skipTests. 例子 m ...
- Better Linux Disk Caching & Performance with vm.dirty_ratio & vm.dirty_background_ratio
In previous posts on vm.swappiness and using RAM disks we talked about how the memory on a Linux gue ...
- vm.dirty_ratio & vm.dirty_background_ratio
https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ Better Lin ...
- 性能优化(一个)Hibernate 使用缓存(一个、两、查询)提高系统性能
在hibernate有三种类型的高速缓存,我们使用最频繁.分别缓存.缓存和查询缓存.下面我们使用这三个缓存中的项目和分析的优点和缺点. 缓存它的作用在于提高性能系统性能,介于应用系统与数据库之间而存在 ...
随机推荐
- [CSS3 Animation] TweenMax.staggerTo()
<!DOCTYPE html> <html> <head lang="en"> <meta charset="UTF-8&quo ...
- boost.asio源码剖析(二) ---- 架构浅析
* 架构浅析 先来看一下asio的0层的组件图. (图1.0) io_object是I/O对象的集合,其中包含大家所熟悉的socket.deadline_tim ...
- google对js延迟加载方案的建议
浏览器在执行JavaScript代码时会停止处理页面,当页面中有很多JavaScript文件或代码要加载时,将导致严重的延迟.尽管可以使用defer.异步或将JavaScript代码放到页面底部来延迟 ...
- APP快速通过苹果AppStore审核九大诀窍
[IT168技术]对于移动应用开发者来说, 最令人沮丧的可能莫过于辛辛苦苦开发的应用, 没能通过苹果AppStore的审核,或者在应用更新时遭遇下架.苹果的AppStore的审核流程和标准, 一向不透 ...
- Linux进程管理之task_struct结构
转载:http://blog.csdn.net/npy_lp/article/details/7335187 内核源码:linux-2.6.38.8.tar.bz2 目标平台:ARM体系结构 进程是处 ...
- sqoop1.99.4安装与简介
1.什么是sqoop clouder公司开发的一个在关系数据库和hdfs,hive之间数据导入导出的一个工具 2.版本及其区别 现在分为sqoop1和sqoop2 1)架构图(sqoop1和hadoo ...
- CSS 之 嵌套 margin-top 处理
如下代码: <div style=" width:1000px; height:700px; margin:auto;"> <div style=" w ...
- Java中transient有何作用?
transient关键字用来防止序列化域.如果一个引用类型被transient修饰,则其反序列化的结果是null.基本类型则为0.如果引用类型时不可序列化的类,则也应该使用transient修饰,它在 ...
- JQuery总结:选择器归纳、DOM遍历和事件处理、DOM完全操作和动画 (转)
JQuery总结:选择器归纳.DOM遍历和事件处理.DOM完全操作和动画 转至元数据结尾 我们后台可能用到的页面一般都是用jquery取值赋值的,发现一片不错的文章 目录 JQuery总结一:选择器归 ...
- 【Linux】gdb调试core文件
编写服务器端程序,很容易遇到Crash问题,比较幸运的是Linux提供了core file,保留了Crash的现场.有时候,根据当前的调用栈,并且打印出当前栈的变量就可以分析出crash的原因,但是, ...