Linux IO/NFS tunning 性能优化及检测

Linux IO/NFS tunning:

IO Test
=======
dd 测试读性能的命令
# time dd if=/nfsfolder/testfile of=/dev/null bs=1k
dd 测试写性能的命令
time -p dd if=/dev/zero of=test.file bs=1k count=100000

[1]Write
 time -p dd if=/dev/zero of=test.file bs=1k count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 25.2433 s, 4.1 MB/s
real 25.24
user 0.03
sys 1.49

time -p dd if=/dev/zero of=test.file bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.477919 s, 219 MB/s
real 0.48
user 0.00
sys 0.11

[2] read file
time dd if=./test.file of=/dev/null bs=1k
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 14.1445 s, 7.2 MB/s

real    0m14.150s
user    0m0.054s
sys    0m1.929s

time dd if=./test.file of=/dev/null bs=1024k
97+1 records in
97+1 records out
102400000 bytes (102 MB) copied, 0.0465829 s, 2.2 GB/s

real    0m0.052s
user    0m0.000s
sys    0m0.037s

time rm -rf test.file

real    0m0.093s
user    0m0.000s
sys    0m0.091s

time rm -rf test.file

real    0m0.101s
user    0m0.000s
sys    0m0.098s

Tools:
======

sar -b 1 10

-b     Report I/O and transfer rate statistics.  The following values are displayed:

              tps
                     Total  number of transfers per second that were issued to physical devices.  A
                     transfer is an I/O request to a physical device. Multiple logical requests can
                     be  combined  into a single I/O request to the device.  A transfer is of inde\u2010
                     terminate size.

              rtps
                     Total number of read requests per second issued to physical devices.

              wtps
                     Total number of write requests per second issued to physical devices.

              bread/s
                     Total amount of data read from the devices in blocks per second.   Blocks  are
                     equivalent to sectors and therefore have a size of 512 bytes.

              bwrtn/s
                     Total amount of data written to devices in blocks per second.

注：每1秒 显示 1次 显示 10次

sar -d -p 3 10  //3s 10 times.

Monitor NFS
1 second interval to monitor nfsstat
nfsstat -cn -Z1

Client nfs v3:
null         getattr      setattr      lookup       access       readlink     
0         0% 777      58% 0         0% 0         0% 111       8% 0         0%
read         write        create       mkdir        symlink      mknod        
0         0% 432      32% 0         0% 0         0% 0         0% 0         0%
remove       rmdir        rename       link         readdir      readdirplus  
0         0% 0         0% 0         0% 0         0% 0         0% 0         0%
fsstat       fsinfo       pathconf     commit       
0         0% 0         0% 0         0% 0         0%

nfsstat -rc
Client rpc stats:
calls      retrans    authrefrsh
66906770   5          66907434

If retrans value is quite high, meaning that retransmissions were often necessary since the last reboot. This is a clear indication that the number of available NFS kernel threads on the server is insufficient to handle the requests from this client. The default number of threads for rpc.nfsd to start is typically eight threads.

To tell rpc.nfsd to use more kernel threads, the number of threads must be passed as an argument to it. Typically, most distributions will have a file such as /etc/sysconfig/nfs to configure this;

NFS server side:

nfsstat --all

dstat用来监控cpu,disk,network等, 类似vmstat.如下,比较直观.
[root@LMG3ADFBE2  ~] # dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  1   1  98   0   0   0| 244B   66k|   0     0 |   0     0 |2139  2048
  0   3  97   0   0   0|   0     0 | 195B  838B|   0     0 |2865  8355
  0   1  99   0   0   0|   0     0 |1038B  628B|   0     0 | 723   537
  0   2  98   0   0   0|   0     0 |  66B  358B|   0     0 |1884  1000
dstat常用的选项有：
        -c     显示cpu使用情况
        -d     显示磁盘使用情况
       -g,     显示页面数据  
        -i      启用中断数据
        -l      平均负载统计（1分钟，5分钟，15分钟）
        -m     显示内存使用信息
        -n     显示网络信息
        -p      enable process stats (runnable,uninterruptible, new)
       -r       enable I/O request stats(read, write requests)
       -s       显示交换分区使用信息
       -t       启用时间/日期输出
       -y       系统统计(中断上下文切换)
       --aio    显示异步io统计数据
       --ipc     ipc相关的信息
       --raw:    raw socket
       --tcp     tcp socket
       --udp    udp socket
       --socket  raw, tcp, udp
       --unix    unix sock
       --top-cpu 显示最占用CPU的进程
       --top-bio 显示最占用block IO的进程
       --top-mem: 显示最耗费内存的进程
       --top-io   最占用IO的进程

http://www.softpanorama.org/Net/Application_layer/NFS/nfs_performance_tuning.shtml

关于NFS ac/noac 和nocto

cto / nocto    Selects  whether  to  use close-to-open cache coherence semantics.
                      If neither option is specified  (or  if  cto  is  specified),  the
                      client  uses close-to-open cache coherence semantics. If the nocto
                      option is specified, the client uses a non-standard  heuristic  to
                      determine when files on the server have changed.

                      Using  the  nocto  option  may  improve  performance for read-only
                      mounts, but should be used only if the data on the server  changes
                      only  occasionally.   The DATA AND METADATA COHERENCE section dis-
                      cusses the behavior of this option in more detail.

设置这个参数如nocto后, 行为表现为.

Server A and Server B mount the same storage via NFS.

Server B nocto, ac, 当A写入一行到一个文件， 在B上看不到A写的内容。知道触发ac的timeout， 即actimo

ac/noac

ac / noac      Selects  whether  the client may cache file attributes. If neither
                      option is specified (or if ac is  specified),  the  client  caches
                      file attributes.

                      To  improve  performance, NFS clients cache file attributes. Every
                      few seconds, an NFS client checks the  server\u2019s  version  of  each
                      file\u2019s  attributes  for updates.  Changes that occur on the server
                      in those small intervals remain undetected until the client checks
                      the  server  again.  The noac option prevents clients from caching
                      file attributes so that applications can more quickly detect  file
                      changes on the server.

                      In addition to preventing the client from caching file attributes,
                      the noac option forces application writes to become synchronous so
                      that  local changes to a file become visible on the server immedi-
                      ately.  That way, other clients can quickly detect  recent  writes
                      when they check the file\u2019s attributes.

                      Using  the  noac option provides greater cache coherence among NFS
                      clients accessing the same files, but it  extracts  a  significant
                      performance  penalty.   As  such, judicious use of file locking is
                      encouraged instead.  The DATA AND METADATA COHERENCE section  con-
                      tains a detailed discussion of these trade-offs.

如果ac enable， 则文件属性及目录属性会在本地cache中（metadata cache）， 可以touch一个新文件来强制刷新这些cache。

如果server A with ac， server B with noac, 在A上向文件写入一行， 则B上可以看到最新的内容。当然如果B上设置了nocto，则看不到。

- Shall noac option on server B trigger the sync operation on server A? Yes. When the client mounted with noac option reads a file from the server, It will always fetch the most recent copy from the NFS server. If the NFS server thinks that a lock has been acquired for the same file by another client (clients acquire locks before performing write), then the NFS server asks that client to forcefully write the data, then the updated copy is provided to the client mounted with noac option.