iozone的三大神器之一fileop

iozone用了很久，查看源码，发现iozone其实还附带两个工具fileop和pit_server，fileop测试了POSIX常用的函数，pit_server用来测试TCP或UDP服务

今天用了一下fileop测试ceph的读写瓶颈，一下就找到问题了。首先看一下帮助

[root@node2 iozone]# ./fileop -h

     --------------------------------------
     |              Fileop                |
     |         $Revision: 1.61 $          |
     |                                    |
     |                by                  |
     |                                    |
     |             Don Capps              |
     --------------------------------------

     fileop [-f X ]|[-l # -u #] [-s Y] [-e] [-b] [-w] [-d <dir>] [-t] [-v] [-h]

     -f #      Force factor. X^ files will be created and removed.
     -l #      Lower limit on the value of the Force factor.
     -u #      Upper limit on the value of the Force factor.
     -s #      Optional. Sets filesize for the create/write. May use suffix 'K' or 'M'.
     -e        Excel importable format.
     -b        Output best case results.
     -i #      Increment force factor by this increment.
     -w        Output worst case results.
     -d <dir>  Specify starting directory.
     -U <dir>  Mount point to remount between tests.
     -t        Verbose output option.
     -v        Version information.
     -h        Help text.

     The structure of the file tree is:
     X number of Level  directories, with X number of
     level  directories, with X number of files in each
     of the level  directories.

     Example:  fileop 

             dir_1                        dir_2
            /     \                      /     \
      sdir_1       sdir_2          sdir_1       sdir_2
      /     \     /     \          /     \      /     \
   file_1 file_2 file_1 file_2   file_1 file_2 file_1 file_2

   Each file will be created, and then Y bytes is written to the file.

例如：1.我们测试一个文件读写最好最坏和平均的结果。(单位：次/s)

[root@node2 iozone]# ./fileop -f  -d ./ -e  -b -w
 .       mkdir   chdir   rmdir  create    open    read   write   close    stat  access   chmod readdir  link    unlink  delete  Total_files
A
B
W                                                                     

2.我们用verbose打印详细日志查看结果

[root@node2 iozone]# ./fileop -t -f  -d ./ 

Fileop:  Working in ., File size is ,  Output is in Ops/sec. (A=Avg, B=Best, W=Worst)
mkdir:   Dirs =          Total Time =  0.006633043 seconds
         Avg mkdir(s)/sec     =       452.28 ( 0.002211014 seconds/op)
         Best mkdir(s)/sec    =       529.38 ( 0.001888990 seconds/op)
         Worst mkdir(s)/sec   =       411.17 ( 0.002432108 seconds/op)

chdir:   Dirs =          Total Time =  0.000052691 seconds
         Avg chdir(s)/sec     =     56936.25 ( 0.000017564 seconds/op)
         Best chdir(s)/sec    =    113359.57 ( 0.000008821 seconds/op)
         Worst chdir(s)/sec   =     28532.68 ( 0.000035048 seconds/op)

rmdir:   Dirs =          Total Time =  0.011839867 seconds
         Avg rmdir(s)/sec     =       253.38 ( 0.003946622 seconds/op)
         Best rmdir(s)/sec    =       909.24 ( 0.001099825 seconds/op)
         Worst rmdir(s)/sec   =       165.32 ( 0.006048918 seconds/op)

create:  Files =          Total Time =  0.002378941 seconds
         Avg create(s)/sec    =       420.36 ( 0.002378941 seconds/op)
         Best create(s)/sec   =       420.36 ( 0.002378941 seconds/op)
         Worst create(s)/sec  =       420.36 ( 0.002378941 seconds/op)

write:   Files =          Total Time =  0.000038147 seconds
         Avg write(s)/sec     =     26214.40 ( 0.000038147 seconds/op)
         Best write(s)/sec    =     26214.40 ( 0.000038147 seconds/op)
         Worst write(s)/sec   =     26214.40 ( 0.000038147 seconds/op)

close:   Files =          Total Time =  0.000033855 seconds
         Avg close(s)/sec     =     29537.35 ( 0.000033855 seconds/op)
         Best close(s)/sec    =     29537.35 ( 0.000033855 seconds/op)
         Worst close(s)/sec   =     29537.35 ( 0.000033855 seconds/op)

stat:    Files =          Total Time =  0.000011206 seconds
         Avg stat(s)/sec      =     89240.51 ( 0.000011206 seconds/op)
         Best stat(s)/sec     =     89240.51 ( 0.000011206 seconds/op)
         Worst stat(s)/sec    =     89240.51 ( 0.000011206 seconds/op)

open:    Files =          Total Time =  0.000014067 seconds
         Avg open(s)/sec      =     71089.90 ( 0.000014067 seconds/op)
         Best open(s)/sec     =     71089.90 ( 0.000014067 seconds/op)
         Worst open(s)/sec    =     71089.90 ( 0.000014067 seconds/op)

read:    Files =          Total Time =  0.000015020 seconds
         Avg read(s)/sec      =     66576.25 ( 0.000015020 seconds/op)
         Best read(s)/sec     =     66576.25 ( 0.000015020 seconds/op)
         Worst read(s)/sec    =     66576.25 ( 0.000015020 seconds/op)

access:  Files =          Total Time =  0.000008821 seconds
         Avg access(s)/sec    =    113359.57 ( 0.000008821 seconds/op)
         Best access(s)/sec   =    113359.57 ( 0.000008821 seconds/op)
         Worst access(s)/sec  =    113359.57 ( 0.000008821 seconds/op)

chmod:   Files =          Total Time =  0.000021935 seconds
         Avg chmod(s)/sec     =     45590.26 ( 0.000021935 seconds/op)
         Best chmod(s)/sec    =     45590.26 ( 0.000021935 seconds/op)
         Worst chmod(s)/sec   =     45590.26 ( 0.000021935 seconds/op)

readdir: Files =          Total Time =  0.001602888 seconds
         Avg readdir(s)/sec   =       623.87 ( 0.001602888 seconds/op)
         Best readdir(s)/sec  =       623.87 ( 0.001602888 seconds/op)
         Worst readdir(s)/sec =       623.87 ( 0.001602888 seconds/op)

link:    Files =          Total Time =  0.000864983 seconds
         Avg link(s)/sec      =      1156.09 ( 0.000864983 seconds/op)
         Best link(s)/sec     =      1156.09 ( 0.000864983 seconds/op)
         Worst link(s)/sec    =      1156.09 ( 0.000864983 seconds/op)

unlink:  Files =          Total Time =  0.000801086 seconds
         Avg unlink(s)/sec    =      1248.30 ( 0.000801086 seconds/op)
         Best unlink(s)/sec   =      1248.30 ( 0.000801086 seconds/op)
         Worst unlink(s)/sec  =      1248.30 ( 0.000801086 seconds/op)

delete:  Files =          Total Time =  0.000800133 seconds
         Avg delete(s)/sec    =      1249.79 ( 0.000800133 seconds/op)
         Best delete(s)/sec   =      1249.79 ( 0.000800133 seconds/op)
         Worst delete(s)/sec  =      1249.79 ( 0.000800133 seconds/op)

从结果中，我们可以得出结论，ceph在FS挂载的情况下，其实性能瓶颈主要是在readdir:rmdir:mkdir:三个函数上面。