GlusterFS PERFORMANCE TUNING

众所周知，glusterfs对小文件而言，就是个鸡肋，特别是在一个目录下有过W的小文件图片时，ls简单就是个坑，下面我对线上的glusterfs参数做一些优化调整，调整的命令：

gluster volume set VOLNAME OPTION PARAMETER

参数如下：

cluster.data-self-heal-algorithm      ( full/diff )
cluster.min-free-disk                 ( Required minimum free disk space as a percentage like 30%)
cluster.self-heal-window-size         ( 0 < data-self-heal-window-size < 1025)
cluster.stripe-block-size             (size in bytes )
cluster.quorum-type                   (quorum method)
cluster.quorum-count                  (# needed for quorum)
diagnostics.brick-log-level           ( DEBUG|INFO|WARNING|ERROR|CRITICAL|NONE|TRACE )
diagnostics.client-log-level          (DEBUG|INFO|WARNING|ERROR|CRITICAL|NONE|TRACE )
diagnostics.latency-measurement       ( ON|OFF )
diagnostics.dump-fd-stats             ( ON|OFF )
features.quota-timeout          ( 0 < 3600 secs )
geo-replication.indexing          ( ON|OFF )
network.frame-timeout          ( 1800 secs)
network.ping-timeout          ( 42 secs)
performance.cache-max-file-size       ( size in bytes )
performance.cache-min-file-size       ( size in bytes )
performance.cache-refresh-timeout     ( 0 < cache-timeout < 61 )
performance.cache-size       ( size in bytes )
performance.flush-behind              ( ON|OFF)
performance.io-thread-count   ( 0 < io-threads < 65 )
performance.write-behind-window-size  ( Write-behind cache size )
auth.allow                            ( valid IP address which includes wild card patterns including *, such as 192.168.1.* )
auth.reject
nfs.disable                  (ON|OFF)
nfs.enable-ino32                      (ON|OFF)
nfs.export-dir              (Enable|Disable)
nfs.export-volumes       (ON|OFF)
nfs.addr-namelookup     (ON|OFF)
nfs.register-with-portmap   (ON|OFF)
nfs.port  (38465 to 38467)
nfs.rpc-auth-unix           (ON|OFF)
nfs.rpc-auth-null          (ON|OFF)
nfs.rpc-auth-allow        (IP address or Host name)
nfs.rpc-auth-reject      (IP address or Host name)
nfs.ports-insecure              (ON|OFF)
nfs.trusted-sync        (ON|OFF)
nfs.trusted-write      (ON|OFF)
nfs.volume-access      (read-write|read-only)
rpc-auth-allow-insecure       (ON|OFF)

各个参数说明：

Note: The default options given here are hard coded in the source file and is subject to modification at any given time. The following default values may not be the same for all versions.

调整案例：

# gluster volume set v3_upload performance.cache-size 4GB
volume set: success
# gluster volume set v3_upload auth.allow 10.16.1.93
volume set: success
# gluster volume set v3_upload performance.io-thread-count 32
volume set: success

# gluster volume info

Volume Name: v3_upload
Type: Striped-Replicate
Volume ID: 401b5343-df8f-4c5d-a1c2-0363fa9d4591
Status: Started
Number of Bricks: 1 x 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.16.1.103:/data/dsrv1/v1
Brick2: 10.16.1.97:/data/dsrv1/v2
Brick3: 10.16.1.103:/data/dsrv2/v3
Brick4: 10.16.1.97:/data/dsrv2/v4
Options Reconfigured:
auth.allow: 10.16.1.93
performance.io-thread-count: 32
performance.cache-size: 4GB

对一个4W多文件的目录进行ls查看：

调整前：

real    1m16.063s
user    0m1.077s
sys     0m1.825s

调整后，当然第一次读取肯定是慢的，缓存之后，执行时间上还是有所提升的：

one:
real    0m48.976s
user    0m1.137s
sys     0m1.525s
two:
real    0m55.031s
user    0m1.212s
sys     0m1.794s

原文参考http://ju.outofmemory.cn/entry/109435

还可以参考https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/