iowait过高处理

网管告警：

告警主机：YiDHLWJKFZ-js-app-
主机IP：192.168.***.***
告警项目：system.cpu.util[,iowait]
告警时间：2019.02. ::
告警等级：Warning
告警信息：Disk I/O is overloaded on YiDHLWJKFZ-js-app-
问题详情：CPU iowait time:20.14 %
当前状态：PROBLEM:20.14 %
事件ID：

top查看：（wa值为17.7%）

[root@localhost vmuser]# top

top - :: up  :,   user,  load average: 3.56, 3.45, 3.40
Tasks:  total,    running,  sleeping,    stopped,    zombie
Cpu(s):  1.0%us,  2.6%sy,  0.0%ni, 78.2%id, 17.7%wa,  0.2%hi,  0.3%si,  0.0%st
Mem:  16467984k total,  2823808k used, 13644176k free,   331524k buffers
Swap: 16383992k total,        0k used, 16383992k free,   565436k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
   root                       D  5.3  0.0   :16.37 jbd2/sda3-
  postfix        104m  46m  S  3.7  0.3   :45.87 qmgr
  root           242m    S  3.0  0.0   :17.92 rsyslogd
 postfix          S  1.7  0.0   :00.05 local
  root             S  1.3  0.0   :44.87 master
 postfix          S  1.3  0.0   :00.05 cleanup
 root             D  1.3  0.0   :00.04 local
  postfix          S  1.0  0.0   :29.32 trivial-rewrite
 postfix          S  1.0  0.0   :00.06 cleanup
  postfix          D  0.7  0.0   :39.73 pickup
 postfix          D  0.7  0.0   :00.03 cleanup     

iostat查看：（iowait值一直过高）

[root@localhost vmuser]# iostat  -x
Linux 2.6.-.el6.x86_64 (localhost.localdomain)      //      _x86_64_        ( CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.96    0.00    2.14   19.78    0.00   75.12

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda              12.58   361.59   28.73 1621.40   616.25 15618.31     9.84     3.72    2.25   0.53  86.86
sdb             307.71     0.01    6.02    0.00  1255.11     0.05   208.45     0.14   22.52  14.07   8.47

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.84    0.00    2.12   21.38    0.00   75.66

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   465.00   36.00 1641.00   288.00 16508.00    10.02     3.27    1.98   0.58  97.30
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.44    0.00    4.49   15.09    0.00   78.97

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00  1167.00   18.00 4016.50   144.00 40720.00    10.13     3.61    0.88   0.22  87.95
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.44    0.00    1.19   24.88    0.00   73.50

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   351.00   22.50  879.00   180.00  9696.00    10.96     3.25    3.68   1.10  98.90
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.87    0.00    2.31   20.07    0.00   76.75

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   435.50   19.00 2666.50   152.00 24512.00     9.18     3.23    1.21   0.35  93.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

[root@localhost vmuser]#

iotop查看：

[root@localhost vmuser]# iotop 

Total DISK READ: 124.32 K/s | Total DISK WRITE: 6.69 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
   be/ root        0.00 B/s  236.99 K/s  0.00 % 92.26 % [jbd2/sda3-]
  be/ postfix   124.32 K/s    0.00 B/s  0.00 % 88.09 % pickup -l -t fifo -u
  be/ postfix     0.00 B/s    0.00 B/s  0.00 %  6.21 % qmgr -l -t fifo -u
 be/ postfix     0.00 B/s   58.28 K/s  0.00 %  6.02 % cleanup -z -t unix -u
  be/ root        0.00 B/s   27.20 K/s  0.00 %  3.82 % rsyslogd -c
 be/ postfix     0.00 B/s  158.28 K/s  0.00 %  1.79 % cleanup -z -t unix -u
 be/ postfix     0.00 B/s  260.31 K/s  0.00 %  1.62 % local -t unix unix -u
 be/ postfix     0.00 B/s   19.43 K/s  0.00 %  0.75 % bounce -z -t unix -u
 be/ postfix     0.00 B/s   58.28 K/s  0.00 %  0.75 % cleanup -z -t unix -uth :/usr/local/sa/sa-agent/main/~ocal/sa/sa-agent com.transfar.sa.agent.main.Bootstrap
    rt/ root        0.00 B/s   54.39 B/s  0.00 %  0.67 % bounce -z -t unix -u
 rt/ postfix     0.00 B/s   38.85 B/s  0.00 %  0.60 % bounce -z -t unix -u
    be/ root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/]
    rt/ root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/]
    rt/ root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/]
    rt/ root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/]
    rt/ root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/]]

关闭不必要的postfix进程

[root@localhost vmuser]# service postfix stop
Shutting down postfix: [  OK  ]
[root@localhost vmuser]#