Linux 运行进程实时监控pidstat命令详解

简介

pidstat主要用于监控全部或指定进程占用系统资源的情况，如CPU，内存、设备IO、任务切换、线程等。pidstat首次运行时显示自系统启动开始的各项统计信息，之后运行pidstat将显示自上次运行该命令以后的统计信息。用户可以通过指定统计的次数和时间来获得所需的统计信息。

实例讲解

默认参数

执行pidstat，将输出系统启动后所有活动进程的cpu统计信息：

 linux:~ # pidstat

Linux 2.6.32.12-0.7-default (linux)             06/18/12        _x86_64_

11:37:19          PID    %usr %system  %guest    %CPU   CPU  Command
……
11:37:19        11452    0.00    0.00    0.00    0.00     2  bash
11:37:19        11509    0.00    0.00    0.00    0.00     3  dd

 指定采样周期和采样次数

pidstat命令指定采样周期和采样次数，命令形式为”pidstat [option] interval [count]”，以下pidstat输出以2秒为采样周期，输出10次cpu使用统计信息：

pidstat 2 10

cpu使用情况统计(-u)

使用-u选项，pidstat将显示各活动进程的cpu使用统计，执行”pidstat -u”与单独执行”pidstat”的效果一样。

内存使用情况统计(-r)

使用-r选项，pidstat将显示各活动进程的内存使用统计：

 linux:~ # pidstat -r -p 13084 1

Linux 2.6.32.12-0.7-default (linux)             06/18/12        _x86_64_

15:08:18          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
15:08:19        13084 133835.00      0.00 15720284 15716896  96.26  mmmm
15:08:20        13084  35807.00      0.00 15863504 15849756  97.07  mmmm
15:08:21        13084  19273.87      0.00 15949040 15792944  96.72  mmmm

 以上各列输出的含义如下：

minflt/s: 每秒次缺页错误次数(minor page faults)，次缺页错误次数意即虚拟内存地址映射成物理内存地址产生的page fault次数
majflt/s: 每秒主缺页错误次数(major page faults)，当虚拟内存地址映射成物理内存地址时，相应的page在swap中，这样的page fault为major page fault，一般在内存使用紧张时产生
VSZ:      该进程使用的虚拟内存(以kB为单位)
RSS:      该进程使用的物理内存(以kB为单位)
%MEM:     该进程使用内存的百分比
Command:  拉起进程对应的命令

IO情况统计(-d)

使用-d选项，我们可以查看进程IO的统计信息：

 linux:~ # pidstat -d 1 2

Linux 2.6.32.12-0.7-default (linux)             06/18/12        _x86_64_

17:11:36          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
17:11:37        14579 124988.24      0.00      0.00  dd

17:11:37          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
17:11:38        14579 105441.58      0.00      0.00  dd

 输出信息含义

kB_rd/s: 每秒进程从磁盘读取的数据量(以kB为单位)
kB_wr/s: 每秒进程向磁盘写的数据量(以kB为单位)
Command: 拉起进程对应的命令

针对特定进程统计(-p)

使用-p选项，我们可以查看特定进程的系统资源使用情况：

 linux:~ # pidstat -r -p 1 1

Linux 2.6.32.12-0.7-default (linux)             06/18/12        _x86_64_

18:26:17          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
18:26:18            1      0.00      0.00   10380    640   0.00  init
18:26:19            1      0.00      0.00   10380    640   0.00  init
……

pidstat常用命令

使用pidstat进行问题定位时，以下命令常被用到：

pidstat -u 1

pidstat -r 1

pidstat -d 1

以上命令以1秒为信息采集周期，分别获取cpu、内存和磁盘IO的统计信息。

pidstat实际上也是将/proc/pid下的统计信息统筹后展现给用户。

pidstat例子

1. 列出IO统计信息：-d

2. 列出内存使用统计，page fault：-r

3. CPU统计信息：-u

4. 上下文切换统计信息：-w

5. 指定输出的维度

指定命令： -C command

指定进程号：-p { pid [,...] | SELF | ALL }

将所有列输出到单行、便于导入单张表中：-h

按每个CPU核的统计：-I

按所有CPU核统计：默认

列出命令的完整内容，包括参数：-l

列出线程统计信息：-t

按进程维度单独统计、按进程全局+子进程单独统计、按进程全局、单独统计同时按子任务单独统计：-T { TASK | CHILD | ALL }

实例，统计5秒的值并输出

#pidstat -d -r -u -w -l -h -p ALL 5 1|less  

#      Time       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s   cswch/s nvcswch/s  Command
 1488541252         1    0.00    0.00    0.00    0.00     0      0.00      0.00   19348   1512   0.00      0.00      0.00      0.00      0.00      0.00  /sbin/init
 1488541252         2    0.00    0.00    0.00    0.00    50      0.00      0.00       0      0   0.00      0.00      0.00      0.00      0.00      0.00  kthreadd
 1488541252         3    0.00    0.00    0.00    0.00     0      0.00      0.00       0      0   0.00      0.00      0.00      0.00      0.59      0.00  migration/0
 1488541252         4    0.00    0.00    0.00    0.00     0      0.00      0.00       0      0   0.00      0.00      0.00      0.00      0.00      0.00  ksoftirqd/0
 1488541252         5    0.00    0.00    0.00    0.00     0      0.00      0.00       0      0   0.00      0.00      0.00      0.00      0.00      0.00  migration/0
 1488541252         6    0.00    0.00    0.00    0.00     0      0.00      0.00       0      0   0.00      0.00      0.00      0.00      0.00      0.00  watchdog/0  

 ......  

 1488541264     18523    0.00    0.20    0.00    0.20     5     10.78      0.00 137774940 3002232   0.57      0.00      0.00      0.00      1.96      0.20  /home/digoal/pgsql9.6/bin/postgres
 1488541264     18549    0.00    0.00    0.00    0.00    61      0.00      0.00  140892   2028   0.00      0.00      0.00      0.00      0.00      0.00  postgres: logger process
 1488541264     18552    0.00    0.00    0.00    0.00     0      0.00      0.00 137779116 5509520   1.04      0.00      0.00      0.00      0.00      0.00  postgres: checkpointer process
 1488541264     18554    0.00    0.00    0.00    0.00    20      0.00      0.00 137776328 1058572   0.20      0.00      0.00      0.00      3.92      0.00  postgres: writer process
 1488541264     18556    0.00    0.00    0.00    0.00    15      0.00      0.00 137774940  19120   0.00      0.00      0.00      0.00      3.92      0.00  postgres: wal writer process
 1488541264     18557    0.00    0.00    0.00    0.00    21      2.94      0.00 137779092   3404   0.00      0.00      0.00      0.00      3.92      0.00  postgres: autovacuum launcher process
 1488541264     18559    0.00    0.00    0.00    0.00    11      0.00      0.00  142988   2092   0.00      0.00      0.00      0.00      0.00      0.00  postgres: archiver process   last was 0000000100000030000000B2
 1488541264     18561    0.00    0.00    0.00    0.00    53      4.90      0.00  143556   2624   0.00      0.00    317.65    317.65      4.12      0.20  postgres: stats collector process

NAME
pidstat - Report statistics for Linux tasks.

SYNOPSIS
pidstat [ -d ] [ -h ] [ -I ] [ -l ] [ -r ] [ -s ] [ -t ] [ -U [ username ] ] [ -u ] [ -V ] [ -w ] [ -C comm ] [ -p { pid [,...] | SELF | ALL } ] [ -T { TASK | CHILD | ALL }
] [ interval [ count ] ]

DESCRIPTION
The pidstat command is used for monitoring individual tasks currently being managed by the Linux kernel. It writes to standard output activities for every task selected
with option -p or for every task managed by the Linux kernel if option -p ALL has been used. Not selecting any tasks is equivalent to specifying -p ALL but only active
tasks (tasks with non-zero statistics values) will appear in the report.

The pidstat command can also be used for monitoring the child processes of selected tasks. Read about option -T below.

The interval parameter specifies the amount of time in seconds between each report. A value of 0 (or no parameters at all) indicates that tasks statistics are to be
reported for the time since system startup (boot). The count parameter can be specified in conjunction with the interval parameter if this one is not set to zero. The
value of count determines the number of reports generated at interval seconds apart. If the interval parameter is specified without the count parameter, the pidstat command
generates reports continuously.

You can select information about specific task activities using flags. Not specifying any flags selects only CPU activity.

OPTIONS
-C comm
Display only tasks whose command name includes the string comm. This string can be a regular expression.

-d Report I/O statistics (kernels 2.6.20 and later only). The following values may be displayed:

UID
The real user identification number of the task being monitored.

USER
The name of the real user owning the task being monitored.

PID
The identification number of the task being monitored.

kB_rd/s
Number of kilobytes the task has caused to be read from disk per second.

kB_wr/s
Number of kilobytes the task has caused, or shall cause to be written to disk per second.

kB_ccwr/s
Number of kilobytes whose writing to disk has been cancelled by the task. This may occur when the task truncates some dirty pagecache. In this case, some IO
which another task has been accounted for will not be happening.

Command
The command name of the task.

-h Display all activities horizontally on a single line. This is intended to make it easier to be parsed by other programs.

-I In an SMP environment, indicate that tasks CPU usage (as displayed by option -u ) should be divided by the total number of processors.

-l Display the process command name and all its arguments.

-p { pid [,...] | SELF | ALL }
Select tasks (processes) for which statistics are to be reported. pid is the process identification number. The SELF keyword indicates that statistics are to be
reported for the pidstat process itself, whereas the ALL keyword indicates that statistics are to be reported for all the tasks managed by the system.

-r Report page faults and memory utilization.

When reporting statistics for individual tasks, the following values may be displayed:

UID
The real user identification number of the task being monitored.

USER
The name of the real user owning the task being monitored.

PID
The identification number of the task being monitored.

minflt/s
Total number of minor faults the task has made per second, those which have not required loading a memory page from disk.

majflt/s
Total number of major faults the task has made per second, those which have required loading a memory page from disk.

VSZ
Virtual Size: The virtual memory usage of entire task in kilobytes.

RSS
Resident Set Size: The non-swapped physical memory used by the task in kilobytes.

%MEM
The tasks's currently used share of available physical memory.

Command
The command name of the task.

When reporting global statistics for tasks and all their children, the following values may be displayed:

UID
The real user identification number of the task which is being monitored together with its children.

USER
The name of the real user owning the task which is being monitored together with its children.

PID
The identification number of the task which is being monitored together with its children.

minflt-nr
Total number of minor faults made by the task and all its children, and collected during the interval of time.

majflt-nr
Total number of major faults made by the task and all its children, and collected during the interval of time.

Command
The command name of the task which is being monitored together with its children.

-s Report stack utilization. The following values may be displayed:

UID
The real user identification number of the task being monitored.

USER
The name of the real user owning the task being monitored.

PID
The identification number of the task being monitored.

StkSize
The amount of memory in kilobytes reserved for the task as stack, but not necessarily used.

StkRef
The amount of memory in kilobytes used as stack, referenced by the task.

Command
The command name of the task.

-T { TASK | CHILD | ALL }
This option specifies what has to be monitored by the pidstat command. The TASK keyword indicates that statistics are to be reported for individual tasks (this is
the default option) whereas the CHILD keyword indicates that statistics are to be globally reported for the selected tasks and all their children. The ALL keyword
indicates that statistics are to be reported for individual tasks and globally for the selected tasks and their children.

Note: Global statistics for tasks and all their children are not available for all options of pidstat. Also these statistics are not necessarily relevant to current
time interval: The statistics of a child process are collected only when it finishes or it is killed.

-t Also display statistics for threads associated with selected tasks.

This option adds the following values to the reports:

TGID
The identification number of the thread group leader.

TID
The identification number of the thread being monitored.

-U [ username ]
Display the real user name of the tasks being monitored instead of the UID. If username is specified, then only tasks belonging to the specified user are displayed.

-u Report CPU utilization.

When reporting statistics for individual tasks, the following values may be displayed:

UID
The real user identification number of the task being monitored.

USER
The name of the real user owning the task being monitored.

PID
The identification number of the task being monitored.

%usr
Percentage of CPU used by the task while executing at the user level (application), with or without nice priority. Note that this field does NOT include time
spent running a virtual processor.

%system
Percentage of CPU used by the task while executing at the system level (kernel).

%guest
Percentage of CPU spent by the task in virtual machine (running a virtual processor).

%CPU
Total percentage of CPU time used by the task. In an SMP environment, the task's CPU usage will be divided by the total number of CPU's if option -I has been
entered on the command line.

CPU
Processor number to which the task is attached.

Command
The command name of the task.

When reporting global statistics for tasks and all their children, the following values may be displayed:

UID
The real user identification number of the task which is being monitored together with its children.

USER
The name of the real user owning the task which is being monitored together with its children.

PID
The identification number of the task which is being monitored together with its children.

usr-ms
Total number of milliseconds spent by the task and all its children while executing at the user level (application), with or without nice priority, and col‐
lected during the interval of time. Note that this field does NOT include time spent running a virtual processor.

system-ms
Total number of milliseconds spent by the task and all its children while executing at the system level (kernel), and collected during the interval of time.

guest-ms
Total number of milliseconds spent by the task and all its children in virtual machine (running a virtual processor).

Command
The command name of the task which is being monitored together with its children.

-V Print version number then exit.

-w Report task switching activity (kernels 2.6.23 and later only). The following values may be displayed:

UID
The real user identification number of the task being monitored.

USER
The name of the real user owning the task being monitored.

PID
The identification number of the task being monitored.

cswch/s
Total number of voluntary context switches the task made per second. A voluntary context switch occurs when a task blocks because it requires a resource that
is unavailable.

nvcswch/s
Total number of non voluntary context switches the task made per second. A involuntary context switch takes place when a task executes for the duration of
its time slice and then is forced to relinquish the processor.

Command
The command name of the task.

ENVIRONMENT
The pidstat command takes into account the following environment variable:

S_TIME_FORMAT
If this variable exists and its value is ISO then the current locale will be ignored when printing the date in the report header. The pidstat command will use the
ISO 8601 format (YYYY-MM-DD) instead.

EXAMPLES
pidstat 2 5
Display five reports of CPU statistics for every active task in the system at two second intervals.

pidstat -r -p 1643 2 5
Display five reports of page faults and memory statistics for PID 1643 at two second intervals.

pidstat -C "fox|bird" -r -p ALL
Display global page faults and memory statistics for all the processes whose command name includes the string "fox" or "bird".

pidstat -T CHILD -r 2 5
Display five reports of page faults statistics at two second intervals for the child processes of all tasks in the system. Only child processes with non-zero statis‐
tics values are displayed.

BUGS
/proc filesystem must be mounted for the pidstat command to work.

FILES
/proc contains various files with system statistics.

AUTHOR
Sebastien Godard (sysstat <at> orange.fr)

SEE ALSO
sar(1), top(1), ps(1), mpstat(1), iostat(1), vmstat(8)

http://pagesperso-orange.fr/sebastien.godard/

Linux MARCH 2013 PIDSTAT(1)