Prometheus Node_exporter 之 CPU Memory Net Disk

1. CPU

type: Graph
Unit: short
max: "100"
min: "0"
Label: Percentage
System - cpu 在内核模式下执行的进程占比

metrics:

sum by (mode)(irate(node_cpu_seconds_total{mode="system",instance=~"$node:$port",job=~"$job"}[5m])) * 100

User - cpu 在用户模式下执行的正常进程占比

metrics:

sum by (mode)(irate(node_cpu_seconds_total{mode='user',instance=~"$node:$port",job=~"$job"}[5m])) * 100

Nice - cpu 在用户模式下执行的 nice 进程占比

metrics:

sum by (mode)(irate(node_cpu_seconds_total{mode='nice',instance=~"$node:$port",job=~"$job"}[5m])) * 100

Idle - cpu 在空闲模式下的占比

metrics:

sum by (mode)(irate(node_cpu_seconds_total{mode='idle',instance=~"$node:$port",job=~"$job"}[5m])) * 100

Iowait - cpu 在 io 等待的占比

metrics:

sum by (mode)(irate(node_cpu_seconds_total{mode='iowait',instance=~"$node:$port",job=~"$job"}[5m])) * 100

Irq - cpu 在服务中断的占比

metrics:

sum by (mode)(irate(node_cpu_seconds_total{mode='irq',instance=~"$node:$port",job=~"$job"}[5m])) * 100

Softirq - cpu 在服务软中断的占比

metrics:

sum by (mode)(irate(node_cpu_seconds_total{mode='softirq',instance=~"$node:$port",job=~"$job"}[5m])) * 100

Steal - 在 VM 中运行时其他 VM 占用的本 VM 的 cpu 的占比

metrics:

sum by (mode)(irate(node_cpu_seconds_total{mode='steal',instance=~"$node:$port",job=~"$job"}[5m])) * 100

Guest - 运行各种 VM 使用的 CPU 占比

metrics:

sum by (mode)(irate(node_cpu_seconds_total{mode='guest',instance=~"$node:$port",job=~"$job"}[5m])) * 100

2. Memory Stack 内存堆栈 /proc/meminfo

type: Graph
Unit: bytes
min: "0"
Label: Bytes
Apps - 用户空间应用程序使用的内存

metrics:

node_memory_MemTotal_bytes{instance=~"$node:$port",job=~"$job"} - node_memory_MemFree_bytes{instance=~"$node:$port",job=~"$job"}
- node_memory_Buffers_bytes{instance=~"$node:$port",job=~"$job"} - node_memory_Cached_bytes{instance=~"$node:$port",job=~"$job"}
- node_memory_Slab_bytes{instance=~"$node:$port",job=~"$job"} - node_memory_PageTables_bytes{instance=~"$node:$port",job=~"$job"}
- node_memory_SwapCached_bytes{instance=~"$node:$port",job=~"$job"}

PageTables - 用于在虚拟和物理内存地址之间映射的内存

metrics:

node_memory_PageTables_bytes{instance=~"$node:$port",job=~"$job"}

SwapCache - 用于跟踪已从交换区中提取出来但尚未修改的页面的内存

metrics:

node_memory_SwapCached_bytes{instance=~"$node:$port",job=~"$job"}

Slab - 内核用于缓存数据结构以供自己使用的内存（如 inode，dentry 等缓存）

metrics:

node_memory_Slab_bytes{instance=~"$node:$port",job=~"$job"}

Cache - 频繁访问的文件数据或内容的缓存

metrics:

node_memory_Cached_bytes{instance=~"$node:$port",job=~"$job"}

Buffers - 块设备（例如硬盘）缓存

metrics:

node_memory_Buffers_bytes{instance=~"$node:$port",job=~"$job"}

Unused - 未使用的内存大小

metrics:

node_memory_MemFree_bytes{instance=~"$node:$port",job=~"$job"}

Swap - 交换分区使用的空间

metrics:

(node_memory_SwapTotal_bytes{instance=~"$node:$port",job=~"$job"} - node_memory_SwapFree_bytes{instance=~"$node:$port",job=~"$job"})

Harware Corrupted - 内核识别为已损坏或不工作的内存量

metrics:

node_memory_HardwareCorrupted_bytes{instance=~"$node:$port",job=~"$job"}

3. Network Traffic 各个网络接口的传输速率

type: Graph
Unit: bytes/sec
Label: Bytes out(-)/in(+)
{{device}} - Receive 各个网络接口下载速率

metrics:

irate(node_network_receive_bytes_total{instance=~"$node:$port",job=~"$job"}[5m])

{{device}} - Transmit 各个网络接口上传速率

metrics:

irate(node_network_transmit_bytes_total{instance=~"$node:$port",job=~"$job"}[5m])

4. Disk Space Used 所有挂载的文件系统的磁盘空间大小

type: Graph
Unit: bytes
min: "0"
Label: Bytes
metrics:

node_filesystem_size_bytes{instance=~"$node:$port",job=~"$job",device!~'rootfs'} - node_filesystem_avail_bytes{instance=~"$node:$port",job=~"$job",device!~'rootfs'}

5. Disk IOps 磁盘读写

type: Graph
Unit: I/O ops/sec (iops)
Label: IO read(-)/write(+)
{{device}} - Reads completed 磁盘的读取速率（五分钟内）

metrics:

irate(node_disk_reads_completed_total{instance=~"$node:$port",job=~"$job",device=~"[a-z]*[a-z]"}[5m])

{{device}} - Writes completed 磁盘的写入速率（五分钟内）

metrics:

irate(node_disk_writes_completed_total{instance=~"$node:$port",job=~"$job",device=~"[a-z]*[a-z]"}[5m])

6. I/O Usage Read / Write

type: Graph
Unit: bytes
Label: Bytes read(-)/write(+)
成功读取的字节数（五分钟内）

metrics:

irate(node_disk_read_bytes_total{instance=~"$node:$port",job=~"$job",device=~"[a-z]*[a-z]"}[5m])

成功写入的字节数（五分钟内）

metrics:

irate(node_disk_written_bytes_total{instance=~"$node:$port",job=~"$job",device=~"[a-z]*[a-z]"}[5m])

7. I/O Usage Times 执行I / O所花费的总时间。

type: Graph
Unit: ms
Label: Milliseconds
metrics:

irate(node_disk_io_time_seconds_total{instance=~"$node:$port",job=~"$job",device=~"[a-z]*[a-z]"} [5m])