host主机监控规则
1.先在 Prometheus 主程序目录下创建rules目录,然后在该目录下创建 host.yml文件,内容如下:
内容很多,可以根据实际情况进行调整。
规则参考网址:https://awesome-prometheus-alerts.grep.to/rules
参考网址的规则中,有些地方需要修改,比如:
- alert: HostNetworkReceiveErrors
expr: increase(node_network_receive_errs_total[5m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Host Network Receive Errors (instance {{ $labels.instance }})"
description: "{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf "%.0f" $value }} receive errors in the last five minutes.\n VALUE = {{ $value }}\n LABELS: {{ $labels }}" ```
具体使用的时候需要修改description,最外层的双引号修改成单引号,因为里面的`"%.0f"`已经使用双引号了。修改成:
'{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf "%.0f" $value }} receive errors in the last five minutes.\n VALUE = {{ $value }}\n LABELS: {{ $labels }}'
注意:目录和文件的权限:chown -R prometheus:prometheus rules
```yaml
groups:
- name: Host and hardware
rules:
- alert: HostOutOfMemory
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
for: 5m
labels:
severity: warning
annotations:
summary: "Host out of memory (instance {{ $labels.instance }})"
description: "Node memory is filling up (< 10% left)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostMemoryUnderMemoryPressure
expr: rate(node_vmstat_pgmajfault[1m]) > 1000
for: 5m
labels:
severity: warning
annotations:
summary: "Host memory under memory pressure (instance {{ $labels.instance }})"
description: "The node is under heavy memory pressure. High rate of major page faults\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostUnusualNetworkThroughputIn
expr: sum by (instance) (irate(node_network_receive_bytes_total[2m])) / 1024 / 1024 > 100
for: 5m
labels:
severity: warning
annotations:
summary: "Host unusual network throughput in (instance {{ $labels.instance }})"
description: "Host network interfaces are probably receiving too much data (> 100 MB/s)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostUnusualNetworkThroughputOut
expr: sum by (instance) (irate(node_network_transmit_bytes_total[2m])) / 1024 / 1024 > 100
for: 5m
labels:
severity: warning
annotations:
summary: "Host unusual network throughput out (instance {{ $labels.instance }})"
description: "Host network interfaces are probably sending too much data (> 100 MB/s)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostUnusualDiskReadRate
expr: sum by (instance) (irate(node_disk_read_bytes_total[2m])) / 1024 / 1024 > 50
for: 5m
labels:
severity: warning
annotations:
summary: "Host unusual disk read rate (instance {{ $labels.instance }})"
description: "Disk is probably reading too much data (> 50 MB/s)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostUnusualDiskWriteRate
expr: sum by (instance) (irate(node_disk_written_bytes_total[2m])) / 1024 / 1024 > 50
for: 5m
labels:
severity: warning
annotations:
summary: "Host unusual disk write rate (instance {{ $labels.instance }})"
description: "Disk is probably writing too much data (> 50 MB/s)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostOutOfDiskSpace
expr: (node_filesystem_avail_bytes{mountpoint="/rootfs"} * 100) / node_filesystem_size_bytes{mountpoint="/rootfs"} < 10
for: 5m
labels:
severity: warning
annotations:
summary: "Host out of disk space (instance {{ $labels.instance }})"
description: "Disk is almost full (< 10% left)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostDiskWillFillIn4Hours
expr: predict_linear(node_filesystem_free_bytes{fstype!~"tmpfs"}[1h], 4 * 3600) < 0
for: 5m
labels:
severity: warning
annotations:
summary: "Host disk will fill in 4 hours (instance {{ $labels.instance }})"
description: "Disk will fill in 4 hours at current write rate\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostOutOfInodes
expr: node_filesystem_files_free{mountpoint ="/rootfs"} / node_filesystem_files{mountpoint ="/rootfs"} * 100 < 10
for: 5m
labels:
severity: warning
annotations:
summary: "Host out of inodes (instance {{ $labels.instance }})"
description: "Disk is almost running out of available inodes (< 10% left)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostUnusualDiskReadLatency
expr: rate(node_disk_read_time_seconds_total[1m]) / rate(node_disk_reads_completed_total[1m]) > 100
for: 5m
labels:
severity: warning
annotations:
summary: "Host unusual disk read latency (instance {{ $labels.instance }})"
description: "Disk latency is growing (read operations > 100ms)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostUnusualDiskWriteLatency
expr: rate(node_disk_write_time_seconds_total[1m]) / rate(node_disk_writes_completed_total[1m]) > 100
for: 5m
labels:
severity: warning
annotations:
summary: "Host unusual disk write latency (instance {{ $labels.instance }})"
description: "Disk latency is growing (write operations > 100ms)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostHighCpuLoad
expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 5m
labels:
severity: warning
annotations:
summary: "Host high CPU load (instance {{ $labels.instance }})"
description: "CPU load is > 80%\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostContextSwitching
expr: (rate(node_context_switches_total[5m])) / (count without(cpu, mode) (node_cpu_seconds_total{mode="idle"})) > 1000
for: 5m
labels:
severity: warning
annotations:
summary: "Host context switching (instance {{ $labels.instance }})"
description: "Context switching is growing on node (> 1000 / s)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostSwapIsFillingUp
expr: (1 - (node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes)) * 100 > 80
for: 5m
labels:
severity: warning
annotations:
summary: "Host swap is filling up (instance {{ $labels.instance }})"
description: "Swap is filling up (>80%)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostSystemdServiceCrashed
expr: node_systemd_unit_state{state="failed"} == 1
for: 5m
labels:
severity: warning
annotations:
summary: "Host SystemD service crashed (instance {{ $labels.instance }})"
description: "SystemD service crashed\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostPhysicalComponentTooHot
expr: node_hwmon_temp_celsius > 75
for: 5m
labels:
severity: warning
annotations:
summary: "Host physical component too hot (instance {{ $labels.instance }})"
description: "Physical hardware component too hot\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostNodeOvertemperatureAlarm
expr: node_hwmon_temp_alarm == 1
for: 5m
labels:
severity: critical
annotations:
summary: "Host node overtemperature alarm (instance {{ $labels.instance }})"
description: "Physical node temperature alarm triggered\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostRaidArrayGotInactive
expr: node_md_state{state="inactive"} > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Host RAID array got inactive (instance {{ $labels.instance }})"
description: "RAID array {{ $labels.device }} is in degraded state due to one or more disks failures. Number of spare drives is insufficient to fix issue automatically.\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostRaidDiskFailure
expr: node_md_disks{state="fail"} > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Host RAID disk failure (instance {{ $labels.instance }})"
description: "At least one device in RAID array on {{ $labels.instance }} failed. Array {{ $labels.md_device }} needs attention and possibly a disk swap\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostKernelVersionDeviations
expr: count(sum(label_replace(node_uname_info, "kernel", "$1", "release", "([0-9]+.[0-9]+.[0-9]+).*")) by (kernel)) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "Host kernel version deviations (instance {{ $labels.instance }})"
description: "Different kernel versions are running\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostOomKillDetected
expr: increase(node_vmstat_oom_kill[5m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Host OOM kill detected (instance {{ $labels.instance }})"
description: "OOM kill detected\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: HostEdacCorrectableErrorsDetected
expr: increase(node_edac_correctable_errors_total[5m]) > 0
for: 5m
labels:
severity: info
annotations:
summary: "Host EDAC Correctable Errors detected (instance {{ $labels.instance }})"
description: '{{ $labels.instance }} has had {{ printf "%.0f" $value }} correctable memory errors reported by EDAC in the last 5 minutes.\n VALUE = {{ $value }}\n LABELS: {{ $labels }}'
- alert: HostEdacUncorrectableErrorsDetected
expr: node_edac_uncorrectable_errors_total > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Host EDAC Uncorrectable Errors detected (instance {{ $labels.instance }})"
description: '{{ $labels.instance }} has had {{ printf "%.0f" $value }} uncorrectable memory errors reported by EDAC in the last 5 minutes.\n VALUE = {{ $value }}\n LABELS: {{ $labels }}'
- alert: HostNetworkReceiveErrors
expr: increase(node_network_receive_errs_total[5m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Host Network Receive Errors (instance {{ $labels.instance }})"
description: '{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf "%.0f" $value }} receive errors in the last five minutes.\n VALUE = {{ $value }}\n LABELS: {{ $labels }}'
- alert: HostNetworkTransmitErrors
expr: increase(node_network_transmit_errs_total[5m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Host Network Transmit Errors (instance {{ $labels.instance }})"
description: '{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf "%.0f" $value }} transmit errors in the last five minutes.\n VALUE = {{ $value }}\n LABELS: {{ $labels }}'
- alert: JvmMemoryFillingUp
expr: jvm_memory_bytes_used / jvm_memory_bytes_max{area="heap"} > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "JVM memory filling up (instance {{ $labels.instance }})"
description: "JVM memory is filling up (> 80%)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: SpeedtestSlowInternetDownload
expr: avg_over_time(speedtest_download[30m]) < 75
for: 5m
labels:
severity: warning
annotations:
summary: "SpeedTest Slow Internet Download (instance {{ $labels.instance }})"
description: "Internet download speed is currently {{humanize $value}} Mbps.\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: SpeedtestSlowInternetUpload
expr: avg_over_time(speedtest_upload[30m]) < 20
for: 5m
labels:
severity: warning
annotations:
summary: "SpeedTest Slow Internet Upload (instance {{ $labels.instance }})"
description: "Internet upload speed is currently {{humanize $value}} Mbps.\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
2.在 Prometheus 主程序目录下的prometheus.yml进行修改,引用上述告警规则
rule_files:
- "rules/*.yml"
3.重启Prometheus
host主机监控规则的更多相关文章
- redis监控规则
其他说明参考host主机监控规则:https://www.cnblogs.com/sanduzxcvbnm/p/13589848.html groups: - name: Redis monitori ...
- cAdvisor容器监控规则
其他说明参考host主机监控规则:https://www.cnblogs.com/sanduzxcvbnm/p/13589848.html 在prometheus主程序目录下的rules目录下新建do ...
- Weave Scope 多主机监控 - 每天5分钟玩转 Docker 容器技术(81)
除了监控容器,Weave Scope 还可以监控 Docker Host. 点击顶部 HOSTS 菜单项,地图将显示当前 host. 与容器类似,点击该 host 图标将显示详细信息. host 当前 ...
- 081、Weave Scope 多主机监控(2019-04-29 周一)
参考https://www.cnblogs.com/CloudMan6/p/7674011.html Weave Scope 除了监控容器,还可以监控Docker Host. 点击顶部 HOS ...
- 第 10 章 容器监控 - 081 - Weave Scope 多主机监控
除了监控容器,Weave Scope 还可以监控 Docker Host 点击顶部 HOSTS 菜单项,地图将显示当前 host. 与容器类似,点击该 host 图标将显示详细信息 host 当前的资 ...
- Zabbix学习之路(二)之添加主机监控及自定义item监控
1.zabbix_get命令详解 安装zabbix-get命令 [root@linux-node1 ~]# yum install -y zabbix_get 参数说明: -s --host: 指定客 ...
- /cloudmonitor.log 主机监控
989866842 INFO 2018-09-30 01:38:58.58 [ricGatherServiceHttp] 提交指标完成,耗时:18ms. SystemInfo [serialNumbe ...
- zabbix如何添加主机监控
1,首先,监控的主机安装zabbix客户端.zabbix提供多种监控方式,我们这里监控的主机上边安装agentd守护端进行数据收集并监测. 其中客户端安装我们这里就不介绍了,请参考之前教程里边的客户端 ...
- 源码解析.Net中Host主机的构建过程
前言 本篇文章着重讲一下在.Net中Host主机的构建过程,依旧延续之前文章的思路,着重讲解其源码,如果有不知道有哪些用法的同学可以点击这里,废话不多说,咱们直接进入正题 Host构建过程 下图是我自 ...
随机推荐
- [原创]移远RM500U-CN模组驱动移植
1. 简介 中国广电正式放号了,为了支持广电700MHz的5G基站,需要换用新的5G模组.移远通信的RM500U模组正好可以满足我们的使用要求; 我们选用该模组的原因:双卡单待 支持SIM卡热插拔 支 ...
- 7 什么是dubbo
什么是dubbo 快速入门dubbo 了解什么是dubbo之前,我们得先了解什么是分布式系统? <分布式系统原理与范型>定义: 分布式系统是若干独立计算机的集合,这些计算机对于用户来说就像 ...
- 流程控制语句continue
continue语句 用于结束当前循环,进入下一次循环,同样通常与if分支结构一起使用 (这边和前面的break可以结合在一起与C中的一样的理解) 注意这个不是终止整个循环只是终止当前循环进行下一次循 ...
- 2022-7-19 第五组 pan小堂 封装和 this
this关键字 this关键字由来和使用: A:this:代表所在类的对象引用方法被哪个对象调用,this就代表那个对象 B:什么时候使用this呢 ? 局部变量和成员变量重名 set 和 get 方 ...
- DP の 百题大过关(5/100)
动态规划自古以来是DALAO凌虐萌新的分水岭,但有些OIer认为并没有这么重要--会打暴力,大不了记忆化.但是其实,动态规划学得好不好,可以彰显出一个OIerOIer的基本素养--能否富有逻辑地思 ...
- NRooks采样类定义和测试
类声明: #pragma once #ifndef __NROOKS_HEADER__ #define __NROOKS_HEADER__ #include "sampler.h" ...
- java-Servlet编码/异常处理
1. Servlet输出中文(1)为什么会有乱码?out.println方法在输出时或者表单提交的时候,浏览器会对表单中的中文参数值进行编码; 注:会使用表单所在的页面打开时使用的编码方式进行编码服务 ...
- 【HTML】学习路径4-align对齐-标签属性
每个标签都可以设置各种属性,比如可以给一个段落标签添加一个name的属性: <p name="段落标签1"> 这一节我们学习一个属性:align对齐标签 第一章:ali ...
- SpringMVC 04: SpringMVC中4种页面跳转方式
转发和重定向的页面跳转方式 页面跳转方式,本质上只有2种方式:转发 + 重定向 但在SpringMVC的具体实现上,转发可以细分为:普通的页面转发 + 经由action方法的页面转发 重定向可以细分为 ...
- KingbaseES R6集群归档备份故障分析解决案例
案例说明: 在使用ps工具查看主库进程,发现主库'archiver'进程失败,检查sys_log日志可以发现归档失败的信息.通过sys_log日志提取归档语句手工执行归档操作,提示"当前数据 ...