Prometheus + Alertmanager 实现企微告警

上一篇：二进制安装Prometheus

 下面准备在监控的流程中呈现到告警到企微

查看企业ID，用于后续配置文件

四、安装Alertmanager
1、准备安装的包

--选择上面链接给的Linux的tar包

alertmanager-0.22.2.linux-amd64.tar.gz
wget https://github.com/prometheus/alertmanager/releases/download/v0.22.2/alertmanager-0.22.2.linux-amd64.tar.gz

2、下载完之后直接解压并放到/usr/local/prometheus目录,便于管理

[root@zhoujt prometheus]# tar -zxvf alertmanager-0.22.2.linux-amd64.tar.gz
[root@zhoujt prometheus]# cp -r alertmanager-0.22.2.linux-amd64 /usr/local/prometheus/alertmanager
[root@zhoujt prometheus]# cd /usr/local/prometheus/alertmanager/
[root@zhoujt alertmanager]# ls
alertmanager alertmanager.yml amtool LICENSE NOTICE
[root@zhoujt alertmanager]# ./alertmanager --version
alertmanager, version 0.22.2 (branch: HEAD, revision: 44f8adc06af5101ad64bd8b9c8b18273f2922051)
build user: root@b595c7f32520
build date: 20210602-07:50:37
go version: go1.16.4
platform: linux/amd64

3、配置alertmanager

[zhoujt@zhoujt alertmanager]$ cat alertmanager.yml
global: #每五分钟检查一次是否恢复
resolve_timeout: 5m
# SMTP的相关配置
# smtp_smarthost: smtp.263.net:587
# smtp_from: no-reply@xxx.com
# smtp_auth_username: no-reply@xxx.com
# smtp_auth_password: xxx

# 路由的根节点，每个传进来的报警从这里开始
route:
group_by: ['alertname'] # 将传入的报警中有这些标签的分为一个组
group_wait: 10s # 第一次触发报警的延时
group_interval: 10s # 自第一次告警等待多久发送压缩的警报
repeat_interval: 1m # 重复告警发送间隔
receiver: 'wechat' # 定义告警接收的对象
receivers: # 告警接收对象
- name: 'wechat' #告警接收名称，与route的receiver对应
wechat_configs:
- corp_id: 'wwfaxxxxxxxxxxxx' # 企业微信唯一ID，我的企业--企业信息
to_party: '1' # 告警需要发送的组
to_user: '1' # 告警发送的用户ID
agent_id: '1000002' # 自己创建应用的ID
api_secret: 'o22cBPAm3xxxxxxxxxxxxxxxxxxx' # 应用密钥
send_resolved: true # 告警解决后是否发送通知
inhibit_rules: # 告警抑制规则，比如阈值告警，达到critical肯定也达到了warning了，没必要发送两个告警
- source_match:
severity: 'major'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
templates: #告警消息模板
- '/usr/local/prometheus/alertmanager/*.tmpl'

4、配置告警信息模板
注意：配置这些配置文件时，一定要是utf-8的形式，否则无法启动服务
- file filename # 查看文件属性
UTF-8 Unicode text
- set fileencoding=utf-8
ps: 模板的时间切记不要改，这个是go语言定义的一月二号下午三点四分五秒，06年时区是-7

{{ define "wechat.default.message" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts -}}
======== 异常告警 ========
告警名称：{{ $alert.Labels.alertname }}
告警级别：{{ $alert.Labels.severity }}
告警机器：{{ $alert.Labels.instance }} {{ $alert.Labels.device }}
告警详情：{{ $alert.Annotations.summary }}
告警时间：{{ $alert.StartsAt.Format "2006-01-02 15:04:05" }}
========== END ==========
{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $index, $alert := .Alerts -}}
======== 告警恢复 ========
告警名称：{{ $alert.Labels.alertname }}
告警级别：{{ $alert.Labels.severity }}
告警机器：{{ $alert.Labels.instance }}
告警详情：{{ $alert.Annotations.summary }}
告警时间：{{ $alert.StartsAt.Format "2006-01-02 15:04:05" }}
恢复时间：{{ $alert.EndsAt.Format "2006-01-02 15:04:05" }}
========== END ==========
{{- end }}
{{- end }}
{{- end }}

5、测试告警是否正常，首先编辑告警触发规则

groups:
- name: mem-rule
rules:
- alert: "内存报警"
expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes )) / node_memory_MemTotal_bytes * 100 > 10
for: 30s
labels:
severity: warning
annotations:
summary: "服务名:{{$labels.alertname}} 内存报警"
description: "{{ $labels.alertname }} 内存资源利用率大于 10%"
value: "{{ $value }}"
- name: node-up
rules:
- alert: "节点状态"
expr: up{job="node-exporter"} == 0 #测试的话可以把节点改为1,不方便停止节点的时候
for: 5s
labels:
severity: ERROR
level: error
annotations:
summary: "{{ $labels.instance }} 已停止15s!"
description: "{{ $labels.instance }} 检测到异常！请重点关注！！！"
value: "{{ $value }}"

- name: node_health
rules:
- alert: HighMemoryUsage
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes > 0.9
for: 1m
labels:
severity: warning
annotations:
summary: High memory usage

- alert: HighDiskUsage
expr: node_filesystem_free_bytes{mountpoint='/'} / node_filesystem_size_bytes{mountpoint='/'} > 0.7
for: 1m
labels:
severity: major
annotations:
summary: High Disk usage

- alert: HighDiskUsage
expr: node_filesystem_free_bytes{mountpoint='/'} / node_filesystem_size_bytes{mountpoint='/'} > 0.71
for: 1m
labels:
severity: warning
annotations:
summary: High Disk usage

6、配置systemd对应服务，便于自启动和管理

[zhoujt@zhoujt rules]$ cat /usr/lib/systemd/system/alertmanager.service
[Unit]
Description=altermanager
After=network.target

[Service]
ExecStart=/usr/local/prometheus/alertmanager/alertmanager --config.file=/usr/local/prometheus/alertmanager/alertmanager.yml
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

[zhoujt@zhoujt prometheus]$ cat /usr/lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/
After=network.target
[Service]
# Type设置为notify时，服务会不断重启
Type=simple
User=prometheus
# --storage.tsdb.path是可选项，默认数据目录在运行目录的./data目录中
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/home/prometheus/prometheus-data --web.enable-lifecycle
#ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/home/prometheus/prometheus-date --web.listen-address=:9099
Restart=on-failure
[Install]
WantedBy=multi-user.target

7、配置Prometheus的配置文件，使用alertmanager

# Alertmanager configuration
#alerting:
# alertmanagers:
# - static_configs:
# - targets:
# - 127.0.0.1:9093
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']

8、基本配置已完成，开始启动服务，查看端口

- 重载Prometheus: curl -X POST http://localhost:9090/-/reload
或者: systemctl reload prometheus
- 启动Alertmanager: systemctl enable alertmanager&& systemctl start alertmanager
tcp6 0 0 :::9090 :::* LISTEN 11401/prometheus
tcp6 0 0 :::9093 :::* LISTEN 30974/alertmanager
tcp6 0 0 :::9094 :::* LISTEN 30974/alertmanager
访问 9090 9093 可以查看当前状态

9、服务启动成功

10、测试的话，将rule里面改几个参数，

告警时：

恢复后：

这里除了监控节点是否存活外，还可以监控很多很多指标，例如 CPU 负载告警、Mem 使用量告警、Disk 存储空间告警、Network 负载告警等等，这些都可以通过自定义 PromQL 表达式验证值来定义一些列的告警规则，来丰富日常工作中需要的各种告警

到这里，企微告警已完成，后续可以配置邮件告警，在配置文件中注释掉了