一套完整的中小级别的企业级监控prometheus

一   相信有很多博客都已经详细的说明了prometheus的作用以及相关的作用以及原理，这里不在赘述，仅仅从部署和配置2个方面来记录一下，为公司产品组搭建的prometheus告警平台的过程以及踩过的坑,废话不多说，直接开始搭建部署，需要在一台服务器上面搭建prometheus+grafana+alertmanager+pushgateway，其余被监控的节点部署node_exporter,也可以在prometheus服务端部署node_exporter

　　1.1 部署prometheus，并且使用systemctl进行管控

　　　　安装版本：prometheus-2.6.1

百度云下载：https://pan.baidu.com/s/1w16lQZKw8PCHqlRuSK2i7A

提取码：lw1q

　　　　　之后将包解压到: /usr/local/prometheus目录下面，建议使用ansible脚本进行部署

　　　　　这里附上安装管理的管理文件以及目录地址/usr/lib/systemd/system/prometheus.service

[Unit]
  Description=https://prometheus.io

  [Service]
  Restart=on-failure
  ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml

  [Install]
  WantedBy=multi-user.target

　　 1.2  整理后的prometheus配置文件,添加新的监控节点job_name和机器的节点，并且节点需要安装相应的node_exporter

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 172.16.5.3:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
   - "rules/first_rules.yml"
   - "rules/second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090','172.16.5.3:9100']- job_name: 'pushgateway'
    scrape_interval: 5s
    static_configs:
    - targets: ['172.16.5.3:9091']
      labels:
        instance: pushgateway

　　1.3 对服务器的基础监控项如如下所示

#cat second_rules.yml
groups:
- name: 实例存活告警规则
  rules:
  - alert: 实例存活告警
    expr: up{job="prometheus"} == 0 or up{job="Linux-host"} == 0
    for: 1m
    labels:
      user: prometheus
      severity: emergency
      team: HTY
    annotations:
      summary: "Instance {{ $labels.instance }} is down"
      description: "Instance {{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes."
      value: "{{ $value }}"

- name: 内存告警规则
  rules:
  - alert: "内存使用率告警"
    expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes )) / node_memory_MemTotal_bytes * 100 > 30
    for: 1m
    labels:
      team: C3
      user: prometheus
      severity: warning
    annotations:
      summary: "服务器: {{$labels.alertname}} 内存报警"
      description: "{{ $labels.alertname }} 内存资源利用率大于30%！(当前值: {{ $value }}%)"
      value: "{{ $value }}"

- name: 内存告警规则2
  rules:
  - alert: "内存使用率告警2"
    expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes )) / node_memory_MemTotal_bytes * 100 > 50
    for: 1m
    labels:
      team: C3
      user: prometheus
      severity: critical
    annotations:
      summary: "服务器: {{$labels.alertname}} 内存报警"
      description: "{{ $labels.alertname }} 内存资源利用率大于50%！(当前值: {{ $value }}%)"
      value: "{{ $value }}"

- name: CPU报警规则
  rules:
  - alert: CPU使用率告警
    expr: 100 - (avg by (instance)(irate(node_cpu_seconds_total{mode="idle"}[1m]) )) * 100 > 70
    for: 1m
    labels:
      user: prometheus
      severity: warning
    annotations:
      summary: "服务器: {{$labels.alertname}} CPU报警"
      description: "服务器: CPU使用超过70%！(当前值: {{ $value }}%)"
      value: "{{ $value }}"

- name: 磁盘报警规则
  rules:
  - alert: 磁盘使用率告警
    expr: (node_filesystem_size_bytes - node_filesystem_avail_bytes) / node_filesystem_size_bytes * 100 > 80
    for: 1m
    labels:
      user: prometheus
      severity: warning
    annotations:
      summary: "服务器: {{$labels.alertname}} 磁盘报警"
      description: "服务器:{{$labels.alertname}},磁盘设备: 使用超过80%！(挂载点: {{ $labels.mountpoint }} 当前值: {{ $value }}%)"
      value: "{{ $value }}"

　　2 安装以及配置alertmanager

global:
  # 企业微信告警配置
  resolve_timeout: 5m
  wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin/'
  wechat_api_corp_id: 'ww41a2b13ef47aac58'
  wechat_api_secret: 'xxxxx'
  # qq邮箱告警配置
  smtp_from: xxx@qq.com
  smtp_auth_username: xx@qq.com
  smtp_auth_password: xxxx #需要从qq邮箱上面获取
  smtp_require_tls: false
  smtp_smarthost: 'smtp.qq.com:465'
templates:
  - "/usr/local/alertmanager/template/*.tmpl"
route:
  receiver: 'default-receiver'
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 1m
  group_by: ['team']
  routes:
  - group_by: ['test']
    group_wait: 10s
    group_interval: 30s
    repeat_interval: 1m
    receiver: 'wechat'
    match:
      team: test1
receivers:
- name: 'wechat'
  wechat_configs:
  - send_resolved: true
    message: '{{ template "wechat.default.message" .}}'
    to_party: 'xxxx'
    agent_id: "xxx"需要从企业微信上面获取
    api_secret: 'xxxxxxxx'
- name: 'default-receiver'
  email_configs:
  - to: 'xxxxxx@qq.com'
    send_resolved: true
    # html: '{{ template "wechat.default.message" .}}'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['env','team','instance','type','group','job','alertname']

　　获取企业微信的方式参考这个链接：https://www.cnblogs.com/miaocbin/p/13706164.html

　　获取qq邮箱参考这个链接：https://blog.csdn.net/knight_zhou/article/details/105137581　

3 附上模版信息

{{ define "wechat.default.message" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts -}}
{{- if eq $index 0 }}
========= 监控报警 =========
告警状态：{{   .Status }}
告警级别：{{ .Labels.severity }}
告警类型：{{ $alert.Labels.alertname }}
故障主机: {{ $alert.Labels.instance }}
告警主题: {{ $alert.Annotations.summary }}
告警详情: {{ $alert.Annotations.message }}{{ $alert.Annotations.description}};
触发阀值：{{ .Annotations.value }}
故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
========= = end =  =========
{{- end }}
{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $index, $alert := .Alerts -}}
{{- if eq $index 0 }}
========= 异常恢复 =========
告警类型：{{ .Labels.alertname }}
告警状态：{{   .Status }}
告警主题: {{ $alert.Annotations.summary }}
告警详情: {{ $alert.Annotations.message }}{{ $alert.Annotations.description}};
故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
恢复时间: {{ ($alert.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
{{- if gt (len $alert.Labels.instance) 0 }}
实例信息: {{ $alert.Labels.instance }}
{{- end }}
========= = end =  =========
{{- end }}
{{- end }}
{{- end }}
{{- end }}

　　4. 安装以及部署grafana，推荐安装最新版的prometheus，然后使用插件，附上一个比较简洁的grafana看板

　　直接倒入模板，倒入步骤参考这便博客：https://www.cnblogs.com/wukc/p/14231042.html