jmx_prometheus_javaagent+prometheus+alertmanager+grafana完成容器化java监控告警（二）

一、拓扑图

二、收集数据

2.1前期准备

创建共享目录，即为了各节点都创建该目录,有两个文件，做数据共享
/home/target/prom-jvm-demo
1、下载文件
jmx_prometheus_javaagent-0.3.1.jar
2、创建文件
[root@manager prom-jvm-demo]# cat simple-config.yml
---
rules:
- pattern: ".*"

2.2启动容器

注：6060作为java监控数据暴露端口，每一个服务单独占用一个端口，即启动java服务便伴随着启动java监控服务

2.2.1 docker 启动（测试）

docker run -d \
  --name tomcat-1 \
  -v /home/target/prom-jvm-demo:/jmx-exporter \
  -e CATALINA_OPTS="-Xms64m -Xmx128m -javaagent:/jmx-exporter/jmx_prometheus_javaagent-0.3.1.jar=6060:/jmx-exporter/simple-config.yml" \
  -p 6060:6060 \
  -p 8012:8080 \
  10.0.0.194:4999/tomcat:8.5.61

2.2.2 docker 启动

docker service create --name bj_ywjy_yxsmgr2 \
 --publish 6061:6060 \
 --publish 8081:8080 \
 --mount type=bind,src=/home/target/bj_ywjy_yxsmgr,dst=/usr/local/tomcat/webapps  \
 --mount type=bind,src=/home/target/bj_ywjy_yxsmgr/logs,dst=/usr/local/tomcat/logs  \
 --mount type=bind,src=/home/target/prom-jvm-demo,dst=/jmx-exporter  \
 --env CATALINA_OPTS="-Xms64m -Xmx128m -javaagent:/jmx-exporter/jmx_prometheus_javaagent-0.3.1.jar=6060:/jmx-exporter/simple-config.yml" \
 --network springcloud-overlay  \
 --log-driver=gelf --log-opt gelf-address=udp://10.0.0.184:12201  \
 --log-opt tag=bj_ywjy_yxsmgr2  10.0.0.194:4999/tomcat:8.5.61

2.2.3 访问获取到的数据

http://10.0.0.196:6060/metrics
 

三、prometheus配置

3.1 将获取到的java数据关联到prometheus中

[root@autodeploy prometheus-2.25.2]# cat prometheus.yml
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['10.0.0.189:9090']

  - job_name: 'Container monitoring'
    static_configs:
    - targets:
      - "10.0.0.194:8888"
      - "10.0.0.195:8888"
      - "10.0.0.196:8888"

  - job_name: 'bj_ywjy_yxsmgr'
    static_configs:
    - targets:
      - "10.0.0.194:6060"
  - job_name: 'bj_ywjy_yxsmgr2'
    static_configs:
    - targets:
      - "10.0.0.194:6061"

配置jvm监控规则可根据 cadvisor+prometheus+alertmanager+grafana完成容器化监控告警（一）配置参数调整改进，这里就不体现了！

3.2重启prometheus

[root@autodeploy prometheus-2.25.2]# cat start.sh
#!/bin/bash
pid=`ps -ef|grep prometheus|grep -v grep|awk '{print $2}'`
kill -9 $pid
nohup ./prometheus --config.file=prometheus.yml &

3.3访问测试

http://10.0.0.189:9090/

四、alertmanager配置告警

4.1告警模板

[root@autodeploy rules]# cat weixin.tmpl
{{ define "wechat.default.message" }}
{{ range $i, $alert :=.Alerts }}
========监控报警====
告警状态: {{   .Status }}
告警级别: {{ $alert.Labels.severity }}
告警类型: {{ $alert.Labels.alertname }}
告警应用: {{ $alert.Labels.name }} {{ $alert.Labels.job }}
告警主机: {{ $alert.Labels.instance }}
当前最新值: {{ $alert.Annotations.summary }}
告警详情: {{ $alert.Annotations.description }}
告警时间: {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
========end=========
{{ end }}
{{ end }}

4.2重启alertmanager

[root@autodeploy alertmanager-0.21.0]# cat start.sh
#!/bin/bash
pid=`ps -ef|grep aler|grep -v grep|awk '{print $2}'`
kill -9 $pid
nohup ./alertmanager --config.file=alertmanager.yml --storage.path=data --log.level=debug &

五、微信告警展示