09 . Prometheus监控tomcat+jvm

List

  CentOS7.3
  prometheus-2.2.1.linux-amd64.tar.gz
  redis_exporter-v0.30.0.linux-amd64.tar.gz

节点名	IP	软件版本	硬件	网络	说明
Prometheus	192.168.43.144	list 里面都有	2C4G	Nat，内网	测试环境
Tomcat	192.168.43.134	list 里面都有	2C4G	Nat，内网	测试环境

部署Prometheus

部署Promehteus请看我前面写的文章

https://www.cnblogs.com/you-men/p/12839535.html

部署Tomcat

下载安装tomcat

wget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.55/bin/apache-tomcat-8.5.55.tar.gz
tar xvf apache-tomcat-8.5.47.tar.gz -C /usr/local/
ln -s /usr/local/apache-tomcat-8.5.47/ /usr/local/tomcat
tail -1 /etc/profile
export CATALINA_HOME=/usr/local/tomcat

source /etc/profile
env |grep -i home
CATALINA_HOME=/usr/local/tomcat
HOME=/root

# 查看熵池的大小：
cat /proc/sys/kernel/random/entropy_avail 180
增加熵池大小，解决Tomcat在CentOS 7巨慢的问题

# 安装rng服务,增大熵池
yum -y install rng-tools
systemctl start rngd && systemctl enable rngd
# 启动服务访问Tomcat页面快一百倍..

启动tomcat

/usr/local/tomcat/bin/catalina.sh start
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/local/jdk
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
ss -antp |grep java
LISTEN     0      100          *:8009                     *:*                   users:(("java",pid=857,fd=58))
LISTEN     0      100          *:8080                     *:*                   users:(("java",pid=857,fd=53))
LISTEN     0      1      127.0.0.1:8005                     *:*                   users:(("java",pid=857,fd=73))

tomcat三个端口

# 8005:    是tomcat本身的端口，本地管理端口号
# 8080:    主端口号，是tomcat负责建立http连接的端口，通过浏览器访问tomcat服务器的web 应用时，使用此端口
# 8009:    是tomcat负责和其他http服务器建立连接的端口，ajp的编辑器

tomcat启动脚本

vim /etc/init.d/tomcat
#!/bin/bash
#chkconfig: 2345 96 14
export JAVA_HOME=/usr/java/jdk1.8.0_121
export CATALINA_HOME=/usr/local/tomcat
exec $CATALINA_HOME/bin/catalina.sh $*

# $*将脚本本身自带的参数全部传递过来，就是catalina.sh --help看到的所有的参数start stop等

chmod +x /etc/init.d/tomcat
chkconfig --add tomcat
chkconfig tomcat on
service tomcat start
systemctl status tomcat
tail -f catalina.out 服务启动不起来查看这个日志文件

配置JMX Exporter

项目地址:

https://github.com/prometheus/jmx_exporter

# jar包地址:
https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar

mkdir -p /usr/local/jmx/
cp jmx_prometheus_javaagent-0.3.1.jar /usr/local/jmx/
# 下载地址：https://github.com/prometheus/jmx_exporter/blob/master/example_configs/tomcat.yml
cp /usr/local/src/tomcat.yaml /usr/local/jmx/config.yaml 

cat /usr/local/jmx/config.yaml
---
lowercaseOutputLabelNames: true
lowercaseOutputName: true
rules:
- pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
  name: tomcat_$3_total
  labels:
    port: "$2"
    protocol: "$1"
  help: Tomcat global $3
  type: COUNTER
- pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
  name: tomcat_servlet_$3_total
  labels:
    module: "$1"
    servlet: "$2"
  help: Tomcat servlet $3 total
  type: COUNTER
- pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
  name: tomcat_threadpool_$3
  labels:
    port: "$2"
    protocol: "$1"
  help: Tomcat threadpool $3
  type: GAUGE
- pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
  name: tomcat_session_$3_total
  labels:
    context: "$2"
    host: "$1"
  help: Tomcat session $3 total
  type: COUNTER

收集tomcat数据

Jar包应用

java -javaagent:./jmx_prometheus_javaagent-0.3.0.jar=9151:config.yaml -jar yourJar.jar

War包应用

进入bin目录$TOMCAT_HOME/bin，将jmx_exporter.jar包文件和config.yaml文件复制到这里。然后修改里面的一个catalina.sh的脚本，找到JAVA_OPTS,加上以下配置(代理)：

如果有多tomcat,建议将jmx_prometheus_javaagent和config.yaml文件放到固定的目录,$TOMCAT_HOME/bin/catalina.sh文件中写绝对路径.

# 修改bin/catalina.sh 文件
# 添加：
JAVA_OPTS="-javaagent:/usr/local/jmx/jmx_prometheus_javaagent-0.3.1.jar=39081:/usr/local/jmx/config.yaml"

# service tomcat restart

ss -atnlp |grep java
LISTEN     0      100          *:8080                     *:*                   users:(("java",pid=2110,fd=57))
LISTEN     0      1      127.0.0.1:8005                     *:*                   users:(("java",pid=2110,fd=76))
LISTEN     0      100          *:8009                     *:*                   users:(("java",pid=2110,fd=61))
LISTEN     0      3            *:39081                    *:*                   users:(("java",pid=2110,fd=14))

配置Prometheus

添加tomcat到prometheus

tail -3 /usr/local/prometheus/prometheus.yml
  - job_name: 'tomcat'
    static_configs:
      - targets: ['192.168.43.226:39081']

重启服务

systemctl restart prometheus

配置Grafana

寻找tomcat模板

grafana导入模板

查看tomcat仪表盘