通过k8s部署dubbo微服务并接入ELK架构

需要这样一套日志收集、分析的系统：

• 收集 -- 能够采集多种来源的日志数据 (流式日志收集器)
• 传输 -- 能够稳定的把日志数据传输到中央系统 (消息队列)
• 存储 -- 可以将日志以结构化数据的形式存储起来 (搜索引擎)
• 分析 -- 支持方便的分析、检索方法，最好有GUI管理系统 (前端)
• 警告 -- 能够提供错误报告，监控机制 (监控工具)

优秀的社区开源解决方案 ---- ELK Stack

• E ----- ElasticSearch
• L ----- LogStash
• K ----- Kibana

1.传统ELK模型

缺点：

• Logstash使用Jruby语言开发，吃资源，大量部署消耗极高
• 业务程序与Logstash耦合过松，不利于业务迁移
• 日志收集与ES解耦又过紧，易打爆、丢数据
• 在容器云环境下，传统ELK模型难以完成工作

2.ELK架构图

3.制作tomcat容器的底包镜像

Tomcat官网：https://tomcat.apache.org/

(1)下载Tomcat二进制包

在运维主机(mfyxw50)上执行

Tomcat 8.5.57下载地址：https://mirror.bit.edu.cn/apache/tomcat/tomcat-8/v8.5.57/bin/apache-tomcat-8.5.57.tar.gz

下载Tomcat二进制包并保存至/opt/src目录

[root@mfyxw50 ~]# cd /opt/src
[root@mfyxw50 src]#
[root@mfyxw50 src]# wget https://mirror.bit.edu.cn/apache/tomcat/tomcat-8/v8.5.57/bin/apache-tomcat-8.5.57.tar.gz
--2020-08-17 22:52:23--  https://mirror.bit.edu.cn/apache/tomcat/tomcat-8/v8.5.57/bin/apache-tomcat-8.5.57.tar.gz
Resolving mirror.bit.edu.cn (mirror.bit.edu.cn)... 219.143.204.117, 202.204.80.77, 2001:da8:204:1205::22
Connecting to mirror.bit.edu.cn (mirror.bit.edu.cn)|219.143.204.117|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10379806 (9.9M) [application/octet-stream]
Saving to: ‘apache-tomcat-8.5.57.tar.gz’

100%[==================================================================================================>] 10,379,806   153KB/s   in 70s    

2020-08-17 22:53:33 (144 KB/s) - ‘apache-tomcat-8.5.57.tar.gz’ saved [10379806/10379806]

创建存放Tomcat的目录并解压至此目录

[root@mfyxw50 ~]# mkdir -p /data/dockerfile/tomcat8
[root@mfyxw50 ~]# tar -xf /opt/src/apache-tomcat-8.5.57.tar.gz -C /data/dockerfile/tomcat8

(2)简单配置tomcat

关闭ajp端口

在/data/dockerfile/tomcat8/apache-tomcat-8.5.57/conf/server.xml找到AJP的，添加上注释即可关闭AJP端口

<!-- <Connector protocol="AJP/1.3"
               address="::1"
               port="8009"
               redirectPort="8443" />
-- >

(3)配置日志

删除3manager，4host-manager的handlers并注释相关的内容

文件路径如下：/data/dockerfile/tomcat8/apache-tomcat-8.5.57/conf/logging.properties

修改好后如下所示

handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler,  java.util.logging.ConsoleHandler

注释3manager和4host-manager的日志配置内容

#3manager.org.apache.juli.AsyncFileHandler.level = FINE
#3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
#3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
#3manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8

#4host-manager.org.apache.juli.AsyncFileHandler.level = FINE
#4host-manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
#4host-manager.org.apache.juli.AsyncFileHandler.prefix = host-manager.
#4host-manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8

将其它的日志的等级修改为INFO

1catalina.org.apache.juli.AsyncFileHandler.level = INFO
2localhost.org.apache.juli.AsyncFileHandler.level = INFO
java.util.logging.ConsoleHandler.level = INFO

(4)准备Dockerfile文件

dockerfile文件内容如下：

[root@mfyxw50 ~]# cat > /data/dockerfile/tomcat8/Dockerfile << EOF
From harbor.od.com/base/jre8:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo 'Asia/Shanghai' >/etc/timezone
ENV CATALINA_HOME /opt/tomcat
ENV LANG zh_CN.UTF-8
ADD apache-tomcat-8.5.57/ /opt/tomcat
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/jmx_javaagent-0.3.1.jar
WORKDIR /opt/tomcat
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]
EOF

config.yml文件内容如下：

[root@mfyxw50 ~]# cat > /data/dockerfile/tomcat8/config.yml << EOF
---
rules:
  - pattern: '.*'
EOF

下载jmx_javaagent-0.3.1.jar文件

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O /data/dockerfile/tomcat8/jmx_javaagent-0.3.1.jar

entrypoint.sh文件内容如下：

[root@mfyxw50 ~]# cat > /data/dockerfile/tomcat8/entrypoint.sh << EOF
#!/bin/bash
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=\$(hostname -i):\${M_PORT:-"12346"}:/opt/prom/config.yml"
C_OPTS=\${C_OPTS}
MIN_HEAP=\${MIN_HEAP:-"128m"}
MAX_HEAP=\${MAX_HEAP:-"128m"}
JAVA_OPTS=\${JAVA_OPTS:-"-Xmn384m -Xss256k -Duser.timezone=GMT+08  -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram  -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8"}
CATALINA_OPTS="\${CATALINA_OPTS}"
JAVA_OPTS="\${M_OPTS} \${C_OPTS} -Xms\${MIN_HEAP} -Xmx\${MAX_HEAP} \${JAVA_OPTS}"
sed -i -e "1a\JAVA_OPTS=\"\$JAVA_OPTS\"" -e "1a\CATALINA_OPTS=\"\$CATALINA_OPTS\"" /opt/tomcat/bin/catalina.sh

cd /opt/tomcat && /opt/tomcat/bin/catalina.sh run 2>&1 >> /opt/tomcat/logs/stdout.log
EOF

给entrypoint.sh文件加可执行文件

[root@mfyxw50 ~]# ls -l /data/dockerfile/tomcat8/entrypoint.sh
-rw-r--r-- 1 root root 827 Aug 17 23:47 /data/dockerfile/tomcat8/entrypoint.sh
[root@mfyxw50 ~]#
[root@mfyxw50 ~]# chmod u+x /data/dockerfile/tomcat8/entrypoint.sh
[root@mfyxw50 ~]#
[root@mfyxw50 ~]# ls -l /data/dockerfile/tomcat8/entrypoint.sh
-rwxr--r-- 1 root root 827 Aug 17 23:47 /data/dockerfile/tomcat8/entrypoint.sh

(5)构建tomcat底包

[root@mfyxw50 ~]# cd /data/dockerfile/tomcat8/
[root@mfyxw50 tomcat8]#
[root@mfyxw50 tomcat8]# docker build . -t harbor.od.com/base/tomcat:v8.5.57
Sending build context to Docker daemon  10.35MB
Step 1/10 : From harbor.od.com/base/jre8:8u112
 ---> 1237758f0be9
Step 2/10 : RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo 'Asia/Shanghai' >/etc/timezone
 ---> Running in dd43a1973ae6
Removing intermediate container dd43a1973ae6
 ---> 7d376f75a369
Step 3/10 : ENV CATALINA_HOME /opt/tomcat
 ---> Running in 5d1a8302488f
Removing intermediate container 5d1a8302488f
 ---> e6d8a5069f4b
Step 4/10 : ENV LANG zh_CN.UTF-8
 ---> Running in 9ab19ad646de
Removing intermediate container 9ab19ad646de
 ---> c61931622aae
Step 5/10 : ADD apache-tomcat-8.5.57/ /opt/tomcat
 ---> 6953dffb9b11
Step 6/10 : ADD config.yml /opt/prom/config.yml
 ---> 4d67798f76f5
Step 7/10 : ADD jmx_javaagent-0.3.1.jar /opt/prom/jmx_javaagent-0.3.1.jar
 ---> 2ff30950c856
Step 8/10 : WORKDIR /opt/tomcat
 ---> Running in f0692b96c235
Removing intermediate container f0692b96c235
 ---> 00847c31b601
Step 9/10 : ADD entrypoint.sh /entrypoint.sh
 ---> 6a44a6205708
Step 10/10 : CMD ["/entrypoint.sh"]
 ---> Running in d2e6b80af0af
Removing intermediate container d2e6b80af0af
 ---> c3c4fcdbe8fd
Successfully built c3c4fcdbe8fd
Successfully tagged harbor.od.com/base/tomcat:v8.5.57

(6)将构建好的tomcat底包上传至私有仓库

[root@mfyxw50 ~]# docker login harbor.od.com
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@mfyxw50 ~]#
[root@mfyxw50 ~]# docker push harbor.od.com/base/tomcat:v8.5.57
The push refers to repository [harbor.od.com/base/tomcat]
1adedb0df456: Pushed
c456d7815bdc: Pushed
adbd684689e5: Pushed
fc2e7ea50383: Pushed
1d6b6320a33e: Pushed
c012afdfa38e: Mounted from base/jre8
30934063c5fd: Mounted from base/jre8
0378026a5ac0: Mounted from base/jre8
12ac448620a2: Mounted from base/jre8
78c3079c29e7: Mounted from base/jre8
0690f10a63a5: Mounted from base/jre8
c843b2cf4e12: Mounted from base/jre8
fddd8887b725: Mounted from base/jre8
42052a19230c: Mounted from base/jre8
8d4d1ab5ff74: Mounted from base/jre8
v8.5.57: digest: sha256:83098849296b452d1f4886f9c84db8978c3d8d16b12224f8b76f20ba79abd8d6 size: 3448

4.实战交付tomcat形式的dubbo服务消费者到K8S集群

(1)创建Tomcat的jenkins流水线

给予tomcat创建流水线

使用admin用户登录，New Item ----> Create new jobs ----> Enter an item name，输入流水线的名字，比如：tomcat-demo

下面是设置流水线的参数：

• Discard old builds

  Days to keep builds: 3

  Max # of builds to keep: 30

• This project is parameterized

  1. Add Parameter ---> String Parameter

     Name: app_name

     Default value:

     Description: project name. e.g: dubbo-demo-web

     • [√] Trim the string

  2. Add Parameter ---> String Parameter

     Name: image_name

     Default Value:

     Description :project docker image name. e.g: app/dubbo-demo-web

     • [√] Trim the string

  3. Add Parameter ---> String Parameter

     Name: git_repo

     Default Value:

     Description : project git repository. e.g: git@gitee.com