k8s实战案例之部署Nginx+Tomcat+NFS实现动静分离
1、基于镜像分层构建及自定义镜像运行Nginx及Java服务并基于NFS实现动静分离
1.1、业务镜像设计规划
根据业务的不同，我们可以导入官方基础镜像，在官方基础镜像的基础上自定义需要用的工具和环境，然后构建成自定义出自定义基础镜像，后续再基于自定义基础镜像，来构建不同服务的基础镜像，最后基于服务的自定义基础镜像构建出对应业务镜像；最后将这些镜像上传至本地harbor仓库，然后通过k8s配置清单，将对应业务运行至k8s集群之上；
1.2、Nginx+Tomcat+NFS实现动静分离架构图
客户端通过负载均衡器的反向代理来访问k8s上的服务， nginx pod和tomcat pod 由k8s svc 资源进行关联；所有数据（静态资源和动态资源）通过存储挂载至对应pod中；nginx作为服务入口，它负责接收客户端的请求，同时响应静态资源（到存储上读取，比如js文件，css文件，图片等）；后端动态资源，由nginx将请求转发至后端tomcat server 完成（tomcat负责数据写入，比如用户的上传的图片等等）；
2、自定义centos基础镜像构建
root@k8s-master01:~/k8s-data/dockerfile/system/centos# ls
CentOS7-aliyun-Base.repo CentOS7-aliyun-epel.repo Dockerfile build-command.sh filebeat-7.12.1-x86_64.rpm
root@k8s-master01:~/k8s-data/dockerfile/system/centos# cat Dockerfile
#自定义Centos 基础镜像
FROM centos:7.9.2009
ADD filebeat-7.12.1-x86_64.rpm /tmp
# 添加阿里源
ADD CentOS7-aliyun-Base.repo CentOS7-aliyun-epel.repo /etc/yum.repos.d/
# 自定义安装工具和环境
RUN yum makecache &&yum install -y /tmp/filebeat-7.12.1-x86_64.rpm vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop && rm -rf /etc/localtime /tmp/filebeat-7.12.1-x86_64.rpm && ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && useradd nginx -u 2088
root@k8s-master01:~/k8s-data/dockerfile/system/centos# cat build-command.sh
#!/bin/bash
#docker build -t harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009 .
#docker push harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009
/usr/local/bin/nerdctl build -t harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009 .
/usr/local/bin/nerdctl push harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009
root@k8s-master01:~/k8s-data/dockerfile/system/centos#
2.1、构建自定义centos基础镜像
2.2、验证自定义centos基础镜像
在harbor上验证镜像是否正常上传？
运行镜像为容器，验证对应镜像是否有我们添加的工具和环境？
3、基于自定义centos基础镜像构建nginx镜像
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base# ls
Dockerfile build-command.sh nginx-1.22.0.tar.gz
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base# cat Dockerfile
#Nginx Base Image
# 导入自定义centos基础镜像
FROM harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009
# 安装编译环境
RUN yum install -y vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop
# 添加nginx源码至/usr/local/src/
ADD nginx-1.22.0.tar.gz /usr/local/src/
# 编译nginx
RUN cd /usr/local/src/nginx-1.22.0 && ./configure && make && make install && ln -sv /usr/local/nginx/sbin/nginx /usr/sbin/nginx &&rm -rf /usr/local/src/nginx-1.22.0.tar.gz
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base# cat build-command.sh
#!/bin/bash
#docker build -t harbor.ik8s.cc/pub-images/nginx-base:v1.18.0 .
#docker push harbor.ik8s.cc/pub-images/nginx-base:v1.18.0
nerdctl build -t harbor.ik8s.cc/pub-images/nginx-base:v1.22.0 .
nerdctl push harbor.ik8s.cc/pub-images/nginx-base:v1.22.0
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base#
3.1、构建自定义nginx基础镜像
3.2、验证自定义nginx基础镜像
验证nginx基础镜像是否上传至harbor？
把nginx基础镜像运行为容器，看看nginx是否正常安装？
能够将nginx基础镜像运行为容器，并在容器内部启动nginx，表示nginx基础镜像就构建好了；
3.3、构建自定义nginx业务镜像
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# ls
Dockerfile app1.tar.gz build-command.sh index.html nginx.conf webapp
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# cat Dockerfile
#Nginx 1.22.0
# 导入nginx基础镜像
FROM harbor.ik8s.cc/pub-images/nginx-base:v1.22.0
# 添加nginx配置文件
ADD nginx.conf /usr/local/nginx/conf/nginx.conf
# 添加业务代码
ADD app1.tar.gz /usr/local/nginx/html/webapp/
ADD index.html /usr/local/nginx/html/index.html
# 创建静态资源挂载路径
RUN mkdir -p /usr/local/nginx/html/webapp/static /usr/local/nginx/html/webapp/images
# 暴露端口
EXPOSE 80 443
# 运行nginx
CMD ["nginx"]
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# cat nginx.conf
user nginx nginx;
worker_processes auto;
daemon off;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
upstream tomcat_webserver {
server magedu-tomcat-app1-service.magedu:80;
}
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
location /webapp {
root html;
index index.html index.htm;
}
location /app1 {
proxy_pass http://tomcat_webserver;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# cat build-command.sh
#!/bin/bash
TAG=$1
#docker build -t harbor.ik8s.cc/magedu/nginx-web1:${TAG} .
#echo "镜像构建完成，即将上传到harbor"
#sleep 1
#docker push harbor.ik8s.cc/magedu/nginx-web1:${TAG}
#echo "镜像上传到harbor完成"
nerdctl build -t harbor.ik8s.cc/magedu/nginx-web1:${TAG} .
nerdctl push harbor.ik8s.cc/magedu/nginx-web1:${TAG}
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx#
上述Dockerfile中主要基于nginx基础镜像添加业务代码，添加配置，以及定义运行nginx和暴露服务端口；
3.4、验证自定义nginx业务镜像
验证nginx业务镜像是否上传至harbor？
运行nginx业务镜像为容器，看看对应业务是否能够正常访问？
这里提示找不到magedu-tomcat-app1-service.magedu:80这个upstream ，这是因为我们在配置文件中写死了nginx调用后端tomcat在k8s中 svc的地址；所以该镜像只能运行在k8s环境中，并且在运行该镜像前对应k8s环境中，tomcat的svc必须存在；
4、基于自定义centos基础镜像构建tomcat镜像
4.1、基于自定义centos基础镜像构建jdk基础镜像
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# ll
total 190464
drwxr-xr-x 2 root root 4096 Jun 4 04:04 ./
drwxr-xr-x 6 root root 4096 Aug 9 2022 ../
-rw-r--r-- 1 root root 389 Jun 4 04:04 Dockerfile
-rw-r--r-- 1 root root 259 Jun 4 04:02 build-command.sh
-rw-r--r-- 1 root root 195013152 Jun 22 2021 jdk-8u212-linux-x64.tar.gz
-rw-r--r-- 1 root root 2105 Jun 22 2021 profile
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# cat Dockerfile
#JDK Base Image
# 导入自定义centos基础镜像
FROM harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009
# 安装jdk环境
ADD jdk-8u212-linux-x64.tar.gz /usr/local/src/
RUN ln -sv /usr/local/src/jdk1.8.0_212 /usr/local/jdk
ADD profile /etc/profile
ENV JAVA_HOME /usr/local/jdk
ENV JRE_HOME $JAVA_HOME/jre
ENV CLASSPATH $JAVA_HOME/lib/:$JRE_HOME/lib/
ENV PATH $PATH:$JAVA_HOME/bin
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# cat build-command.sh
#!/bin/bash
#docker build -t harbor.ik8s.cc/pub-images/jdk-base:v8.212 .
#sleep 1
#docker push harbor.ik8s.cc/pub-images/jdk-base:v8.212
nerdctl build -t harbor.ik8s.cc/pub-images/jdk-base:v8.212 .
nerdctl push harbor.ik8s.cc/pub-images/jdk-base:v8.212
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212#
4.1.1、构建自定义jdk基础镜像
4.1.2、验证自定义jdk基础镜像
验证jdk基础镜像是否上传至harbor?
运行jdk基础镜像为容器，看看jdk环境是否安装？
能够正常在容器内部执行java 命令表示jdk镜像构建没有问题；
4.2、基于自定义jdk镜像构建tomcat基础镜像
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# ll
total 9508
drwxr-xr-x 2 root root 4096 Jun 4 04:23 ./
drwxr-xr-x 6 root root 4096 Aug 9 2022 ../
-rw-r--r-- 1 root root 390 Jun 4 04:22 Dockerfile
-rw-r--r-- 1 root root 9717059 Jun 22 2021 apache-tomcat-8.5.43.tar.gz
-rw-r--r-- 1 root root 275 Jun 4 04:23 build-command.sh
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# cat Dockerfile
#Tomcat 8.5.43基础镜像
# 导入自定义jdk镜像
FROM harbor.ik8s.cc/pub-images/jdk-base:v8.212
# 创建tomcat安装目录、数据目录和日志目录
RUN mkdir /apps /data/tomcat/webapps /data/tomcat/logs -pv
# 安装tomcat
ADD apache-tomcat-8.5.43.tar.gz /apps
RUN useradd tomcat -u 2050 && ln -sv /apps/apache-tomcat-8.5.43 /apps/tomcat && chown -R tomcat.tomcat /apps /data -R
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# cat build-command.sh
#!/bin/bash
#docker build -t harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43 .
#sleep 3
#docker push harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43
nerdctl build -t harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43 .
nerdctl push harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43#
4.2.1、构建自定义tomcat基础镜像
4.2.2、验证自定义tomcat基础镜像
验证自定义tomcat基础镜像是否上传至harbor？
将自定义tomcat镜像运行为容器，看看tomcat是否可正常访问呢？
root@k8s-node01:~# nerdctl run -it --rm -p 8080:8080 harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43 /bin/bash
WARN[0000] skipping verifying HTTPS certs for "harbor.ik8s.cc"
harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:ba362a6c099d965ffae43aae04e3ecb15e86e39d9d88259f4be72378b75bd1e5: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:1d5e5f5e1dcb8ce1c7f7d6ce9fcc967275309c14baaaddbf3aa86de26054d1d5: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fa5fdb4dc02a5e79b212be196324b9936efbc850390f86283498a0e01b344ec3: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:00bab63d153828acf58242f1781bd40769cc8b69659f37a2a49918ff3bfca68c: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6ddb864b9c4e53f3079ec4839ea3bace75b0d5d3daeae0ae910c171646bc4f96: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.4 s total: 18.6 M (7.7 MiB/s)
[root@f5752bba588f /]# ll /apps/
total 4
drwxr-xr-x 1 tomcat tomcat 4096 Jun 4 12:25 apache-tomcat-8.5.43
lrwxrwxrwx 1 tomcat tomcat 26 Jun 4 12:25 tomcat -> /apps/apache-tomcat-8.5.43
[root@f5752bba588f /]# cd /apps/tomcat/bin/
[root@f5752bba588f bin]# ls
bootstrap.jar catalina.sh commons-daemon-native.tar.gz configtest.sh digest.sh shutdown.bat startup.sh tool-wrapper.bat version.sh
catalina-tasks.xml ciphers.bat commons-daemon.jar daemon.sh setclasspath.bat shutdown.sh tomcat-juli.jar tool-wrapper.sh
catalina.bat ciphers.sh configtest.bat digest.bat setclasspath.sh startup.bat tomcat-native.tar.gz version.bat
[root@f5752bba588f bin]# ./catalina.sh run
Using CATALINA_BASE: /apps/tomcat
Using CATALINA_HOME: /apps/tomcat
Using CATALINA_TMPDIR: /apps/tomcat/temp
Using JRE_HOME: /usr/local/jdk/jre
Using CLASSPATH: /apps/tomcat/bin/bootstrap.jar:/apps/tomcat/bin/tomcat-juli.jar
04-Jun-2023 12:40:29.845 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.43
04-Jun-2023 12:40:29.851 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Jul 4 2019 20:53:15 UTC
04-Jun-2023 12:40:29.851 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.5.43.0
04-Jun-2023 12:40:29.851 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.15.0-72-generic
04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/local/src/jdk1.8.0_212/jre
04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_212-b10
04-Jun-2023 12:40:29.853 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
04-Jun-2023 12:40:29.853 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /apps/apache-tomcat-8.5.43
04-Jun-2023 12:40:29.853 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /apps/apache-tomcat-8.5.43
04-Jun-2023 12:40:29.854 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/apps/tomcat/conf/logging.properties
04-Jun-2023 12:40:29.854 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
04-Jun-2023 12:40:29.855 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
04-Jun-2023 12:40:29.855 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
04-Jun-2023 12:40:29.855 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
04-Jun-2023 12:40:29.856 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
04-Jun-2023 12:40:29.856 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/apps/tomcat
04-Jun-2023 12:40:29.856 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/apps/tomcat
04-Jun-2023 12:40:29.857 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/apps/tomcat/temp
04-Jun-2023 12:40:29.857 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
04-Jun-2023 12:40:30.166 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
04-Jun-2023 12:40:30.193 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
04-Jun-2023 12:40:30.232 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"]
04-Jun-2023 12:40:30.236 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
04-Jun-2023 12:40:30.237 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1262 ms
04-Jun-2023 12:40:30.303 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
04-Jun-2023 12:40:30.303 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.43
04-Jun-2023 12:40:30.324 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/docs]
04-Jun-2023 12:40:30.918 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/docs] has finished in [593] ms
04-Jun-2023 12:40:30.919 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/examples]
04-Jun-2023 12:40:31.524 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/examples] has finished in [605] ms
04-Jun-2023 12:40:31.524 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/ROOT]
04-Jun-2023 12:40:31.540 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/ROOT] has finished in [16] ms
04-Jun-2023 12:40:31.540 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/manager]
04-Jun-2023 12:40:31.565 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/manager] has finished in [25] ms
04-Jun-2023 12:40:31.565 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/host-manager]
04-Jun-2023 12:40:31.597 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/host-manager] has finished in [32] ms
04-Jun-2023 12:40:31.600 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
04-Jun-2023 12:40:31.608 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
04-Jun-2023 12:40:31.611 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 1373 ms
访问tomcat
能够正常访问tomcat,说明tomcat基础镜像就构建的没有问题；
4.3、基于自定义tomcat镜像构建tomcat业务镜像
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# ll
total 23588
drwxr-xr-x 3 root root 4096 Jun 4 04:58 ./
drwxr-xr-x 11 root root 4096 Aug 9 2022 ../
-rw-r--r-- 1 root root 603 Jun 4 04:57 Dockerfile
-rw-r--r-- 1 root root 144 May 11 06:26 app1.tar.gz
-rwxr-xr-x 1 root root 261 Jun 4 04:58 build-command.sh*
-rwxr-xr-x 1 root root 23611 Jun 22 2021 catalina.sh*
-rw-r--r-- 1 root root 24086235 Jun 22 2021 filebeat-7.5.1-x86_64.rpm
-rw-r--r-- 1 root root 667 Oct 24 2021 filebeat.yml
drwxr-xr-x 2 root root 4096 May 11 06:26 myapp/
-rwxr-xr-x 1 root root 372 Jan 22 2022 run_tomcat.sh*
-rw-r--r-- 1 root root 6462 Oct 10 2021 server.xml
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# cat Dockerfile
#tomcat web1
# 导入自定义tomcat镜像
FROM harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43
# 添加启动脚本和配置文件
ADD catalina.sh /apps/tomcat/bin/catalina.sh
ADD server.xml /apps/tomcat/conf/server.xml
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
# 添加业务代码
ADD app1.tar.gz /data/tomcat/webapps/app1/
#ADD filebeat.yml /etc/filebeat/filebeat.yml
RUN chown -R nginx.nginx /data/ /apps/
#ADD filebeat-7.5.1-x86_64.rpm /tmp/
#RUN cd /tmp && yum localinstall -y filebeat-7.5.1-amd64.deb
# 暴露端口
EXPOSE 8080 8443
# 启动tomcat
CMD ["/apps/tomcat/bin/run_tomcat.sh"]
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# cat build-command.sh
#!/bin/bash
TAG=$1
#docker build -t harbor.ik8s.cc/magedu/tomcat-app1:${TAG} .
#sleep 3
#docker push harbor.ik8s.cc/magedu/tomcat-app1:${TAG}
nerdctl build -t harbor.ik8s.cc/magedu/tomcat-app1:${TAG} .
nerdctl push harbor.ik8s.cc/magedu/tomcat-app1:${TAG}
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1#
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# cat run_tomcat.sh
#!/bin/bash
#echo "nameserver 223.6.6.6" > /etc/resolv.conf
#echo "192.168.7.248 k8s-vip.example.com" >> /etc/hosts
#/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat &
su - nginx -c "/apps/tomcat/bin/catalina.sh start"
tail -f /etc/hosts
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1#
4.3.1、构建自定义tomcat业务镜像
4.3.2、验证自定义tomcat业务镜像
验证自定义tomcat业务镜像是否上传至harbor？
将自定义tomcat业务镜像运行为容器，看看对应业务是否正常访问？
访问业务
能够正常访问app1说明业务容器的镜像构建没有问题；
5、在k8s环境中运行tomcat
root@k8s-master01:~/k8s-data/yaml/namespaces# ls
magedu-ns.yaml
root@k8s-master01:~/k8s-data/yaml/namespaces# cat magedu-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: magedu
root@k8s-master01:~/k8s-data/yaml/namespaces# cd ../magedu/tomcat-app1/
root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# ll
total 16
drwxr-xr-x 2 root root 4096 Jun 4 06:16 ./
drwxr-xr-x 12 root root 4096 Aug 9 2022 ../
-rw-r--r-- 1 root root 596 Jun 22 2021 hpa.yaml
-rw-r--r-- 1 root root 1849 Jun 4 05:18 tomcat-app1.yaml
root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# cat hpa.yaml
#apiVersion: autoscaling/v2beta1
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
namespace: magedu
name: magedu-tomcat-app1-podautoscaler
labels:
app: magedu-tomcat-app1
version: v2beta1
spec:
scaleTargetRef:
apiVersion: apps/v1
#apiVersion: extensions/v1beta1
kind: Deployment
name: magedu-tomcat-app1-deployment
minReplicas: 2
maxReplicas: 20
targetCPUUtilizationPercentage: 60
#metrics:
#- type: Resource
# resource:
# name: cpu
# targetAverageUtilization: 60
#- type: Resource
# resource:
# name: memory
root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# cat tomcat-app1.yaml
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:
labels:
app: magedu-tomcat-app1-deployment-label
name: magedu-tomcat-app1-deployment
namespace: magedu
spec:
replicas: 1
selector:
matchLabels:
app: magedu-tomcat-app1-selector
template:
metadata:
labels:
app: magedu-tomcat-app1-selector
spec:
containers:
- name: magedu-tomcat-app1-container
image: harbor.ik8s.cc/magedu/tomcat-app1:v1
#command: ["/apps/tomcat/bin/run_tomcat.sh"]
imagePullPolicy: IfNotPresent
#imagePullPolicy: Always
ports:
- containerPort: 8080
protocol: TCP
name: http
env:
- name: "password"
value: "123456"
- name: "age"
value: "18"
#resources:
# limits:
# cpu: 1
# memory: "512Mi"
# requests:
# cpu: 500m
# memory: "512Mi"
volumeMounts:
- name: magedu-images
mountPath: /usr/local/nginx/html/webapp/images
readOnly: false
- name: magedu-static
mountPath: /usr/local/nginx/html/webapp/static
readOnly: false
volumes:
- name: magedu-images
nfs:
server: 192.168.0.42
path: /data/k8sdata/magedu/images
- name: magedu-static
nfs:
server: 192.168.0.42
path: /data/k8sdata/magedu/static
# nodeSelector:
# project: magedu
# app: tomcat
---
kind: Service
apiVersion: v1
metadata:
labels:
app: magedu-tomcat-app1-service-label
name: magedu-tomcat-app1-service
namespace: magedu
spec:
type: NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
nodePort: 30092
selector:
app: magedu-tomcat-app1-selector
root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1#
5.1、准备后端存储相关目录
root@harbor:~# mkdir -pv /data/k8sdata/magedu/images
mkdir: created directory '/data/k8sdata/magedu'
mkdir: created directory '/data/k8sdata/magedu/images'
root@harbor:~# mkdir -pv /data/k8sdata/magedu/static
mkdir: created directory '/data/k8sdata/magedu/static'
root@harbor:~# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
#
/data/k8sdata/kuboard *(rw,no_root_squash)
/data/volumes *(rw,no_root_squash)
/pod-vol *(rw,no_root_squash)
/data/k8sdata/myserver *(rw,no_root_squash)
/data/k8sdata/mysite *(rw,no_root_squash)
/data/k8sdata/magedu/images *(rw,no_root_squash)
/data/k8sdata/magedu/static *(rw,no_root_squash)
root@harbor:~# exportfs -av
exportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/kuboard".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/volumes".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [3]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/pod-vol".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [4]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/myserver".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [5]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/mysite".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [7]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/images".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exportfs: /etc/exports [8]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/static".
Assuming default behaviour ('no_subtree_check').
NOTE: this default has changed since nfs-utils version 1.0.x
exporting *:/data/k8sdata/magedu/static
exporting *:/data/k8sdata/magedu/images
exporting *:/data/k8sdata/mysite
exporting *:/data/k8sdata/myserver
exporting *:/pod-vol
exporting *:/data/volumes
exporting *:/data/k8sdata/kuboard
root@harbor:~#
5.2、将tomcat业务部署至k8s
root@k8s-master01:~/k8s-data/yaml# cd namespaces/
root@k8s-master01:~/k8s-data/yaml/namespaces# ls
magedu-ns.yaml
root@k8s-master01:~/k8s-data/yaml/namespaces# kubectl apply -f magedu-ns.yaml
namespace/magedu created
root@k8s-master01:~/k8s-data/yaml/namespaces# ls
magedu-ns.yaml
root@k8s-master01:~/k8s-data/yaml/namespaces# cd ../magedu/tomcat-app1/
root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# ls
hpa.yaml tomcat-app1.yaml
root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# kubectl apply -f .
horizontalpodautoscaler.autoscaling/magedu-tomcat-app1-podautoscaler created
deployment.apps/magedu-tomcat-app1-deployment created
service/magedu-tomcat-app1-service created
root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1#
5.3、验证tomcat pod是否正常running?服务是否可正常访问？
root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# kubectl get pods -n magedu
NAME READY STATUS RESTARTS AGE
magedu-tomcat-app1-deployment-7754c8549c-prglk 1/1 Running 0 2m2s
magedu-tomcat-app1-deployment-7754c8549c-xmg9l 1/1 Running 0 2m17s
root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# kubectl get svc -n magedu
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
magedu-tomcat-app1-service NodePort 10.100.129.23 <none> 80:30092/TCP 2m23s
root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1#
访问k8s集群节点的30092端口，看看对应tomcat服务是否能够正常访问？
能够通过k8s集群节点访问tomcat服务，说明tomcat服务已经正常部署至k8s环境；
5.4、更改tomcat service 类型为无头服务，重新apply配置清单，让后端tomcat服务只能在k8s内部环境访问
6、在k8s环境中运行nginx，实现nginx+tomcat动静分离
root@k8s-master01:~/k8s-data/yaml/magedu/nginx# cat nginx.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: magedu-nginx-deployment-label
name: magedu-nginx-deployment
namespace: magedu
spec:
replicas: 1
selector:
matchLabels:
app: magedu-nginx-selector
template:
metadata:
labels:
app: magedu-nginx-selector
spec:
containers:
- name: magedu-nginx-container
image: harbor.ik8s.cc/magedu/nginx-web1:v1
#command: ["/apps/tomcat/bin/run_tomcat.sh"]
#imagePullPolicy: IfNotPresent
imagePullPolicy: Always
ports:
- containerPort: 80
protocol: TCP
name: http
- containerPort: 443
protocol: TCP
name: https
env:
- name: "password"
value: "123456"
- name: "age"
value: "20"
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 500m
memory: 256Mi
volumeMounts:
- name: magedu-images
mountPath: /usr/local/nginx/html/webapp/images
readOnly: false
- name: magedu-static
mountPath: /usr/local/nginx/html/webapp/static
readOnly: false
volumes:
- name: magedu-images
nfs:
server: 192.168.0.42
path: /data/k8sdata/magedu/images
- name: magedu-static
nfs:
server: 192.168.0.42
path: /data/k8sdata/magedu/static
#nodeSelector:
# group: magedu
---
kind: Service
apiVersion: v1
metadata:
labels:
app: magedu-nginx-service-label
name: magedu-nginx-service
namespace: magedu
spec:
type: NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
nodePort: 30090
- name: https
port: 443
protocol: TCP
targetPort: 443
nodePort: 30091
selector:
app: magedu-nginx-selector
root@k8s-master01:~/k8s-data/yaml/magedu/nginx#
6.1、部署nginx业务部署至k8s
root@k8s-master01:~/k8s-data/yaml/magedu/nginx# kubectl apply -f .
deployment.apps/magedu-nginx-deployment created
service/magedu-nginx-service created
root@k8s-master01:~/k8s-data/yaml/magedu/nginx# kubectl get pod -n magedu
NAME READY STATUS RESTARTS AGE
magedu-nginx-deployment-5589bbf4bc-6gd2w 1/1 Running 0 14s
magedu-tomcat-app1-deployment-7754c8549c-c7rtb 1/1 Running 0 8m7s
magedu-tomcat-app1-deployment-7754c8549c-prglk 1/1 Running 0 19m
root@k8s-master01:~/k8s-data/yaml/magedu/nginx# kubectl get svc -n magedu
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
magedu-nginx-service NodePort 10.100.94.118 <none> 80:30090/TCP,443:30091/TCP 24s
magedu-tomcat-app1-service ClusterIP 10.100.129.23 <none> 80/TCP 19m
root@k8s-master01:~/k8s-data/yaml/magedu/nginx#
6.2、验证nginx 服务是否正常可访问？
能够通过访问k8s集群节点的30090正常访问到nginx，说明nginx服务已经正常部署至k8s环境；
6.3、通过nginx 访问tomcat服务，看看tomcat是否能够被nginx代理？
能够通过nginx访问后端tomcat服务，说明nginx能够正常代理tomcat服务；
7、在负载均衡器上代理nginx服务
root@k8s-ha01:~# cat /etc/haproxy/haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
listen k8s_apiserver_6443
bind 192.168.0.111:6443
mode tcp
#balance leastconn
server k8s-master01 192.168.0.31:6443 check inter 2000 fall 3 rise 5
server k8s-master02 192.168.0.32:6443 check inter 2000 fall 3 rise 5
server k8s-master03 192.168.0.33:6443 check inter 2000 fall 3 rise 5
listen nginx-svc-80
bind 192.168.0.111:80
mode tcp
server k8s-node01 192.168.0.34:30090 check inter 2000 fall 3 rise 5
server k8s-node02 192.168.0.35:30090 check inter 2000 fall 3 rise 5
server k8s-node03 192.168.0.36:30090 check inter 2000 fall 3 rise 5
root@k8s-ha01:~# systemctl restart haproxy
root@k8s-ha01:~#
8、验证通过vip访问nginx服务，看看是否能够正常访问？
9、在存储服务器上上传静态资源，看看nginx是正常读取静态资源？
root@k8s-master01:~/ubuntu/html# scp -rp * 192.168.0.42:/data/k8sdata/magedu/images/
root@192.168.0.42's password:
1.jpg 100% 40KB 8.6MB/s 00:00
index.html 100% 277 282.5KB/s 00:00
root@k8s-master01:~/ubuntu/html#
验证资源是否正常上传至存储对应目录？
root@harbor:~# ll /data/k8sdata/magedu/images/
total 16
drwxr-xr-x 3 root root 4096 Jun 4 07:56 ./
drwxr-xr-x 4 root root 4096 Jun 4 06:32 ../
drwxr-xr-x 2 root root 4096 May 31 18:37 images/
-rw-r--r-- 1 root root 277 Aug 5 2022 index.html
root@harbor:~#
访问nginx服务，看看对应资源是否能够被读取？
k8s实战案例之部署Nginx+Tomcat+NFS实现动静分离的更多相关文章
- nginx+tomcat负载均衡+动静分离+redis集中管理session
1.服务器A安装ng,服务器B.C安装tomcat: 2.服务器A建立/data/www目录,用于发布静态文件: 3.ng无动静分离配置: user root root; worker_process ...
- 自动部署Nginx和nfs并架设Nginx集群脚本
本人经过多次尝试,简单完成了自动部署Nginx和nfs脚本,并且能够自动部署web反向代理集群,下面详细的阐述一下本人的思路.(以下脚本本人处于初学阶段,写的并不是很完善,所以需要后期进行整理和修正, ...
- 使用docker部署nginx+tomcat架构(3)：使用docker-compose简化部署操作
经历了之前的两篇,我们已经大体上了解了docker部署容器的过程. 使用docker部署nginx+tomcat架构 使用docker部署nginx+tomcat架构():访问mysql数据库 不过, ...
- 使用docker部署nginx+tomcat架构(2)：访问mysql数据库
上一篇完成了通过docker部署nginx+tomcat的基础软件架构,但是距离一个真正可用的软件架构还差得很远.其中最重要的一点是缺少数据库这个角色的存在,那么本篇就来完善这一点. 废话少说,直接进 ...
- Nginx +Tomcat 实现动静态分离(转)
Nginx +Tomcat 实现动静态分离 动静态分离就是Nginx处理客户端的请求的静态页面(html页面)或者图片,Tomcat处理客户端请求的动态页面(jsp页面),因为Nginx处理的静态页面 ...
- 19.Tomcat多实例部署及负载均衡、动静分离
Tomcat多实例部署及负载均衡.动静分离 目录 Tomcat多实例部署及负载均衡.动静分离 Tomcat多实例部署 安装jdk 设置jdk环境变量 安装tomcat 配置 tomcat 环境变量 修 ...
- nginx+tomcat实现动静态分离
===============Tomcat 概述: Tomcat是Apache 软件基金会(Apache Software Foundation)的Jakarta 项目中的一个核心项目,由Apache ...
- （总结）Nginx与Apache、Tomcat、Resin动静分离核心配置
PS:近来有几个刚使用nginx的新童鞋老问我,nginx+fastcgi不够稳定,偶尔出现502错误,怎么解决?本人使用nginx也有3年多了,也认为php-fpm模块不够稳定,在访问量不大的时候没 ...
- Nginx 和 IIS 实现动静分离
前段时间,搞Nginx+IIS的负载均衡,想了解的朋友,可以看这篇文章:<nginx 和 IIS 实现负载均衡>,然后也就顺便研究了Nginx + IIS 实现动静分离.所以,一起总结出来 ...
- Nginx 和 IIS 实现动静分离【转载】
前段时间,搞Nginx+IIS的负载均衡,想了解的朋友,可以看这篇文章:<nginx 和 IIS 实现负载均衡>,然后也就顺便研究了Nginx + IIS 实现动静分离.所以,一起总结出来 ...
随机推荐
- 写书写到一半，强迫症发作跑去给HotChocolate修bug
前言 这是写作<C#与.NET6 开发从入门到实践>时的小故事,作为本书正式上市的宣传,在此分享给大家. 正文 .NET目前有两个比较成熟的GraphQL框架,其中一个是HotChocol ...
- 鸿蒙开发学习笔记-UIAbility-Router页面跳转接口源码分析
在鸿蒙开发中,UIAbility的跳转使用 router 方法. 在使用的时候需导入 import router from '@ohos.router'; 该方法接口成员如下: 1.interface ...
- [数据库/Java SE]MySQL驱动包(mysql-connector-java.jar)问题[com.mysql.jdbc.Driver/org.gjt.mm.mysql.Driver/com.mysql.cj.jdbc.Driver]
MySQL的驱动JAR包----mysql-connector-java.jar,不同版本,其JBDC驱动类Driver的路径均有可能变化. 日后使用时,可根据本文的思路,有依据地进行检查(而不是随便 ...
- vim使用教程（最简单最全面的教程）
本教程是vimtutor的原版内容.这是我读过最简单最全面的vim教程. 偶仅仅是优秀内容的搬运工,放在此处便于查阅,方便学习,版权归原作者所有. 注意:每一节的命令操作将会更改本文.推荐您复制本文的 ...
- Disruptor-简单使用
前言 Disruptor是一个高性能的无锁并发框架,其主要应用场景是在高并发.低延迟的系统中,如金融领域的交易系统,游戏服务器等.其优点就是非常快,号称能支撑每秒600万订单.需要注意的是,Disru ...
- day47:Bootstrap
什么是Bootstrap? Bootstrap是一个开源框架,是对html\css\js\jquery等的封装,用法,复制黏贴一把梭. 关于Bootstrap的一些常用网址 网址: https://w ...
- 基于DotNetCoreNPOI封装特性通用导出excel
基于DotNetCoreNPOI封装特性通用导出excel 目前根据项目中的要求,支持列名定义,列索引排序,行合并单元格,EXCEL单元格的格式也是随着数据的类型做对应的调整. 效果图: 调用方式 可 ...
- java无效发源版本xx
这三个地方统一一下 就可以解决了
- [C++提高编程] 3.4 案例-评委打分
文章目录 3.4 案例-评委打分 3.4.1 案例描述 3.4.2 实现步骤 3.4 案例-评委打分 3.4.1 案例描述 有5名选手:选手ABCDE,10个评委分别对每一名选手打分,去除最高分,去除 ...
- Grafana 系列-统一展示-3-Prometheus 仪表板
系列文章 Grafana 系列文章 知识储备 Prometheus Template Variables 你可以使用变量来代替硬编码的细节,如 server.app 和 pod_name 在 metr ...