Tomcat Docker容器自动重启问题排查

1. 根据现有的docker容器重建镜像及运行

docker在构建镜像包时，就把jvm的参数预先编写好了，在遇到性能问题后，需要修改jvm参数进行调整。

docker commit命令创建新的镜像

1、运行容器

2、修改容器

docker exec -it 容器id sh

#修改配置

3、将容器保存为新镜像

docker commit 0521893d603d reg.xxx.net/preprod/red-xxx:cce7f12d-jvmparams

sh service-update.sh reg.xxx.net/preprod/red-xxx:cce7f12d-jvmparams

2、调整某个docker的内存限制

[ec2-user@ip---- ~]$ docker service update --limit-memory 1G op-m-front

op-m-front

overall progress:  out of  tasks

w5hk1y1rgqeh: running   [==================================================>]

verify: Waiting  seconds to verify that tasks are stable...

Error: No such service: op-m-front

[ec2-user@ip---- ~]$

3、dockerd: time="2019-12-09T05:29:36.169298343Z" level=error msg="fatal task error" error="task: non-zero exit (137)

Issue

If a container is no longer running, use the following command to find the status of the container:

docker container ls -a

This article explains possible reasons for the following exit code:

"task: non-zero exit (137)"

With exit code 137, you might also notice a status of Shutdown or the following failed message:

Failed 42 hours ago

Resolution

The "task: non-zero exit (137)" message is effectively the result of a kill -9 (128 + 9). This can be due to a couple possibilities (seen most often with Java applications):

The container received a docker stop, and the application didn't gracefully handle SIGTERM (kill -15) — whenever a SIGTERM has been issued, the docker daemon waits 10 seconds then issue a SIGKILL (kill -9) to guarantee the shutdown. To test whether your containerized application correctly handles SIGTERM, simply issue a docker stop against the container ID and check to see whether you get the "task: non-zero exit (137)". This is not something to test in a production environment, as you can expect at least a brief interruption of service. Best practices would be to test in a development or test Docker environment.
The application hit an OOM (out of memory) condition. With regards to OOM condition handling, review the node's kernel logs to validate whether this occurred. This would require knowing which node the failed container was running on, or proceed with checking all nodes. Run something like this on your node(s) to help you identify whether you've had a container hit an OOM condition: journalctl -k | grep -i -e memory -e oom Another option would be to inspect the (failed) container: docker inspect <container ID> Review the application's memory requirements and ensure that the container it's running in has sufficient memory. Conversely, set a limit on the container's memory to ensure that wherever it runs, it does not consume memory to the detriment of the node. If the application is Java-based, you may want to review the maximum memory configuration settings.

References

4、docker top 容器id

查看容器里的进程

[ec2-user@ip---- ~]$ docker top 2c1

UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD

ec2-user                                                               :               ?                   ::            sh entrypoint.sh

ec2-user                                                               :               ?                   ::            java -Xmx1344m -Xms1344m -Xmn448M -XX:MaxMetaspaceSize=192M -XX:MetaspaceSize=192M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/loyalty-dump

5、

启动
systemctl start docker
守护进程重启
systemctl daemon-reload
重启docker服务
systemctl restart docker / service docker restart
关闭
docker service docker stop / docker systemctl stop docker