从容器里dump java堆实验探索(原创)

目标：从docker容器里dump java堆

模拟程序

占用空间500M,

设置启动JVM参数

docker启动命令

（PS：经过测试，至少要650M才能启动容器）

方式1: 通过docker exec

先通过 docker exec $containerid ps x 获取进程号

执行 docker exec

此命令实际就是进入docker容器里执行/jdk/bin/jmap dump ，导出的文件也是存放在容器里

经过多次实验，基本上每次都触发容器killed

方式2,: 通过nsenter

参考文章 https://github.com/jpetazzo/nsenter

这里有句重点是”evades resource limitations” ，躲避资源限制

我们使用nsenter试验一下

使用docker-enter  (docker-enter脚本 调用了nsenter)

多次测试，都没有发生docker killed现象

结论: 通过nsenter 方式去 dump java 堆，能比较高成功率， 但仅限于此实验场景，不代表其他线上场景一定能成功

PS: jdk10 新特性里虽然增加了更好的docker支持，但没有提及过jmap 有更好的dump方式

https://bugs.openjdk.java.net/browse/JDK-8146115

To correct these shortcomings and make this support more robust, here's a list of the current cgroup subsystems that we be examined in order to update the internal VM and core library configuration.

Number of CPUs 
----------------------- 
Use a combination of number_of_cpus() and
cpu_sets() in order to determine how many processors are available to the
process and adjust the JVMs os::active_processor_count appropriately. The
number_of_cpus() will be calculated based on the cpu_quota() and cpu_period()
using this formula: number_of_cpus() = cpu_quota() / cpu_period(). If
cpu_shares has been setup for the container, the number_of_cpus() will be
calculated based on cpu_shares()/1024. 1024 is the default and standard unit
for calculating relative cpu usage in cloud based container management
software.

Also add a new VM flag
(-XX:ActiveProcessorCount=xx) that allows the number of CPUs to be overridden.
This flag will be honored even if UseContainerSupport is not enabled.

Total available memory 
------------------------------- 
Use the memory_limit() value from the cgroup
file system to initialize the os::physical_memory() value in the VM. This value
will propagate to all other parts of the Java runtime.

Memory usage 
-------------------- 
Use memory_usage_in_bytes() for providing
os::available_memory() by subtracting the usage from the total available memory
allocated to the container.

As as troubleshooting aid, we will dump any
available container statistics to the hotspot error log and add container
specific information to the JVM logging system. Unified Logging will be added
to help to diagnose issue related to this support. Use -Xlog:os+container=trace
for maximum logging of container information.

A new option -XX:-UseContainerSupport will be
added to allow the container support to be disabled. The default for this flag
will be true. Container support will be enabled by default.

PS: jmap –F 那点事

When run without -F these
tools use Dynamic Attach Mechanism. This
works as follows.

1. Before connecting to Java process 1234, jmap creates
   a file .attach_pid1234 at the working directory of the target process or
   at /tmp.
2. Then jmap sends SIGQUIT to the target process. When JVM catches the signal
   and finds .attach_pid1234, it starts AttachListener thread.
3. AttachListener thread creates UNIX domain socket /tmp/.java_pid1234 to listen to commands from external tools.
4. For security reasons when a connection (from jmap) is
   accepted, JVM verifies that credentials of the socket peer are equal to euid and egid of
   JVM process. That's why jmap will not work if run by different user (even by
   root).
5. jmap connects
   to the socket, and sends dumpheap command.
6. This command is read and executed by AttachListener thread of the JVM. All output is sent back to the
   socket. Since the heap dump is made in-process directly by JVM, the operation
   is really fast. However, JVM can do this only at safepoints. If a safepoint cannot be reached (e.g. the
   process is hung, not responding, or a long GC is in progress), jmap will
   timeout and fail.

Let's
summarize the benefits and the drawbacks of Dynamic Attach.

Pros.

• Heap
  dump and other operations are run collaboratively by JVM at the maximum speed.
• You can
  use any version of jmap or jstack to connect to any other version of JVM.

Cons.

• The
  tool should be run by the same user (euid/egid) as the target JVM.
• Can be
  used only on live and healthy JVM.
• Will
  not work if the target JVM is started with -XX:+DisableAttachMechanism.

---

jmap -F / jstack -F

When run with -F the
tools switch to special mode that features HotSpot Serviceability Agent. In
this mode the target process is frozen; the tools read its memory via OS
debugging facilities, namely, ptrace on
Linux.

1. jmap -F invokes PTRACE_ATTACH on the target JVM. The target process is
   unconditionally suspended in response to SIGSTOP signal.
2. The tool reads JVM memory using PTRACE_PEEKDATA. ptrace can read only one word at a time, so too many calls
   required to read the large heap of the target process. This is very and very
   slow.
3. The tool reconstructs JVM internal structures based on the
   knowledge of the particular JVM version. Since different versions of JVM have
   different memory layout, -F mode works only if jmap comes
   from the same JDK as the target Java process.
4. The tool creates heap dump itself and then resumes the
   target process.

Pros.

• No
  cooperation from target JVM is required. Can be used even on a hung process.
• ptrace works whenever OS-level privileges are enough.
  E.g. root can dump processes of all other users.

Cons.

• Very
  slow for large heaps.
• The
  tool and the target process should be from the same version of JDK.

The safepoint is not guaranteed when the tool attaches in forced mode.
Though jmap tries to handle
all special cases, sometimes it may happen that target JVM is not in a
consistent state.

方式3: 直接在宿主机执行jmap

首先要先获取到容器里映射到宿主极的进程号

21025就是映射到宿主机的进程id

调用jmap –F 21025 ,但是失败了

没有任何头绪之际,突然发现

exe -> /jdk/bin/java 是红色, 这是容器里的java路径, 但宿主机没有这个路径, 会不会是这里有影响, 赶紧在宿主机也创建这个路径

再次执行jmap, 成功导出