第一次JVM分析记录：Out of Memory Error (workgroup.cpp:96), pid=6196, tid=139999645685504

tomcat的catalina.out日志报错如下：

Exception in thread "http-bio-8081-Acceptor-0" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:162)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:142)
at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:533)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:236)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "http-bio-8081-exec-98" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

tomcat下的hs_err_pid6196.log日志部分内容如下：

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (workgroup.cpp:96), pid=6196, tid=139999645685504
#
# JRE version: (7.0_80-b15) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

--------------- T H R E A D ---------------

Current thread (0x00007f542c00a000): JavaThread "Unknown thread" [_thread_in_vm, id=6197, stack(0x00007f5435165000,0x00007f5435266000)]

Stack: [0x00007f5435165000,0x00007f5435266000], sp=0x00007f5435264840, free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x9a320a] VMError::report_and_die()+0x2ea
V [libjvm.so+0x498d3b] report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
V [libjvm.so+0x9b9bfa] WorkGang::initialize_workers()+0xaa
V [libjvm.so+0x56b522] GenCollectedHeap::GenCollectedHeap(GenCollectorPolicy*)+0x12
V [libjvm.so+0x97508a] Universe::initialize_heap()+0xba
V [libjvm.so+0x976269] universe_init()+0x79
V [libjvm.so+0x5b2f25] init_globals()+0x65
V [libjvm.so+0x95db4d] Threads::create_vm(JavaVMInitArgs*, bool*)+0x1ed
V [libjvm.so+0x63b2e4] JNI_CreateJavaVM+0x74
C [libjli.so+0x2f8e] JavaMain+0x9e

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x00007f542c00a000 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=6197, stack(0x00007f5435165000,0x00007f5435266000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events

......

部署在服务器的项目从昨天开始突然频繁挂掉，不断重启不断挂掉，查看相关的日志内容，如上所示（由于权限原因，无法使用jstack等命令跟踪查询线程情况）。

查找原因：运维在启动项目时 设置的项目所属用户的线程数为4096 导致项目创建线程受限

修改用户线程数为65535 解决问题。