java应用挂死故障排查

现象：

　　java开发的web应用无法访问

排查：

1．从resin/log/watchdog-manager.log的日志里可以看出来，jvm的内存满，无法创建新进程

java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:574)
        at com.caucho.util.ThreadPool$ThreadLauncher.startConnection(ThreadPool.java:812)
        at com.caucho.util.ThreadPool$ThreadLauncher.run(ThreadPool.java:851)
        at java.lang.Thread.run(Thread.java:595)

　　

2．用jstat –gcutil pid可以看出jvm的OLD堆内存区域已经100%，与上述判断符合

S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT  

0.00   0.00  99.19 100.00  20.72    577   62.517   322 2953.102 3015.619

　　

3．用jstack pid查看jvm的内部的执行线程情况，可以看到存在大量的下述线程，阻塞

Thread 15810: (state = BLOCKED)

 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park() @bci=5, line=118 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=41, line=1767 (Interpreted frame)
 - com.ctsi.nmp.flux.ecfchart.ECFluxChartService.writeChart(com.ctsi.nmp.flux.ecfchart.ECFluxChartType, com.ctsi.nmp.flux.ecfchart.E

CFluxType, java.lang.String, java.util.Date, java.util.Date, java.util.List, boolean, java.io.OutputStream) @bci=295, line=189 (Compiled frame)

 - com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.executeChart() @bci=1051, line=203 (Compiled frame)
 - sun.reflect.GeneratedMethodAccessor49.invoke(java.lang.Object, java.lang.Object[]) @bci=36 (Compiled frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Compiled frame)
 - com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.execute() @bci=94, line=230 (Interpreted frame)
 - _jsp._nos._flux._flux._chart._ECFluxChart__jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletR
esponse) @bci=130, line=39 (Compiled frame)
 - com.caucho.jsp.JavaPage.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=9, line=61 (Interpreted frame)
 - com.caucho.jsp.Page.pageservice(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=252, line=578

 (Compiled frame)
 - com.caucho.server.dispatch.PageFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=262, line=195 (Compiled frame)
 - com.zhongying.util.EncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain
) @bci=39, line=80 (Compiled frame)
 - com.caucho.server.webapp.WebAppFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=140, line=187 (Compiled frame)
 - com.caucho.server.dispatch.ServletInvocation.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=97, line=266 (Compiled frame)
 - com.caucho.server.hmux.HmuxRequest.handleRequest() @bci=478, line=435 (Compiled frame)
 - com.caucho.server.port.TcpConnection.run() @bci=375, line=678 (Compiled frame)
 - com.caucho.util.ThreadPool$Item.runTasks() @bci=246, line=721 (Compiled frame)
 - com.caucho.util.ThreadPool$Item.run() @bci=106, line=643 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=595 (Interpreted frame)

　　

可以判断是应用的com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.executeChart部分出现了问题，导致大量的线程阻塞，研发优化这部分代码后问题未再重现