weblogic诊断案例-AdminServer平均1-2周崩溃

OS2台：RH5.5 64位

WEBLOGIC VERSION：9.2.3

JDK：1.5 64位

weblogic一个管理服务，4个受管（2台服务器做集群，每台服务器2个受管）。应用部署后通过一个月测试发现adminserver1-2周就崩溃一次。日志错误信息种类很多如下：

####<2014-6-25 下午05时23分35秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <weblogic.timers.TimerThread> <<WLS Kernel>> <> <> <1403688215418> <BEA-002911> <WorkManager weblogic.kernel.System failed to schedule a request due to weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
at weblogic.utils.UnsyncCircularQueue.expandQueue(UnsyncCircularQueue.java:72)
at weblogic.utils.UnsyncCircularQueue.put(UnsyncCircularQueue.java:94)
at weblogic.work.MinThreadsConstraint.add(MinThreadsConstraint.java:85)
at weblogic.work.RequestManager.addToPriorityQueue(RequestManager.java:261)
at weblogic.work.RequestManager.executeIt(RequestManager.java:235)
at weblogic.work.ServerWorkManagerImpl.schedule(ServerWorkManagerImpl.java:142)
at weblogic.timers.internal.TimerManagerImpl.execute(TimerManagerImpl.java:631)
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:285)
>
####<2014-6-26 上午11时44分05秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754245838> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=HomePage1.>
####<2014-6-26 上午11时44分16秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754256969> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpserver1%2CType%3DServer%22%29.>
####<2014-6-26 上午11时46分19秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754379930> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpims1%2CType%3DServer%22%29.>
####<2014-6-26 上午11时46分24秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754384137> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpserver1%2CType%3DServer%22%29.>
####<2014-6-28 上午07时14分45秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403910885281> <BEA-002911> <WorkManager direct failed to schedule a request due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分52秒 CST> <Error> <RMI> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403911072751> <BEA-080001> <Error in Dispatcher
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分57秒 CST> <Warning> <RMI> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911077647> <BEA-080004> <An error was thrown by rmi server: weblogic.jndi.internal.RootNamingNode.lookup(Ljava.lang.String;Ljava.util.Hashtable;)
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时18分08秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911088190> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时18分18秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911098656> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时32分51秒 CST> <Error> <Socket> <dxp1> <AdminServer> <ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403911971079> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时38分39秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912319851> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时40分17秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912417869> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时41分22秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912482057> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时41分51秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912511420> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-7-2 上午10时19分42秒 CST> <Notice> <Security> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267582796> <BEA-090082> <Security initializing using security realm myrealm.>
####<2014-7-2 上午10时19分45秒 CST> <Notice> <WebLogicServer> <dxp1> <AdminServer> <main> <<WLS Kernel>> <> <> <1404267585545> <BEA-000365> <Server state changed to STANDBY>
####<2014-7-2 上午10时19分45秒 CST> <Notice> <WebLogicServer> <dxp1> <AdminServer> <main> <<WLS Kernel>> <> <> <1404267585546> <BEA-000365> <Server state changed to STARTING>
####<2014-7-2 上午10时19分47秒 CST> <Notice> <Log Management> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267587400> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200004> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "638" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200007> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时54分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913260011> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "698" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时54分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913260012> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "718" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-7-2 上午10时19分51秒 CST> <Notice> <Log Management> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267591935> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时46分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912782263> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@26f147c8", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时46分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912782267> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "624" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@2a9faf71", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时47分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912842270> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "718" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@26f147c8", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时47分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912842270> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "684" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@2a9faf71", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-7-2 上午10时19分50秒 CST> <Notice> <Log Management> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267590942> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时17分55秒 CST> <Error> <WebLogicServer> <dxp1> <dxpserver1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911075803> <BEA-000337> <[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "623" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@6c2416ed", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>

具体有：

1、

2、

####<2014-6-28 上午07时14分45秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403910885281> <BEA-002911> <WorkManager direct failed to schedule a request due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space

3、

####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200007> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

针对这些错误做出联合整改方案具体如下：

1、针对weblogic.utils.UnsyncCircularQueue$FullQueueException通过官方文档查看为weblogic bug 需打补丁包：Patch 8179406

WebLogic - Admin Server Stops Responding With "weblogic.utils.UnsyncCircularQueue$FullQueueException" Exceptions (文档 ID 1228224.1)

打补丁步骤：

（1）修改/apps/weblogic/bea/weblogic92/common/bin/commEnv.sh,在文件开始处添加export CLASSPATH=/apps/weblogic/bea/weblogic92/Patches/p8179406.jar:$CLASSPATH
（2）将p8179406.jar补丁拷贝到apps/weblogic/bea/weblogic92/Patches目录下，然后重启WebLogic

2、针对错误3 点击Adminserver -> 协议 -> 常规 -> 启用隧道，然后重启服务

通过这两个解决方案目前观察2个月adminserver运行稳定。没有出现类似错误。

补丁下载： http://pan.baidu.com/s/1c0gr6M0

bug文档： http://pan.baidu.com/s/1c0H5r7y

可参考的连接：https://community.oracle.com/thread/694983?start=0&tstart=0

http://v-vampires.iteye.com/blog/861897

weblogic诊断案例-AdminServer平均1-2周崩溃的更多相关文章

Linux SendMail发送邮件失败诊断案例(三)
一Linux服务器突然发送不出邮件,检查了很多地方都没有发现异常,检查/var/log/maillog发现如下具体信息: Apr 12 00:36:04 mylinux sendmail[4685]: ...
深入浅出Oracle:DBA入门、进阶与诊断案例（读书笔记1）
一.数据库的启动和关闭 Oracle Server共有2部分组成:Instance和Database. Instance是指一组后台进程/线程和一块共享内存区域,而Database是指存储在磁盘上的一 ...
Linux sendmail发送邮件失败诊断案例(一)
在新服务器上测试sendmail发送邮件时,发现邮件发送不成功,检查日志文件发现如下错误(Notice:由于涉及公司服务器,邮箱等,故下面hostname.邮箱地址等信息使用xxx代替) tail - ...
Linux SendMail发送邮件失败诊断案例(二)
Linux上Sendmail经常由于一些配置问题,导致邮件发送失败,下面整理.收集了一些邮件发送失败.异常的案例. 案例1:在新服务器上测试sendmail发送邮件时,发现邮件发送不成功,检查/var ...
Netbeans8下 Weblogic EJB案例
1:接口 @Remote public interface XgmZzsNssb {} 2:实现 @Stateless(mappedName="XgmZzsNssbImpl") @ ...
Linux SendMail发送邮件失败诊断案例(四）
最近又碰到一起Linux下SendMail发送邮件失败的案例,邮件发送后,邮箱收不到具体邮件, 查看日志/var/log/maillog 发现有"DSN: User unknown" ...
深入浅出Oracle:DBA入门、进阶与诊断案例（读书笔记2)
第5章 Buffer Cache与Shared Pool原理 5.1 Buffer Cache原理 Buffer Cache是Oracle SGA中的一个重要部分,通常的数据访问和修改都需要通过Bu ...
转 Linux SendMail发送邮件失败诊断案例(四）
http://www.cnblogs.com/kerrycode/p/7826036.html
深入浅出Oracle：DBA入门、进阶与诊断案例 PDF 下载
网盘地址: 链接:https://pan.baidu.com/s/1tMFoNSUW7ICKOtmSQ5ZazA 提取码:dbnc

随机推荐

alpha预乘
将(r,g,b,a)变为(r*a,g*a,b*a,a)的操作称为alpha预乘. 对于alpha预乘的图片,应使用(One,OneMinusSrcAlpha)进行混合. 使用alpha预乘方式混合出来 ...
Kettle合并记录步骤
转载: http://blog.itpub.net/post/37422/464323 该步骤用于将两个不同来源的数据合并,这两个来源的数据分别为旧数据和新数据,该步骤将旧数据和新数据按照指定的关键字 ...
查询oracle RAC实例名称等信息
select * from gv$instance;
discuz 发布分类信息，能不能设置单版块去掉“发帖子”（默认点发帖后为自定义的默认分类信息模版）
http://www.discuz.net/forum.php?mod=viewthread&tid=3365198&page=1#pid26849156
eclipse注解——作者，创建时间，版本
总结: /** * @author liangyadong * @date ${date} ${time} * @version 1.0 */
Android中实现java与PHP服务器（基于新浪云免费云平台）http通信详解
Android中实现java与PHP服务器(基于新浪云免费云平台)http通信详解 (本文转自: http://blog.csdn.net/yinhaide/article/details/44756 ...
基于jQuery动态创建html元素
在做web前端开发的时候,经常遇到一些数据多少或则类型不能在运行之前就确定下来的情况,此时,数据的展示,就要借助于动态创建html元素来展示了. 常见的动态创建HTML元素的方式,有如下几种,大体都差 ...
使用Servlet Filter做Login checking
1) 建一个Login Servlet: Login.java package com.my; import java.io.*; import javax.servlet.*; import jav ...
关于snprintf的一个warning
在编一段代码时用到snprintf,有个很奇怪的warning 编译提示: warning C4013: 'snprintf' undefined; assuming extern returning ...
Objective-C中nil与release的区别与用法
首先说一下他们两的作用,nil就是把一个对象的指针置为空,只是切断了指针与内存中对象的联系,它对内存的释放没有什么作用:而release才是真正用于内存释放的,release后系统会将该块内存标记为可 ...

weblogic诊断案例-AdminServer平均1-2周崩溃

weblogic诊断案例-AdminServer平均1-2周崩溃的更多相关文章

随机推荐

热门专题