OS2台：RH5.5 64位

WEBLOGIC VERSION：9.2.3

JDK：1.5 64位

weblogic一个管理服务，4个受管（2台服务器做集群，每台服务器2个受管）。应用部署后通过一个月测试发现adminserver1-2周就崩溃一次。日志错误信息种类很多如下：

####<2014-6-25 下午05时23分35秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <weblogic.timers.TimerThread> <<WLS Kernel>> <> <> <1403688215418> <BEA-002911> <WorkManager weblogic.kernel.System failed to schedule a request due to weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
at weblogic.utils.UnsyncCircularQueue.expandQueue(UnsyncCircularQueue.java:72)
at weblogic.utils.UnsyncCircularQueue.put(UnsyncCircularQueue.java:94)
at weblogic.work.MinThreadsConstraint.add(MinThreadsConstraint.java:85)
at weblogic.work.RequestManager.addToPriorityQueue(RequestManager.java:261)
at weblogic.work.RequestManager.executeIt(RequestManager.java:235)
at weblogic.work.ServerWorkManagerImpl.schedule(ServerWorkManagerImpl.java:142)
at weblogic.timers.internal.TimerManagerImpl.execute(TimerManagerImpl.java:631)
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:285)
>
####<2014-6-26 上午11时44分05秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754245838> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=HomePage1.>
####<2014-6-26 上午11时44分16秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754256969> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpserver1%2CType%3DServer%22%29.>
####<2014-6-26 上午11时46分19秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754379930> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpims1%2CType%3DServer%22%29.>
####<2014-6-26 上午11时46分24秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754384137> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpserver1%2CType%3DServer%22%29.>
####<2014-6-28 上午07时14分45秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403910885281> <BEA-002911> <WorkManager direct failed to schedule a request due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分52秒 CST> <Error> <RMI> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403911072751> <BEA-080001> <Error in Dispatcher
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分57秒 CST> <Warning> <RMI> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911077647> <BEA-080004> <An error was thrown by rmi server: weblogic.jndi.internal.RootNamingNode.lookup(Ljava.lang.String;Ljava.util.Hashtable;)
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时18分08秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911088190> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时18分18秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911098656> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时32分51秒 CST> <Error> <Socket> <dxp1> <AdminServer> <ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403911971079> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时38分39秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912319851> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时40分17秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912417869> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时41分22秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912482057> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时41分51秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912511420> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-7-2 上午10时19分42秒 CST> <Notice> <Security> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267582796> <BEA-090082> <Security initializing using security realm myrealm.>
####<2014-7-2 上午10时19分45秒 CST> <Notice> <WebLogicServer> <dxp1> <AdminServer> <main> <<WLS Kernel>> <> <> <1404267585545> <BEA-000365> <Server state changed to STANDBY>
####<2014-7-2 上午10时19分45秒 CST> <Notice> <WebLogicServer> <dxp1> <AdminServer> <main> <<WLS Kernel>> <> <> <1404267585546> <BEA-000365> <Server state changed to STARTING>
####<2014-7-2 上午10时19分47秒 CST> <Notice> <Log Management> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267587400> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200004> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "638" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200007> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时54分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913260011> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "698" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时54分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913260012> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "718" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-7-2 上午10时19分51秒 CST> <Notice> <Log Management> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267591935> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时46分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912782263> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@26f147c8", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时46分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912782267> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "624" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@2a9faf71", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时47分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912842270> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "718" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@26f147c8", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时47分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912842270> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "684" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@2a9faf71", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-7-2 上午10时19分50秒 CST> <Notice> <Log Management> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267590942> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时17分55秒 CST> <Error> <WebLogicServer> <dxp1> <dxpserver1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911075803> <BEA-000337> <[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "623" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@6c2416ed", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>

具体有：

1、

####<2014-6-25 下午05时23分35秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <weblogic.timers.TimerThread> <<WLS Kernel>> <> <> <1403688215418> <BEA-002911> <WorkManager weblogic.kernel.System failed to schedule a request due to weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
at weblogic.utils.UnsyncCircularQueue.expandQueue(UnsyncCircularQueue.java:72)
at weblogic.utils.UnsyncCircularQueue.put(UnsyncCircularQueue.java:94)
at weblogic.work.MinThreadsConstraint.add(MinThreadsConstraint.java:85)
at weblogic.work.RequestManager.addToPriorityQueue(RequestManager.java:261)
at weblogic.work.RequestManager.executeIt(RequestManager.java:235)
at weblogic.work.ServerWorkManagerImpl.schedule(ServerWorkManagerImpl.java:142)
at weblogic.timers.internal.TimerManagerImpl.execute(TimerManagerImpl.java:631)
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:285)

2、

####<2014-6-28 上午07时14分45秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403910885281> <BEA-002911> <WorkManager direct failed to schedule a request due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space

3、

####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200004> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "638" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200007> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

针对这些错误做出联合整改方案具体如下：

1、针对weblogic.utils.UnsyncCircularQueue$FullQueueException通过官方文档查看为weblogic bug 需打补丁包：Patch 8179406

WebLogic - Admin Server Stops Responding With "weblogic.utils.UnsyncCircularQueue$FullQueueException" Exceptions (文档 ID 1228224.1)

打补丁步骤：

（1）修改/apps/weblogic/bea/weblogic92/common/bin/commEnv.sh,在文件开始处添加export CLASSPATH=/apps/weblogic/bea/weblogic92/Patches/p8179406.jar:$CLASSPATH
（2）将p8179406.jar补丁拷贝到apps/weblogic/bea/weblogic92/Patches目录下，然后重启WebLogic

2、针对错误3 点击Adminserver -> 协议 -> 常规 -> 启用隧道 ，然后重启服务

通过这两个解决方案目前观察2个月adminserver运行稳定。没有出现类似错误。

补丁下载：     http://pan.baidu.com/s/1c0gr6M0

bug文档：    http://pan.baidu.com/s/1c0H5r7y

可参考的连接：https://community.oracle.com/thread/694983?start=0&tstart=0

http://v-vampires.iteye.com/blog/861897

weblogic诊断案例-AdminServer平均1-2周崩溃的更多相关文章

  1. Linux SendMail发送邮件失败诊断案例(三)

    一Linux服务器突然发送不出邮件,检查了很多地方都没有发现异常,检查/var/log/maillog发现如下具体信息: Apr 12 00:36:04 mylinux sendmail[4685]: ...

  2. 深入浅出Oracle:DBA入门、进阶与诊断案例（读书笔记1）

    一.数据库的启动和关闭 Oracle Server共有2部分组成:Instance和Database. Instance是指一组后台进程/线程和一块共享内存区域,而Database是指存储在磁盘上的一 ...

  3. Linux sendmail发送邮件失败诊断案例(一)

    在新服务器上测试sendmail发送邮件时,发现邮件发送不成功,检查日志文件发现如下错误(Notice:由于涉及公司服务器,邮箱等,故下面hostname.邮箱地址等信息使用xxx代替) tail - ...

  4. Linux SendMail发送邮件失败诊断案例(二)

    Linux上Sendmail经常由于一些配置问题,导致邮件发送失败,下面整理.收集了一些邮件发送失败.异常的案例. 案例1:在新服务器上测试sendmail发送邮件时,发现邮件发送不成功,检查/var ...

  5. Netbeans8下 Weblogic EJB案例

    1:接口 @Remote public interface XgmZzsNssb {} 2:实现 @Stateless(mappedName="XgmZzsNssbImpl") @ ...

  6. Linux SendMail发送邮件失败诊断案例(四）

    最近又碰到一起Linux下SendMail发送邮件失败的案例,邮件发送后,邮箱收不到具体邮件, 查看日志/var/log/maillog 发现有"DSN: User unknown" ...

  7. 深入浅出Oracle:DBA入门、进阶与诊断案例（读书笔记2)

    第5章  Buffer Cache与Shared Pool原理 5.1 Buffer Cache原理 Buffer Cache是Oracle SGA中的一个重要部分,通常的数据访问和修改都需要通过Bu ...

  8. 转 Linux SendMail发送邮件失败诊断案例(四）

    http://www.cnblogs.com/kerrycode/p/7826036.html

  9. 深入浅出Oracle：DBA入门、进阶与诊断案例 PDF 下载

    网盘地址: 链接:https://pan.baidu.com/s/1tMFoNSUW7ICKOtmSQ5ZazA 提取码:dbnc

随机推荐

  1. Javascript 去掉字符串前后空格的五种方法

    第一种:循环检查替换 [javascript] //供使用者调用 function trim(s){ return trimRight(trimLeft(s)); } //去掉左边的空白 functi ...

  2. java.io.IOException: ORA-22920: 未锁定含有 LOB 值的行

         究其原因是因为没有锁定要更新的行记录.将 mysql="select filebody from filelist where filename=?"中的SQL语句加上 ...

  3. nginx服务器应用中遇到的两个问题

    1>首先是413的错误! client_max_body_size Context: http, server, location It is the maximum size of a cli ...

  4. Unable to resolve target 'android-i'

    重新装完Ecplise+ATD+Android SDK 在Ecplise工作空间导入之前写过的Android项目会出现错误,大部分是SDK 版本不符,如下错误提示:Error:Unable to re ...

  5. Hibernate延迟加载机制详解

    摘自 http://blog.chinaunix.net/uid-20577907-id-3129234.html 1 延迟加载: 延迟加载机制是为了避免一些无谓的性能开销而提出来的,所谓延迟加载就是 ...

  6. 程序员书单_HeadFirst系列

    Head First Jquery(中文版),完整扫描版 http://download.csdn.net/detail/shenzhq1980/9103615 Head First Python(完 ...

  7. 【linux】压缩和解压缩

    .gz格式 压缩gzip: gzip只能压缩文件,且压缩后文件消失,不能压缩目录. [root@andon tmp]# ls ml orbit-gdm pulse-2sLvu7UbjUYf pulse ...

  8. Javascript跨域问题解决方案

    ---本方法来自网上,因最近项目需求,解决了三天未果,最终发现此文并测试可用. 一个 非常优雅绝妙的解决方案!话不多说,直接上解决方案原理图: 该图要解决的问题说明如下:在AAA.com域名下的ind ...

  9. LinearLayout属性baselineAligned的作用及baseline

    相信大家对LinearLayout已经相当熟悉,但你们是否了解它的属性baselineAligned呢? Android官方文档是这么描述的:

  10. 【Spring学习笔记-MVC-3.1】SpringMVC返回Json数据-方式1-扩展

    <Spring学习笔记-MVC>系列文章,讲解返回json数据的文章共有3篇,分别为: [Spring学习笔记-MVC-3]SpringMVC返回Json数据-方式1:http://www ...