OS2台：RH5.5 64位

WEBLOGIC VERSION：9.2.3

JDK：1.5 64位

weblogic一个管理服务，4个受管（2台服务器做集群，每台服务器2个受管）。应用部署后通过一个月测试发现adminserver1-2周就崩溃一次。日志错误信息种类很多如下：

####<2014-6-25 下午05时23分35秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <weblogic.timers.TimerThread> <<WLS Kernel>> <> <> <1403688215418> <BEA-002911> <WorkManager weblogic.kernel.System failed to schedule a request due to weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
at weblogic.utils.UnsyncCircularQueue.expandQueue(UnsyncCircularQueue.java:72)
at weblogic.utils.UnsyncCircularQueue.put(UnsyncCircularQueue.java:94)
at weblogic.work.MinThreadsConstraint.add(MinThreadsConstraint.java:85)
at weblogic.work.RequestManager.addToPriorityQueue(RequestManager.java:261)
at weblogic.work.RequestManager.executeIt(RequestManager.java:235)
at weblogic.work.ServerWorkManagerImpl.schedule(ServerWorkManagerImpl.java:142)
at weblogic.timers.internal.TimerManagerImpl.execute(TimerManagerImpl.java:631)
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:285)
>
####<2014-6-26 上午11时44分05秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754245838> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=HomePage1.>
####<2014-6-26 上午11时44分16秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754256969> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpserver1%2CType%3DServer%22%29.>
####<2014-6-26 上午11时46分19秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754379930> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpims1%2CType%3DServer%22%29.>
####<2014-6-26 上午11时46分24秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754384137> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpserver1%2CType%3DServer%22%29.>
####<2014-6-28 上午07时14分45秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403910885281> <BEA-002911> <WorkManager direct failed to schedule a request due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分52秒 CST> <Error> <RMI> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403911072751> <BEA-080001> <Error in Dispatcher
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分57秒 CST> <Warning> <RMI> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911077647> <BEA-080004> <An error was thrown by rmi server: weblogic.jndi.internal.RootNamingNode.lookup(Ljava.lang.String;Ljava.util.Hashtable;)
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时18分08秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911088190> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时18分18秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911098656> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时32分51秒 CST> <Error> <Socket> <dxp1> <AdminServer> <ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403911971079> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时38分39秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912319851> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时40分17秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912417869> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时41分22秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912482057> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时41分51秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912511420> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-7-2 上午10时19分42秒 CST> <Notice> <Security> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267582796> <BEA-090082> <Security initializing using security realm myrealm.>
####<2014-7-2 上午10时19分45秒 CST> <Notice> <WebLogicServer> <dxp1> <AdminServer> <main> <<WLS Kernel>> <> <> <1404267585545> <BEA-000365> <Server state changed to STANDBY>
####<2014-7-2 上午10时19分45秒 CST> <Notice> <WebLogicServer> <dxp1> <AdminServer> <main> <<WLS Kernel>> <> <> <1404267585546> <BEA-000365> <Server state changed to STARTING>
####<2014-7-2 上午10时19分47秒 CST> <Notice> <Log Management> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267587400> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200004> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "638" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200007> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时54分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913260011> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "698" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时54分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913260012> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "718" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-7-2 上午10时19分51秒 CST> <Notice> <Log Management> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267591935> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时46分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912782263> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@26f147c8", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时46分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912782267> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "624" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@2a9faf71", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时47分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912842270> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "718" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@26f147c8", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时47分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912842270> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "684" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@2a9faf71", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-7-2 上午10时19分50秒 CST> <Notice> <Log Management> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267590942> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时17分55秒 CST> <Error> <WebLogicServer> <dxp1> <dxpserver1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911075803> <BEA-000337> <[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "623" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@6c2416ed", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>

具体有：

1、

####<2014-6-25 下午05时23分35秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <weblogic.timers.TimerThread> <<WLS Kernel>> <> <> <1403688215418> <BEA-002911> <WorkManager weblogic.kernel.System failed to schedule a request due to weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
at weblogic.utils.UnsyncCircularQueue.expandQueue(UnsyncCircularQueue.java:72)
at weblogic.utils.UnsyncCircularQueue.put(UnsyncCircularQueue.java:94)
at weblogic.work.MinThreadsConstraint.add(MinThreadsConstraint.java:85)
at weblogic.work.RequestManager.addToPriorityQueue(RequestManager.java:261)
at weblogic.work.RequestManager.executeIt(RequestManager.java:235)
at weblogic.work.ServerWorkManagerImpl.schedule(ServerWorkManagerImpl.java:142)
at weblogic.timers.internal.TimerManagerImpl.execute(TimerManagerImpl.java:631)
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:285)

2、

####<2014-6-28 上午07时14分45秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403910885281> <BEA-002911> <WorkManager direct failed to schedule a request due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space

3、

####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200004> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "638" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200007> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

针对这些错误做出联合整改方案具体如下：

1、针对weblogic.utils.UnsyncCircularQueue$FullQueueException通过官方文档查看为weblogic bug 需打补丁包：Patch 8179406

WebLogic - Admin Server Stops Responding With "weblogic.utils.UnsyncCircularQueue$FullQueueException" Exceptions (文档 ID 1228224.1)

打补丁步骤：

（1）修改/apps/weblogic/bea/weblogic92/common/bin/commEnv.sh,在文件开始处添加export CLASSPATH=/apps/weblogic/bea/weblogic92/Patches/p8179406.jar:$CLASSPATH
（2）将p8179406.jar补丁拷贝到apps/weblogic/bea/weblogic92/Patches目录下，然后重启WebLogic

2、针对错误3 点击Adminserver -> 协议 -> 常规 -> 启用隧道 ，然后重启服务

通过这两个解决方案目前观察2个月adminserver运行稳定。没有出现类似错误。

补丁下载：     http://pan.baidu.com/s/1c0gr6M0

bug文档：    http://pan.baidu.com/s/1c0H5r7y

可参考的连接：https://community.oracle.com/thread/694983?start=0&tstart=0

http://v-vampires.iteye.com/blog/861897

weblogic诊断案例-AdminServer平均1-2周崩溃的更多相关文章

  1. Linux SendMail发送邮件失败诊断案例(三)

    一Linux服务器突然发送不出邮件,检查了很多地方都没有发现异常,检查/var/log/maillog发现如下具体信息: Apr 12 00:36:04 mylinux sendmail[4685]: ...

  2. 深入浅出Oracle:DBA入门、进阶与诊断案例（读书笔记1）

    一.数据库的启动和关闭 Oracle Server共有2部分组成:Instance和Database. Instance是指一组后台进程/线程和一块共享内存区域,而Database是指存储在磁盘上的一 ...

  3. Linux sendmail发送邮件失败诊断案例(一)

    在新服务器上测试sendmail发送邮件时,发现邮件发送不成功,检查日志文件发现如下错误(Notice:由于涉及公司服务器,邮箱等,故下面hostname.邮箱地址等信息使用xxx代替) tail - ...

  4. Linux SendMail发送邮件失败诊断案例(二)

    Linux上Sendmail经常由于一些配置问题,导致邮件发送失败,下面整理.收集了一些邮件发送失败.异常的案例. 案例1:在新服务器上测试sendmail发送邮件时,发现邮件发送不成功,检查/var ...

  5. Netbeans8下 Weblogic EJB案例

    1:接口 @Remote public interface XgmZzsNssb {} 2:实现 @Stateless(mappedName="XgmZzsNssbImpl") @ ...

  6. Linux SendMail发送邮件失败诊断案例(四）

    最近又碰到一起Linux下SendMail发送邮件失败的案例,邮件发送后,邮箱收不到具体邮件, 查看日志/var/log/maillog 发现有"DSN: User unknown" ...

  7. 深入浅出Oracle:DBA入门、进阶与诊断案例（读书笔记2)

    第5章  Buffer Cache与Shared Pool原理 5.1 Buffer Cache原理 Buffer Cache是Oracle SGA中的一个重要部分,通常的数据访问和修改都需要通过Bu ...

  8. 转 Linux SendMail发送邮件失败诊断案例(四）

    http://www.cnblogs.com/kerrycode/p/7826036.html

  9. 深入浅出Oracle：DBA入门、进阶与诊断案例 PDF 下载

    网盘地址: 链接:https://pan.baidu.com/s/1tMFoNSUW7ICKOtmSQ5ZazA 提取码:dbnc

随机推荐

  1. js给定时器调用传递参数

    给定时器调用传递参数 无论是window.setTimeout 还是window.setInterval,在使用函数名作为调用句柄时都不 能带参数,而在许多场合必需要带参数,这就需要想方法解决.例如对 ...

  2. CSharp 如何通过拼接XML调用存储过程来查询数据

    每查询数据,需要访问一次数据库,这样会影响性能:如果把查询的数据拼接成XML形式,作为一个参数整体传给存储过程来处理,这只访问数据库一次,执行速度会快很多. 1.C#代码如下: /// <sum ...

  3. Microsoft Dynamics CRM2011 必备知识点

    一.CRM基本知识 1.CRM2001 有几个服务端点? 答:对外公开的服务,如Web服务,WCF,Restful API 2.一个ERP系统,要访问CRM的数据,CRM2011有哪些现有的服务入口提 ...

  4. php 导出csv文件

    <?php $sql = "select * from members_sqzj order by id asc"; $result = $db->fetch_All( ...

  5. LintCode "Swap Two Nodes in Linked List"

    Nothing special. Just take care of corner cases. class Solution { public: /** * @param head a ListNo ...

  6. LintCode "Coins in a Line III" !!

    https://codesolutiony.wordpress.com/2015/05/24/lintcode-coins-in-a-line-iii/ A very juicy one! Deser ...

  7. Chrome插件开发入门（二）——消息传递机制

    Chrome插件开发入门(二)——消息传递机制   由于插件的js运行环境有区别,所以消息传递机制是一个重要内容.阅读了很多博文,大家已经说得很清楚了,直接转一篇@姬小光 的博文,总结的挺好.后面附一 ...

  8. 黄聪：WordPress 函数：apply_filters()（创建过滤器）

    apply_filters() 函数用来创建一个过滤器,大多数被用在函数中,是 WordPress 插件机制中非常重要的一个函数,能让其它的主题和插件对一个值进行修改过滤. 用法 apply_filt ...

  9. Linux进程内存分析pmap命令(转)

    名称:       pmap - report memory map of a process(查看进程的内存映像信息)用法       pmap [ -x | -d ] [ -q ] pids... ...

  10. DISPOSE_ON_CLOSE 和 EXIT_ON_CLOSE 的区别

    If you have several JFrames open and you close one that has EXIT_ON_CLOSE it will close all the JFra ...