OS2台：RH5.5 64位

WEBLOGIC VERSION：9.2.3

JDK：1.5 64位

weblogic一个管理服务，4个受管（2台服务器做集群，每台服务器2个受管）。应用部署后通过一个月测试发现adminserver1-2周就崩溃一次。日志错误信息种类很多如下：

####<2014-6-25 下午05时23分35秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <weblogic.timers.TimerThread> <<WLS Kernel>> <> <> <1403688215418> <BEA-002911> <WorkManager weblogic.kernel.System failed to schedule a request due to weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
at weblogic.utils.UnsyncCircularQueue.expandQueue(UnsyncCircularQueue.java:72)
at weblogic.utils.UnsyncCircularQueue.put(UnsyncCircularQueue.java:94)
at weblogic.work.MinThreadsConstraint.add(MinThreadsConstraint.java:85)
at weblogic.work.RequestManager.addToPriorityQueue(RequestManager.java:261)
at weblogic.work.RequestManager.executeIt(RequestManager.java:235)
at weblogic.work.ServerWorkManagerImpl.schedule(ServerWorkManagerImpl.java:142)
at weblogic.timers.internal.TimerManagerImpl.execute(TimerManagerImpl.java:631)
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:285)
>
####<2014-6-26 上午11时44分05秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754245838> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=HomePage1.>
####<2014-6-26 上午11时44分16秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754256969> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpserver1%2CType%3DServer%22%29.>
####<2014-6-26 上午11时46分19秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754379930> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpims1%2CType%3DServer%22%29.>
####<2014-6-26 上午11时46分24秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754384137> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpserver1%2CType%3DServer%22%29.>
####<2014-6-28 上午07时14分45秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403910885281> <BEA-002911> <WorkManager direct failed to schedule a request due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分52秒 CST> <Error> <RMI> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403911072751> <BEA-080001> <Error in Dispatcher
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分57秒 CST> <Warning> <RMI> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911077647> <BEA-080004> <An error was thrown by rmi server: weblogic.jndi.internal.RootNamingNode.lookup(Ljava.lang.String;Ljava.util.Hashtable;)
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时18分08秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911088190> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时18分18秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911098656> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时32分51秒 CST> <Error> <Socket> <dxp1> <AdminServer> <ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403911971079> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时38分39秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912319851> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时40分17秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912417869> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时41分22秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912482057> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时41分51秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912511420> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-7-2 上午10时19分42秒 CST> <Notice> <Security> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267582796> <BEA-090082> <Security initializing using security realm myrealm.>
####<2014-7-2 上午10时19分45秒 CST> <Notice> <WebLogicServer> <dxp1> <AdminServer> <main> <<WLS Kernel>> <> <> <1404267585545> <BEA-000365> <Server state changed to STANDBY>
####<2014-7-2 上午10时19分45秒 CST> <Notice> <WebLogicServer> <dxp1> <AdminServer> <main> <<WLS Kernel>> <> <> <1404267585546> <BEA-000365> <Server state changed to STARTING>
####<2014-7-2 上午10时19分47秒 CST> <Notice> <Log Management> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267587400> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200004> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "638" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200007> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时54分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913260011> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "698" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时54分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913260012> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "718" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-7-2 上午10时19分51秒 CST> <Notice> <Log Management> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267591935> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时46分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912782263> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@26f147c8", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时46分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912782267> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "624" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@2a9faf71", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时47分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912842270> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "718" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@26f147c8", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时47分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912842270> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "684" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@2a9faf71", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-7-2 上午10时19分50秒 CST> <Notice> <Log Management> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267590942> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时17分55秒 CST> <Error> <WebLogicServer> <dxp1> <dxpserver1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911075803> <BEA-000337> <[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "623" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@6c2416ed", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>

具体有：

1、

####<2014-6-25 下午05时23分35秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <weblogic.timers.TimerThread> <<WLS Kernel>> <> <> <1403688215418> <BEA-002911> <WorkManager weblogic.kernel.System failed to schedule a request due to weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
at weblogic.utils.UnsyncCircularQueue.expandQueue(UnsyncCircularQueue.java:72)
at weblogic.utils.UnsyncCircularQueue.put(UnsyncCircularQueue.java:94)
at weblogic.work.MinThreadsConstraint.add(MinThreadsConstraint.java:85)
at weblogic.work.RequestManager.addToPriorityQueue(RequestManager.java:261)
at weblogic.work.RequestManager.executeIt(RequestManager.java:235)
at weblogic.work.ServerWorkManagerImpl.schedule(ServerWorkManagerImpl.java:142)
at weblogic.timers.internal.TimerManagerImpl.execute(TimerManagerImpl.java:631)
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:285)

2、

####<2014-6-28 上午07时14分45秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403910885281> <BEA-002911> <WorkManager direct failed to schedule a request due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space

3、

####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200004> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "638" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200007> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

针对这些错误做出联合整改方案具体如下：

1、针对weblogic.utils.UnsyncCircularQueue$FullQueueException通过官方文档查看为weblogic bug 需打补丁包：Patch 8179406

WebLogic - Admin Server Stops Responding With "weblogic.utils.UnsyncCircularQueue$FullQueueException" Exceptions (文档 ID 1228224.1)

打补丁步骤：

（1）修改/apps/weblogic/bea/weblogic92/common/bin/commEnv.sh,在文件开始处添加export CLASSPATH=/apps/weblogic/bea/weblogic92/Patches/p8179406.jar:$CLASSPATH
（2）将p8179406.jar补丁拷贝到apps/weblogic/bea/weblogic92/Patches目录下，然后重启WebLogic

2、针对错误3 点击Adminserver -> 协议 -> 常规 -> 启用隧道 ，然后重启服务

通过这两个解决方案目前观察2个月adminserver运行稳定。没有出现类似错误。

补丁下载：     http://pan.baidu.com/s/1c0gr6M0

bug文档：    http://pan.baidu.com/s/1c0H5r7y

可参考的连接：https://community.oracle.com/thread/694983?start=0&tstart=0

http://v-vampires.iteye.com/blog/861897

weblogic诊断案例-AdminServer平均1-2周崩溃的更多相关文章

  1. Linux SendMail发送邮件失败诊断案例(三)

    一Linux服务器突然发送不出邮件,检查了很多地方都没有发现异常,检查/var/log/maillog发现如下具体信息: Apr 12 00:36:04 mylinux sendmail[4685]: ...

  2. 深入浅出Oracle:DBA入门、进阶与诊断案例（读书笔记1）

    一.数据库的启动和关闭 Oracle Server共有2部分组成:Instance和Database. Instance是指一组后台进程/线程和一块共享内存区域,而Database是指存储在磁盘上的一 ...

  3. Linux sendmail发送邮件失败诊断案例(一)

    在新服务器上测试sendmail发送邮件时,发现邮件发送不成功,检查日志文件发现如下错误(Notice:由于涉及公司服务器,邮箱等,故下面hostname.邮箱地址等信息使用xxx代替) tail - ...

  4. Linux SendMail发送邮件失败诊断案例(二)

    Linux上Sendmail经常由于一些配置问题,导致邮件发送失败,下面整理.收集了一些邮件发送失败.异常的案例. 案例1:在新服务器上测试sendmail发送邮件时,发现邮件发送不成功,检查/var ...

  5. Netbeans8下 Weblogic EJB案例

    1:接口 @Remote public interface XgmZzsNssb {} 2:实现 @Stateless(mappedName="XgmZzsNssbImpl") @ ...

  6. Linux SendMail发送邮件失败诊断案例(四）

    最近又碰到一起Linux下SendMail发送邮件失败的案例,邮件发送后,邮箱收不到具体邮件, 查看日志/var/log/maillog 发现有"DSN: User unknown" ...

  7. 深入浅出Oracle:DBA入门、进阶与诊断案例（读书笔记2)

    第5章  Buffer Cache与Shared Pool原理 5.1 Buffer Cache原理 Buffer Cache是Oracle SGA中的一个重要部分,通常的数据访问和修改都需要通过Bu ...

  8. 转 Linux SendMail发送邮件失败诊断案例(四）

    http://www.cnblogs.com/kerrycode/p/7826036.html

  9. 深入浅出Oracle：DBA入门、进阶与诊断案例 PDF 下载

    网盘地址: 链接:https://pan.baidu.com/s/1tMFoNSUW7ICKOtmSQ5ZazA 提取码:dbnc

随机推荐

  1. VC ++ MFC activex 控件获取连接的VPN 信息

    vc++  MFC 进行activex  控件的开发步骤就不用多写了,只是简单的说明一下方法,以及具体的代码: 使用的类库是 windows 系统的 rasapi32.dll 记住需要添加的头文件如下 ...

  2. 【转】object标签和embed标签

    我们现在大部分人做网页,都是直接用DW插入flash,而且DW也是所见即所得,直接生成了相应的flash显示代码.可是我们又有多少人了解这些直接由DW生成的代码呢?其实我接触flash player标 ...

  3. JQuery Pagenation 知识点整理——phototype 应用（20150517）(转)

    JS中的phototype是JS中比较难理解的一个部分 本文基于下面几个知识点: 1 原型法设计模式 在.Net中可以使用clone()来实现原型法 原型法的主要思想是,现在有1个类A,我想要创建一个 ...

  4. elasticsearch 集群配置

    2015-10-10 09:56 by 轩脉刃, 999 阅读, 1 评论, 收藏, 编辑 elasticsearch 集群 搭建elasticsearch的集群 现在假设我们有3台es机器,想要把他 ...

  5. javascript取得机器名，用户名，读写注册表，启动应用程序

    javascript取得机器名,用户名,读写注册表,启动应用程序//javascript有个特殊的对象ActiveXObject,通过它可以访问windows的本地文件系统和应用程序,比如:有的时候我 ...

  6. ubuntu 创建快捷方式

    sudo ln -s /opt/eclipse/eclipse /usr/bin/eclipse 这样就可以在命令行中敲:eclipse,来打开eclipse了. 应该还有一种方式,就像快捷方式一样. ...

  7. windows p12(pfx)个人证书安装过程

    证书库个人证书存储区为其中的每个证书维护一个属性CERT_KEY_PROV_INFO_PROP_ID,该属性指定了证书对应的密钥容器的相关信息,包括密钥容器名,CSP名称,CSP类型,密钥用途,以及C ...

  8. php缓存数组到文件

    php缓存数组到文件 static function getIDs($kemuid) { $cachefile="cache/" . $kemuid . ".cache& ...

  9. HDMI接口与协议

    深入了解HDMI接口 一.HDMI接口的工作原理这张图是HDMI接口的架构示意图.从左边的信号源中你可以看到,HDMI接口的信源可以是任何支持HDMI输出的设备,而接入端也可以是任何带有HDMI输 入 ...

  10. 黄聪：Discuz!X/数据库操作方法、DB::table、C::t

    函数 功能 DB::table($tablename) 获取正确带前缀的表名,转换数据库句柄, DB::delete($tablename, 条件,条数限制) 删除表中的数据 DB::insert($ ...