CAS (15) — CAS 线上环境 Ehcache Replication 的非稳定重现错误 java.util.ConcurrentModificationException
CAS (15) — CAS 线上环境 Ehcache Replication 的非稳定重现错误
摘要
线上环境在 EhCache Replication 过程中出现 java.util.ConcurrentModificationException
2016-12-21 14:58:02,022 ERROR [net.sf.ehcache.distribution.RMISynchronousCacheReplicator] - <Exception on replication of putNotification. null. Continuing...
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) java.util.ConcurrentModificationException
版本
tomcat版本:
tomcat-8.0.29
jdk版本:
jdk1.8.0_65
cas版本: 4.1.3
**cas4.1.3 (4.x还在开发过程中不是很稳定，迭代比较快，也会有些bug) * cas-client-3.4.1
Ehcache版本:
ehcache-2.10.1.jar
ehcache-core-2.6.11.jar
内容
Stack Trace
完整堆栈错误信息
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) 2016-12-21 14:58:02,022 ERROR [net.sf.ehcache.distribution.RMISynchronousCacheReplicator] - <Exception on replication of putNotification. null. Continuing...
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) java.util.ConcurrentModificationException
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at java.util.HashMap$EntryIterator.next(HashMap.java:962)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at java.util.HashMap$EntryIterator.next(HashMap.java:960)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at java.util.HashMap.writeObject(HashMap.java:1127)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at sun.reflect.GeneratedMethodAccessor2095.invoke(Unknown Source)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at java.lang.reflect.Method.invoke(Method.java:606)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
14:58:02,023 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:440)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.Element.writeObject(Element.java:835)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at sun.reflect.GeneratedMethodAccessor2108.invoke(Unknown Source)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.lang.reflect.Method.invoke(Method.java:606)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:293)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:152)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.distribution.RMICachePeer_Stub.put(Unknown Source)
14:58:02,024 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicatePutNotification(RMISynchronousCacheReplicator.java:149)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.notifyElementPut(RMISynchronousCacheReplicator.java:132)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.event.RegisteredEventListeners.notifyListener(RegisteredEventListeners.java:294)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.event.RegisteredEventListeners.invokeListener(RegisteredEventListeners.java:284)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.event.RegisteredEventListeners.internalNotifyElementPut(RegisteredEventListeners.java:144)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.event.RegisteredEventListeners.notifyElementPut(RegisteredEventListeners.java:122)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.Cache.notifyPutInternalListeners(Cache.java:1481)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.Cache.putInternal(Cache.java:1456)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.Cache.put(Cache.java:1383)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at net.sf.ehcache.Cache.put(Cache.java:1348)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.cas.ticket.registry.EhCacheTicketRegistry.addTicket_aroundBody0(EhCacheTicketRegistry.java:103)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.cas.ticket.registry.EhCacheTicketRegistry$AjcClosure1.run(EhCacheTicketRegistry.java:1)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.inspektr.aspect.TraceLogAspect.traceMethod(TraceLogAspect.java:44)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.cas.ticket.registry.EhCacheTicketRegistry.addTicket(EhCacheTicketRegistry.java:100)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket_aroundBody2(CentralAuthenticationServiceImpl.java:319)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.cas.CentralAuthenticationServiceImpl$AjcClosure3.run(CentralAuthenticationServiceImpl.java:1)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.inspektr.aspect.TraceLogAspect.traceMethod(TraceLogAspect.java:44)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:241)
14:58:02,025 INFO [stdout] (http-/0.0.0.0:8443-7) at sun.reflect.GeneratedMethodAccessor214.invoke(Unknown Source)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at java.lang.reflect.Method.invoke(Method.java:606)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.inspektr.audit.AuditTrailManagementAspect.handleAuditTrail(AuditTrailManagementAspect.java:128)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at java.lang.reflect.Method.invoke(Method.java:606)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at com.ryantenney.metrics.spring.MeteredMethodInterceptor.invoke(MeteredMethodInterceptor.java:45)
14:58:02,026 INFO [stdout] (http-/0.0.0.0:8443-7) at com.ryantenney.metrics.spring.MeteredMethodInterceptor.invoke(MeteredMethodInterceptor.java:32)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:59)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:48)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:34)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:59)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at com.sun.proxy.$Proxy57.grantServiceTicket(Unknown Source)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:57)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.action.EvaluateAction.doExecute(EvaluateAction.java:77)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.execution.AnnotatedAction.execute(AnnotatedAction.java:145)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:101)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.State.enter(State.java:194)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.Transition.execute(Transition.java:228)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:51)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.State.enter(State.java:194)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.Transition.execute(Transition.java:228)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:51)
14:58:02,027 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.State.enter(State.java:194)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.Transition.execute(Transition.java:228)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:395)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:116)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.Flow.handleEvent(Flow.java:547)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:390)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:105)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.State.enter(State.java:194)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.Flow.start(Flow.java:527)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.impl.FlowExecutionImpl.start(FlowExecutionImpl.java:368)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.engine.impl.FlowExecutionImpl.start(FlowExecutionImpl.java:223)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.executor.FlowExecutorImpl.launchExecution(FlowExecutorImpl.java:140)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:238)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
14:58:02,028 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.cas.security.RequestParameterPolicyEnforcementFilter.doFilter(RequestParameterPolicyEnforcementFilter.java:250)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jasig.inspektr.common.web.ClientInfoThreadLocalFilter.doFilter(ClientInfoThreadLocalFilter.java:62)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
14:58:02,029 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) at java.lang.Thread.run(Thread.java:745)
14:58:02,030 INFO [stdout] (http-/0.0.0.0:8443-7) >
问题排查
定位问题发生的场景
该错误出现在生成 ticket 之后往 Ehcache 中 put 相应的缓存记录。
对比当前版本（2.10.1）的 RMISynchronousCacheReplicator 与最新版本（2.10.3）的修改
发现官方并没有对出现问题的类做过任何 bug 修复
怀疑是否有配置错误？
解决方案
根据 forums.terracotta.org 的建议，将 ehcache 的 copyOnRead 设置为 true 。
可以在 Cache Configuration: copyOnRead and copyOnWrite cache configuration 找到相应的配置示例。
<cache name="copyCache"
maxEntriesLocalHeap="10"
eternal="false"
timeToIdleSeconds="5"
timeToLiveSeconds="10"
copyOnRead="true"
copyOnWrite="true">
<persistence strategy="none"/>
<copyStrategy class="com.company.ehcache.MyCopyStrategy"/>
</cache>
在 CAS 的 ticketRegistry.xml 中对 cas_st 与 cas_tgt 进行修改。
<bean id="abstractTicketCache" abstract="true"
class="org.springframework.cache.ehcache.EhCacheFactoryBean"
p:cacheManager-ref="cacheManager"
p:diskExpiryThreadIntervalSeconds="0"
p:diskPersistent="false"
p:eternal="false"
p:maxElementsInMemory="50000"
p:maxElementsOnDisk="0"
p:memoryStoreEvictionPolicy="LRU"
p:overflowToDisk="false"
p:copyOnRead="true" <== 设置 copyOnRead
p:bootstrapCacheLoader-ref="ticketCacheBootstrapCacheLoader" />
<bean id="serviceTicketsCache"
class="org.springframework.cache.ehcache.EhCacheFactoryBean"
parent="abstractTicketCache"
p:cacheName="cas_st"
p:timeToIdle="0"
p:timeToLive="300"
p:cacheEventListeners-ref="ticketRMISynchronousCacheReplicator" />
<bean id="ticketGrantingTicketsCache"
class="org.springframework.cache.ehcache.EhCacheFactoryBean"
parent="abstractTicketCache"
p:cacheName="cas_tgt"
p:timeToIdle="7201"
p:timeToLive="0"
p:cacheEventListeners-ref="ticketRMIAsynchronousCacheReplicator" />
备选应急方案
如果生产环境的集群仍然出现此错误，可以通过轮流重启不同节点的服务器暂时解决此问题。
参考
参考来源:
CAS Protocol 3.0 Specification
Shiro Eh Cache ConcurrenctModificationException
ConcurrentModificationException when putting an ArrayList as an Element
结束
CAS (15) — CAS 线上环境 Ehcache Replication 的非稳定重现错误 java.util.ConcurrentModificationException的更多相关文章
- 使用Fabric一键批量部署上线/线上环境监控
本文讲述如何使用fabric进行批量部署上线的功能 这个功能对于小应用,可以避免开发部署上线的平台,或者使用linux expect开发不优雅的代码. 前提条件: 1.运行fabric脚本的机器和其他 ...
- 【微信小程序】---线上环境搭建
一.前言 通常我们在本地电脑上开发微信小程序,调用和访问小程序会有很多问题.特别是在配有自己后端的情况下,我们通过真机访问我们的小程序会出现不可访问的问题 二.线上环境搭建 在这里我们主要以腾讯云给大 ...
- robot framework 测试/预发/线上环境快捷切换
通常情况下布署的三套环境:测试.预发及线上环境.调试或者辅助验证测试时,切环境改变量甚是麻烦.这些变量包括但不限于:一些url信息,数据库信息,预置用户信息等. 切换环境方法一:使用变量文件,通过判断 ...
- 线上环境HBASE-1.2.0出现oldWALs无法自动回收情况；
正常情况下,hmaster会定期清理oldWALs文件夹,一般该文件大小也就几百兆,但是我们线上 环境出现了该文件没有自动回收情况,如图: 该目录占用hdfs空间多达7.6T,浪费空间: 后来经过多番 ...
- vue本地和线上环境（域名）配置
vue本身为运行脚手架项目自家搭载了一个nodejs后台环境,本地可通过proxyTable来处理跨域问题,但是上线(或生产环境)之后改域名真是一件麻烦的事情,所以进行一些配置. config/ind ...
- wechat开发笔记之1.线上环境搭建与测试
Wechat开发笔记 线上环境搭建: 申请一个wechat公众平台. 手机个人微信可以用webwechat来测试. Website:https://web.weixin.qq.com/ 手机客户端扫一 ...
- Vue 2.x 3.x 配置项目开发环境跟线上环境
先找到package.json (这是nuxt版的vue 可能会跟一般vue不一样 当然总体上差不多的) "scripts": { "dev": " ...
- Docker + node(koa) + nginx + mysql 线上环境部署
在上一篇 Docker + node(koa) + nginx + mysql 开发环境搭建,我们进行了本地开发环境搭建 现在我们就来开始线上环境部署 如果本地环境搭建没有什么问题,那么线上部署的配置 ...
- express框架开发接口部署线上环境PM2
1.PM2介绍 PM2是一个线上环境下,用于启动nodejs进程守护的工具,用来保证服务的稳定及分摊服务器进程和压力. 2.下载安装 npm install pm2 -g => pm2 --v ...
随机推荐
- Python3练习题系列（08）——代码阅读方法及字典跳转表理解
问题:分析下面代码 cities['_find'] = find_city city_found = cities['_find'](cities, state) 分析过程: 一个函数也可以作为一个变 ...
- iOS Application Extension
链接: iOS App Extension入门 iOS10通知及通知拓展Extension使用详解(附Demo) iOS10 通知extension之 Content Extension
- [转]delphi 有授权许可的字符串拷贝函数源码
一段看上去“貌不惊人”的Delphi插入汇编代码,却需要授权许可,但是与经典的同类型函数比较,确实“身手不凡”. 研究代码的目的在于借鉴,本文通过分析,并用C++重写代码进行比较,再次证明这段代码效率 ...
- 【linux】linux下对java程序生成dump文件，并使用IBM Heap Analyzer进行分析，查找定位内存泄漏的问题代码
1.首先,java程序启动在linux,怎么生成dump文件? 1>第一步,首先你需要得到java程序的PID,最简单的方法使用如下命令 ps -ef|grep java 或者如果是docker ...
- 这可能是最好的RxJava 2.x 入门教程（一）
这可能是最好的 RxJava 2.x 入门教程系列专栏 文章链接: 这可能是最好的 RxJava 2.x 入门教程(完结版)[重磅推出] 这可能是最好的RxJava 2.x 入门教程(一) 这可能是最 ...
- Linux下阅读MHT文件
Linux下阅读MHT文件 2016年02月04日 15:11:32 dj0379 阅读数:1769 mht是一种WEB电子邮件档案,用IE选择保存网页类型时可以看到有这一项,其最大优点是所保存的 ...
- jconsole工具使用
Jconsole,Java Monitoring and Management Console. Jconsole是JDK自带的监控工具,在JDK/bin目录下可以找到.它用于连接正在运行的本地或者远 ...
- Java时间串获取(格式:yyyyMMddHHmmss)
DateFormat df = new SimpleDateFormat("yyyyMMddHHmmss");Calendar calendar = Calendar.getI ...
- 解决IDEA授权报错
今天打开电脑,猛然发现IDEA授权失效,然后重新用账号密码登陆,发现被拒绝,各种百度百思不得其解,抱着试试的态度,点击了重置密码 https://account.jetbrains.com/forgo ...
- C++开源项目等收集
VLC 是一款自由.开源的跨平台多媒体播放器及框架,可播放大多数多媒体文件,以及 DVD.音频 CD.VCD 及各类流媒体协议. Downloading vlc-2.2.4.tar.xz Thanks ...