高并发下hystrix熔断超时及concurrent.RejectedExecutionException: Rejected command because thread-pool queueSize is at rejection threshold问题

我的一个项目并发200+，观察服务器的日志发现出现了大量的熔断超时偶尔会闪现出RejectedExecutionException: Rejected command because thread-pool queueSize is at rejection threshold。

在高并发的前提下出现熔断超时：

1.先确定是否是自己接口的问题，接口平均响应时长是多少？

举个例子，倘若平均响应时长为200ms,单线程处理的话5次/秒，tomcat最大并发线程数按照100个来算的话，那就是100*5次/秒=500次/秒

那也就是正常情况下，可以承受500次/秒的并发请求。

但是现在200次就不停的刷超时熔断！

2.在接口上使用@HystrixCommand注解少配置了参数

之前是按照下面的方式配置的

@HystrixCommand(fallbackMethod = "your_interface", commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "5000") })

注意这里有个坑：

• hystrix.command.default.fallback.isolation.semaphore.maxConcurrentRequests 如果并发数达到该设置值，请求会被拒绝和抛出异常并且fallback不会被调用。默认10
• hystrix.threadpool.default.coreSize 并发执行的最大线程数，默认10
• hystrix.threadpool.default.maxQueueSize BlockingQueue的最大队列数，
• hystrix.threadpool.default.queueSizeRejectionThreshold 即使maxQueueSize没有达到，达到queueSizeRejectionThreshold该值后，请求也会被拒绝。

原来是因为这里如果不显示设置coreSize的化，那么就会按照默认的10来进行处理。这就很好理解为什么大量熔断超时了，10-thread*5次/秒/单线程=50次/秒<200次并发量。

 

然后调整参数，最后解决了这个问题：

@HystrixCommand(fallbackMethod = "otaCheckUpdateFallback", commandProperties = {
        @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "5000"),
        }, threadPoolProperties = { @HystrixProperty(name = "coreSize", value = "100"),
        @HystrixProperty(name = "maxQueueSize", value = "20"),
　　    @HystrixProperty(name = "queueSizeRejectionThreshold", value = "20")})    

同样将参数queueSizeRejectionThreshold调大之后，即便请求频率超过预期，那么当达到queueSizeRejectionThreshold之后会抛出下面的异常。

2018-04-11 20:33:16.294 ERROR [http-nio-9098-exec-30][DirectJDKLog.java:181] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.netflix.hystrix.exception.HystrixRuntimeException: otaCheckUpdate fallback execution rejected.] with root cause
com.netflix.hystrix.exception.HystrixRuntimeException: your_interface fallback execution rejected.
        at com.netflix.hystrix.AbstractCommand.handleFallbackRejectionByEmittingError(AbstractCommand.java:1043) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:875) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at com.netflix.hystrix.AbstractCommand.handleThreadPoolRejectionViaFallback(AbstractCommand.java:993) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at com.netflix.hystrix.AbstractCommand.access$400(AbstractCommand.java:60) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:607) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:601) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar!/:1.1.10]
        at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$3.onError(AbstractCommand.java:1194) ~[hystrix-core-1.5.10.jar!/:1.5.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:54) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]