Java访问Elasticsearch报错Request cannot be executed; I/O reactor status: STOPPED
简介
使用ES过程中遇到一个Request cannot be executed; I/O reactor status: STOPPED 的异常，大概意思是和server端的连接异常终止了。开始以为是引用的版本不对，或者自己使用问题，后来发现就是因为OOM导致程序宕机，进而引发连接终止。
环境
功能
SpringBoot 的程序通过 SpringDataElasticsearch 访问ES-server 获取数据。
ES-SERVER
- 版本：7.15.2
ES-CLIENT
ES-CLIENT 就是 SpringBoot 程序，核心pom依赖：
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
<version>2.5.3</version>
</dependency>
问题
- 部署到测试环境之后，用户量大了之后接口报错，报错信息为：
Request cannot be executed; I/O reactor status: STOPPED
- 后端程序处于假死状态，访问其他接口一直转圈，无法响应内容
- 重启程序之后一切正常，用户量大了之后又有此问题
- 详细报错日志为：
2021-12-31/19:26:31.748||||^_^|[pool-4-thread-1] ERROR o.a.http.impl.nio.client.InternalHttpAsyncClient 66 - I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.OutOfMemoryError: Java heap space
2021-12-31/19:26:32.783||||^_^|[http-nio-8092-exec-3] ERROR com.nghsmart.datacenter.handler.ExceptionHandler 67 - handleRuntimeException:
java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:888)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1624)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1594)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1110)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.lambda$search$11(ElasticsearchRestTemplate.java:296)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:383)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.search(ElasticsearchRestTemplate.java:296)
at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.search(AbstractElasticsearchTemplate.java:513)
at com.nghsmart.datacenter.domain.area.service.impl.AreaDashboradServiceImpl.getRotationStatisticsInfo(AreaDashboradServiceImpl.java:166)
at com.nghsmart.datacenter.domain.area.controller.AreaDashboradController.getRotationStatisticsInfo(AreaDashboradController.java:76)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at com.nghsmart.commonauthentication.filter.JwtAuthenticationTokenFilter.doFilterInternal(JwtAuthenticationTokenFilter.java:52)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at org.apache.http.util.Asserts.check(Asserts.java:46)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:279)
... 69 common frames omitted
- 测试环境用户量一大就报，本地IDEA中一次也没有报这个问题
- 使用JMETER设置50个线程并发访问可以稳定复现此问题
原因
程序接口中将一块很大的数据存进JAVA集合中引发了oom，oom异常导致程序宕机，处于假死状态，进而导致 ES-CLIENT 和 ES-SERVER 端的 http 连接异常终止，然后org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning 方法报异常。
ensureRunning:
protected void ensureRunning() {
final Status currentStatus = this.status.get();
Asserts.check(currentStatus == Status.ACTIVE, "Request cannot be executed; " +
"I/O reactor status: %s", currentStatus);
}
check:
public static void check(final boolean expression, final String message, final Object arg) {
if (!expression) {
throw new IllegalStateException(String.format(message, arg));
}
}
- SpringDataElasticsearch 和ES-SERVER 是长链接，只要报了OOM，当前和 ES-SERVER 的连接线程都将报异常，也就是说，虽然OOM只报了一次，但是可能有多个线程都在 Asserts.check 方法中报异常
- 其实不仅仅是和ES-SERVER 的连接异常关闭，观察大量日志后发现和nacos 的连接也有问题：
2022-01-04 15:59:07.366 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client.printIfInfoEnabled:60 - [117be10e-119f-4253-a348-71c95a8978a1]Server healthy check fail,currentConnection=1641282430796_192.168.1.90_43108
2022-01-04 15:59:07.367 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client.printIfInfoEnabled:60 - [117be10e-119f-4253-a348-71c95a8978a1] try to re connect to a new server ,server is not appointed,will choose a random server.
- 测试环境运行SpringBoot内存配置为：-Xms256m -Xmx512m，使用 java -jar 命令启动。测试环境能复现，本地idea无法复现的原因是内存配置不同导致的，本地idea内存给的2g。编辑 run/debug configuration ，设置 VM options 和测试环境相同，使用jemter压测本地idea中跑的程序，也能复现此问题
解决办法
- 优化接口，解决OOM问题
- 增大测试环境内存配置
解决过程
HttpClient问题
这个思路并没有解决我的问题，但是解决问题的过程中花了大量时间研究。其实应该仔细阅读报错log的上下文和接口代码上下文，一开始没有留意到oom的异常
大概意思是说低版本 httpClient 有这个问题，高版本就没有这个问题，是apache-httpclient 导致的。具体是因为捕获到线程异常之后http连接就主动终止了，解决思路是升级版本或者是重写处理器，遇到异常之后不终止连接。
升级httpclient相关pom依赖：
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
<!-- 排除 httpclient、httpcore、 httpcore-nio这三个依赖，并手动引入最新版本，解决es客户端报Request cannot be executed; I/O reactor status: STOPPED问题 -->
<exclusions>
<!-- <exclusion>-->
<!-- <groupId>org.apache.httpcomponents</groupId>-->
<!-- <artifactId>httpclient</artifactId>-->
<!-- </exclusion>-->
<!-- <exclusion>-->
<!-- <groupId>org.apache.httpcomponents</groupId>-->
<!-- <artifactId>httpcore</artifactId>-->
<!-- </exclusion>-->
<!-- <exclusion>-->
<!-- <groupId>org.apache.httpcomponents</groupId>-->
<!-- <artifactId>httpcore-nio</artifactId>-->
<!-- </exclusion>-->
<exclusion>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- start -->
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>7.15.2</version>
<exclusions>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore-nio</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- end -->
<!-- start：排除 httpclient、httpcore、 httpcore-nio这三个依赖，并手动引入最新版本，解决es客户端报Request cannot be executed; I/O reactor status: STOPPED问题 -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.13</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore-nio</artifactId>
<version>4.4.14</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.4.14</version>
</dependency>
<!-- end -->
重写ElasticsearchRestTemplate：
import com.alibaba.fastjson.JSON;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.Header;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.DefaultConnectionKeepAliveStrategy;
import org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager;
import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor;
import org.apache.http.message.BasicHeader;
import org.apache.http.nio.reactor.IOReactorException;
import org.apache.http.nio.reactor.IOReactorExceptionHandler;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@Slf4j
@Configuration
public class EsClientConfig {
@Value("${es.host}")
private String host;
@Value("${es.port}")
private Integer port;
@Bean
public ElasticsearchRestTemplate initElasticsearchRestTemplate() {
RestClientBuilder builder = RestClient.builder(new HttpHost(host, port))
.setRequestConfigCallback(
config -> config.setConnectTimeout(180000)
.setConnectionRequestTimeout(180000)
.setSocketTimeout(180000))
.setHttpClientConfigCallback(
httpClientBuilder -> {
httpClientBuilder.setMaxConnTotal(100);
httpClientBuilder.setMaxConnPerRoute(50);
List<Header> headers = new ArrayList<>(2);
headers.add(new BasicHeader("Connection", "keep-alive"));
headers.add(new BasicHeader("Keep-Alive", "720"));
httpClientBuilder.setDefaultHeaders(headers);
httpClientBuilder.setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE);
try {
DefaultConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
ioReactor.setExceptionHandler(new IOReactorExceptionHandler() {
@Override
public boolean handle(IOException e) {
log.debug("System may be unstable: IOReactor encountered a checked exception : " + e.getMessage());
log.debug("start setHttpClientConfigCallback handle IOException e printStackTrace");
e.printStackTrace();
log.debug("end setHttpClientConfigCallback handle IOException e printStackTrace");
// Return true to note this exception as handled, it will not be re-thrown
return true;
}
@Override
public boolean handle(RuntimeException e) {
log.debug("System may be unstable: IOReactor encountered a runtime exception : " + e.getMessage() + ",e is {}", e);
log.debug("start setHttpClientConfigCallback handle RuntimeException e printStackTrace ");
e.printStackTrace();
log.debug("end setHttpClientConfigCallback handle RuntimeException e printStackTrace ");
// Return true to note this exception as handled, it will not be re-thrown
return true;
}
});
httpClientBuilder.setConnectionManager(new PoolingNHttpClientConnectionManager(ioReactor));
} catch (IOReactorException e) {
throw new RuntimeException(e);
}
return httpClientBuilder;
}
);
ElasticsearchRestTemplate elasticsearchRestTemplate = new ElasticsearchRestTemplate(new RestHighLevelClient(builder));
return elasticsearchRestTemplate;
}
}
关于此问题相关参考：
- https://www.cnblogs.com/yangchongxing/p/15440197.html
- https://github.com/elastic/elasticsearch/issues/42133
- https://zhuanlan.zhihu.com/p/384269417
- https://cloud.tencent.com/developer/article/1806886
Java访问Elasticsearch报错Request cannot be executed; I/O reactor status: STOPPED的更多相关文章
- 【ElasticSearch】异常 Request cannot be executed; I/O reactor status: STOPPED
Caused by: java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED at or ...
- java 连接 elasticsearch 报错java.lang.NoClassDefFoundError: org/apache/http/auth/Credentials 解决
您的问题是您在应用程序类路径中缺少必需的JAR(这导致ClassNotFound异常).如果您下载了包含IP Camera驱动程序(webcam-capture-driver-ipcam-0.3.10 ...
- elasticsearch报错:None of the configured nodes are available: []
问题:在内网测试的时候可以正常访问,但是部署到外网上客户端连接elasticsearch报错:None of the configured nodes are available: [] 原因:默认情 ...
- Springboot整合Elasticsearch报错availableProcessors is already set to [4], rejecting [4]
Springboot整合Elasticsearch报错 今天使用SpringBoot整合Elasticsearch时候,相关的配置完成后,启动项目就报错了. nested exception is j ...
- vscode点击ctrl键报错Request textDocument/definition failed.
现象 用vscode写java代码的时候突然出现,修复问题点击Ctrl时,输出窗口就打日志,报错Request textDocument/definition failed. 我百度唯一的有用线索就是 ...
- R.java 文件内报错：Underscores can only be used with source level 1.7 or greater。
R.java 文件内报错:Underscores can only be used with source level 1.7 or greater 网上查找后得知是Android工程图片资源命名的问 ...
- java导出excel报错：getOutputStream() has already been called for this response
对于java导出excel报错的问题,查了很多都说是在使用完输出流以后调用以下两行代码即可 out.clear(); out = pageContext.pushBody(); 但这也许是页面上输出时 ...
- Java Web项目报错总结
Java Web项目报错总结 1.java.lang.IllegalStateException java.lang.IllegalStateException Caused by:java.lang ...
- Java类型转化报错
Java类型转化报错 报错如下: java.lang.ClassCastException:java.util.HashMap cannot be cast to java.util.List.
随机推荐
- 【编程思想】【设计模式】【结构模式Structural】3-tier
Pyhon版 https://github.com/faif/python-patterns/blob/master/structural/3-tier.py #!/usr/bin/env pytho ...
- 3.2 go WaitGroup代码示例
sync.WaitGroup提供了一种安全的多协程处理方法,内部使用race.atomic来处理,避免了资源竞争及锁的产生. 主要的方法有Add.Done.Wait,可以等待一组协程全部执行完毕后,主 ...
- 【Vulfocus解题系列】手把手教你使用Vulfocus公开靶场对Apache Log4j2远程命令执行漏洞复现
前言 关于这个漏洞,啥都不用说了,直接发车. 工具下载 JNDIExploit:https://github.com/0x727/JNDIExploit 复现过程 启动靶场环境 直接用vulfocus ...
- 转：Intent 操作常用URI代码示例
以下是常用到的Intent的URI及其示例,包含了大部分应用中用到的共用Intent 一.打开一个网页,类别是Intent.ACTION_VIEW 1 2 Uri uri = Uri.parse(&q ...
- 【论文笔记】 Denoising Implicit Feedback for Recommendation
Denoising Implicit Feedback for Recommendation Authors: 王文杰,冯福利,何向南,聂礼强,蔡达成 WSDM'21 新加坡国立大学,中国科学技术大学 ...
- 数组基础（Excel函数集团）
此处文章均为本妖原创,供下载.学习.探讨! 文章下载源是Office365国内版1Driver,如有链接问题请联系我. 请勿用于商业! 谢谢 下载地址:https://officecommunity- ...
- CF1579A Casimir's String Solitaire 题解
Content 给定一个仅包含 A.B.C 三种字符的字符串 \(s\),有如下两种操作: 删除字符串中的恰好一个 A 和恰好一个 B. 删除字符串中的恰好一个 B 和恰好一个 C. 求是否能够把字符 ...
- CF1547B Alphabetical Strings 题解
Content 我们有一个空的字符串,第 \(i\) 次操作我们可以将字母表中第 \(i\) 个字母加入字符串的最前面或最后面.我们称一个长度为 \(n\) 的字符串是合法的,当且仅当这个字符串可以通 ...
- shell 文件读取批量处理
cat ./redis-keys.txt | while read myline do echo ${myline} /usr/local/bin/redis-cli -h 000.00.00.000 ...
- 总结Vue第一天：简单介绍、基本常用知识、辅助函数
总结Vue第一天:简单介绍.基本常用知识.辅助函数 中文官网:https://cn.vuejs.org/v2/guide/syntax.html 遇到不熟悉的可以先看一下官网,然后再看一下一些别人写的 ...