困扰我多年的Connection reset问题

第一次出现：是thrift的python client去请求server，发现偶尔出现这个问题

第二次：接入第三方的api，去请求数据时，发现一个接入方的api第一次总是报这个错，当时又没有做处理，导致获得信息置空，入缓存后数据就是错误的。做了一个更改就是retry三次，得到解决。

第三次：最近去抓appstore的应用指数又重新出现该问题，使用HttpRequestRetryHandler 重试，设置到20次都无一次成功。

堆栈错误信息：

[app][index-error]: ScreenAnts HD ; priority empty
2014-01-26 14:59:30,668 - I/O exception (java.net.SocketException) caught when processing request: Connection reset
2014-01-26 14:59:30,703 - Retrying request
2014-01-26 14:59:30,668 - [SimpleHttpClient] myRetryHandler, IOException:Connection reset
2014-01-26 14:59:30,668 - I/O exception (java.net.SocketException) caught when processing request: Connection reset
2014-01-26 14:59:30,704 - Retrying request
2014-01-26 14:59:30,668 - I/O exception (java.net.SocketException) caught when processing request: Connection reset
2014-01-26 14:59:30,704 - Retrying request
2014-01-26 14:59:30,704 - [SimpleHttpClient] myRetryHandler, IOException:Connection reset
2014-01-26 14:59:30,704 - [SimpleHttpClient] read http://search.itunes.apple.com/WebObjects/MZSearchHints.woa/wa/hints?q=Deadliest+Animals failed
java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:185)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
	at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.http.impl.conn.CPoolProxy.invoke(CPoolProxy.java:138)
	at $Proxy7.receiveResponseHeader(Unknown Source)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:253)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at com.fanxer.aso.index.utils.SimpleHttpClient.getContent(SimpleHttpClient.java:135)
	at com.fanxer.aso.index.task.AppIndexTask.getXmlBytesFromSearchLink(AppIndexTask.java:163)
	at com.fanxer.aso.index.task.AppIndexTask.access$000(AppIndexTask.java:40)
	at com.fanxer.aso.index.task.AppIndexTask$1.call(AppIndexTask.java:98)
	at com.fanxer.aso.index.task.AppIndexTask$1.call(AppIndexTask.java:93)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:679)

　　

http://bbs.csdn.net/topics/210061352 解释了四种socket异常：

java.net.SocketException: （Connection reset或者Connect reset by peer:Socket write error）。该异常在客户端和服务器端均有可能发生，引起该异常的原因有两个，第一个就是如果一端的Socket被关闭（或主动关闭或者因为异常退出而引起的关闭），另一端仍发送数据，发送的第一个数据包引发该异常(Connect reset by peer)。另一个是一端退出，但退出时并未关闭该连接，另一端如果在从连接中读数据则抛出该异常（Connection reset）。简单的说就是在连接断开后的读和写操作引起的。

经多次测试发现，50个线程并发，最大的连接时间超过了90秒，平均请求结果仅有400KB，很奇怪的现象。猜测是appstore端连接时间过长直接断开连接（是我被连90s也要断啊）。修改下超时，只能让请求更快恢复，

RetryExec.execute 时仍然无法正常连接。

查看源码，InternalHttpClient继承了CloseableHttpClient 实现了父类的抽象方法

protected abstract CloseableHttpResponse doExecute(HttpHost target, HttpRequest request,
HttpContext context) throws IOException, ClientProtocolException;

对request做了封装，host、config和route确定后，转入execChain

execChain是一系列责任链

RedirectExec ->RetryExec ->ProtocolExec->MainClientExec

转给 HttpRequestExecutor 执行请求, 通过DefaultBHttpClientConnection把结果写入response，看程序没发现问题，终于无意中使用curl做测试，发现也是失败

curl: (56) Failure when receiving data from the peer

目前只能通过降低请求频率或优化降低请求次数或者发现错误时多停顿一段时间去解决。