在项目中常用的HttpClient,与我们非常的亲密,为了能处理遇到的Http问题,我们应该去了解里面的机制和实现。  

  官方文档:http://hc.apache.org/httpcomponents-client-ga/tutorial/html/

  Maven

  1. <!-- components.httpclient-->
  2. <dependency>
  3. <groupId>org.apache.httpcomponents</groupId>
  4. <artifactId>httpclient</artifactId>
  5. <version>4.5.5</version>
  6. </dependency>

  HttpClient的看点主要是它的责任链设计、连接池机制、工具类的封装,个人觉得它设计得还比较优雅。

  我们从一个简单的Http请求,就可以步步深入得去阅读它的源码。

  1. HttpGet get = new HttpGet("http://localhost:8080/hello/say");
  2. RequestConfig config = RequestConfig.custom()
  3. .setConnectTimeout(3000)
  4. .setSocketTimeout(3000)
  5. .build();
  6. get.setConfig(config);
  7. HttpClient client = HttpClientBuilder.create()
  8. .setMaxConnTotal(1<<6)
  9. .setMaxConnPerRoute(1<<3)
  10. .evictExpiredConnections()
  11. .build();
  12. HttpResponse response = client.execute(get);
  13. System.out.println(EntityUtils.toString(response.getEntity()));

  首先,发起一个Get请求,它先要实例化请求方法,然后设置请求的配置(连接时间、读取时间)等,然后使用HttpClient的Builder实例化一个按照你配置的HttpClient,最后发起请求拿到响应,在解析响应的时候,它提供了EntityUtils帮你解析响应体。

  当我们从它的execute方法进去的时候,它是一个抽象类CloseableHttpClient;它封装了各式各样发起请求的方法,可以说是走入核心之前的外部接口封装,它的真正功能在doExecute方法里面,这个方法是抽象的,取决与Builder在创建实例的时候,实现类或者内部类实现的这个方法。

   我们通过debug可以知道,它实际上是走入了InternalHttpClient中,这个时候已经走入了责任链的头部了,开始走入设计者想好的路径了;这个类的主要作用是封装内部调用使用的Request和Config,在它的return方法中,可以看出它将走入下一个execChain。

  1. protected CloseableHttpResponse doExecute(HttpHost target, HttpRequest request, HttpContext context) throws IOException, ClientProtocolException {
  2. Args.notNull(request, "HTTP request");
  3. HttpExecutionAware execAware = null;
  4. if(request instanceof HttpExecutionAware) {
  5. execAware = (HttpExecutionAware)request;
  6. }
  7.  
  8. try {
  9. HttpRequestWrapper wrapper = HttpRequestWrapper.wrap(request, target);
  10. HttpClientContext localcontext = HttpClientContext.adapt((HttpContext)(context != null?context:new BasicHttpContext()));
  11. RequestConfig config = null;
  12. if(request instanceof Configurable) {
  13. config = ((Configurable)request).getConfig();
  14. }
  15.  
  16. if(config == null) {
  17. HttpParams params = request.getParams();
  18. if(params instanceof HttpParamsNames) {
  19. if(!((HttpParamsNames)params).getNames().isEmpty()) {
  20. config = HttpClientParamConfig.getRequestConfig(params, this.defaultConfig);
  21. }
  22. } else {
  23. config = HttpClientParamConfig.getRequestConfig(params, this.defaultConfig);
  24. }
  25. }
  26.  
  27. if(config != null) {
  28. localcontext.setRequestConfig(config);
  29. }
  30.  
  31. this.setupContext(localcontext);
  32. HttpRoute route = this.determineRoute(target, wrapper, localcontext);
  33. return this.execChain.execute(route, wrapper, localcontext, execAware);
  34. } catch (HttpException var9) {
  35. throw new ClientProtocolException(var9);
  36. }
  37. }

   再往下走,它将走入RetryClient;这个类的主要作用是控制重试,当下层链出现超时的时候会进行重试。这个重试的次数是之前你创建实例的时候可以指定的,没有指定它也有默认值。

  1. @Override
  2. public CloseableHttpResponse execute(
  3. final HttpRoute route,
  4. final HttpRequestWrapper request,
  5. final HttpClientContext context,
  6. final HttpExecutionAware execAware) throws IOException, HttpException {
  7. Args.notNull(route, "HTTP route");
  8. Args.notNull(request, "HTTP request");
  9. Args.notNull(context, "HTTP context");
  10. final Header[] origheaders = request.getAllHeaders();
  11. for (int execCount = 1;; execCount++) {
  12. try {
  13. return this.requestExecutor.execute(route, request, context, execAware);
  14. } catch (final IOException ex) {
  15. if (execAware != null && execAware.isAborted()) {
  16. this.log.debug("Request has been aborted");
  17. throw ex;
  18. }
  19. if (retryHandler.retryRequest(ex, execCount, context)) {
  20. if (this.log.isInfoEnabled()) {
  21. this.log.info("I/O exception ("+ ex.getClass().getName() +
  22. ") caught when processing request to "
  23. + route +
  24. ": "
  25. + ex.getMessage());
  26. }
  27. if (this.log.isDebugEnabled()) {
  28. this.log.debug(ex.getMessage(), ex);
  29. }
  30. if (!RequestEntityProxy.isRepeatable(request)) {
  31. this.log.debug("Cannot retry non-repeatable request");
  32. throw new NonRepeatableRequestException("Cannot retry request " +
  33. "with a non-repeatable request entity", ex);
  34. }
  35. request.setHeaders(origheaders);
  36. if (this.log.isInfoEnabled()) {
  37. this.log.info("Retrying request to " + route);
  38. }
  39. } else {
  40. if (ex instanceof NoHttpResponseException) {
  41. final NoHttpResponseException updatedex = new NoHttpResponseException(
  42. route.getTargetHost().toHostString() + " failed to respond");
  43. updatedex.setStackTrace(ex.getStackTrace());
  44. throw updatedex;
  45. } else {
  46. throw ex;
  47. }
  48. }
  49. }
  50. }
  51. }

   继续往下,它到达了MainClientExec类;这个类厉害了,控制着连接池的获取、socket连接的建立、链接的释放,可以说是HttpClient的核心了。它的下层调用是HttpRequestExecutor,里面控制着Http请求头的发送,Http Request Line的发送,以及响应的收集。至此,我们可以梳理一个路径。

   它使用了责任链进行拼装,并且每个链条上的抽象很干净,只做它负责的范围的工作。而它们这个链条的形成,是在builder里面组装的。

    我们把调用链抽象出来,结合builder,它的设计是比较优雅的。

  1. package execChain;
  2.  
  3. public interface ExecutionChain {
  4. HttpResponse exec(String host,HttpRequest request,HttpContext context);
  5. }
  1. package execChain;
  2.  
  3. import java.io.IOException;
  4.  
  5. public interface HttpClient {
  6. HttpResponse execute(String route,HttpRequest request,HttpContext context)
  7. throws IOException;
  8. }
  1. package execChain;
  2.  
  3. import java.io.IOException;
  4.  
  5. public abstract class TopHttpClient implements HttpClient{
  6. protected abstract HttpResponse doExecute(String host,HttpRequest request,HttpContext context);
  7.  
  8. @Override
  9. public HttpResponse execute(String route, HttpRequest request, HttpContext context) throws IOException {
  10. return doExecute(route,request,context);
  11. }
  12. }
  1. package execChain;
  2.  
  3. public class RetryExecClient extends TopHttpClient{
  4.  
  5. private ExecutionChain requestExecChain;
  6.  
  7. public RetryExecClient(ExecutionChain requestExecChain) {
  8. this.requestExecChain = requestExecChain;
  9. }
  10.  
  11. @Override
  12. protected HttpResponse doExecute(String host, HttpRequest request, HttpContext context) {
  13. return this.requestExecChain.exec(host,request,context);
  14. }
  15. }
  1. package execChain;
  2.  
  3. public class MainExecClient implements ExecutionChain{
  4.  
  5. @Override
  6. public HttpResponse exec(String host, HttpRequest request, HttpContext context) {
  7. System.out.println("host : "+host);
  8. return new HttpResponse();
  9. }
  10. }
  1. package execChain;
  2.  
  3. public class MyHttpClientBuilder {
  4. public static MyHttpClientBuilder create() {
  5. return new MyHttpClientBuilder();
  6. }
  7. public TopHttpClient build(){
  8. MainExecClient client = new MainExecClient();
  9. // append chain ....
  10. return new RetryExecClient(client);
  11. }
  12. }
  1. TopHttpClient client = MyHttpClientBuilder.create().build();
  2. client.execute("http://www.baidu.com",new HttpRequest(),new HttpContext());

  有关连接池的操作,是在PoolingHttpClientConnectionManager里边,它把对链接的申请称呼为lease,也就是租借的意思。

  1. @Override
  2. public ConnectionRequest requestConnection(
  3. final HttpRoute route,
  4. final Object state) {
  5. Args.notNull(route, "HTTP route");
  6. if (this.log.isDebugEnabled()) {
  7. this.log.debug("Connection request: " + format(route, state) + formatStats(route));
  8. }
  9. final Future<CPoolEntry> future = this.pool.lease(route, state, null);
  10. return new ConnectionRequest() {
  11.  
  12. @Override
  13. public boolean cancel() {
  14. return future.cancel(true);
  15. }
  16.  
  17. @Override
  18. public HttpClientConnection get(
  19. final long timeout,
  20. final TimeUnit tunit) throws InterruptedException, ExecutionException, ConnectionPoolTimeoutException {
  21. final HttpClientConnection conn = leaseConnection(future, timeout, tunit);
  22. if (conn.isOpen()) {
  23. final HttpHost host;
  24. if (route.getProxyHost() != null) {
  25. host = route.getProxyHost();
  26. } else {
  27. host = route.getTargetHost();
  28. }
  29. final SocketConfig socketConfig = resolveSocketConfig(host);
  30. conn.setSocketTimeout(socketConfig.getSoTimeout());
  31. }
  32. return conn;
  33. }
  34.  
  35. };
  36.  
  37. }

   我们观察到它的池子pool是CPool这个类,它的继承关系如下。

   发现大部分的逻辑实在AbstractConnPool里边,租借的时候走了这个方法org.apache.http.pool.AbstractConnPool#getPoolEntryBlocking。在这个方法中,我们发现它往一个RouteSpecificPool中通过route获取了一个池子。

  然后进去看到它池子的定义,结合代码可以看出,它把链接的状态分为:已租借、可用链接、等待三种状态。梳理了一下lease方法,它是将avalible的链接拿出来放到lease中,如果可用队列没有链接,那它将创建一个并放入租借队列,这里如果它这个route的连接数超过了你设置的MaxPerRoute配置,那么它将会方法pending队列,并且await当前线程,直到有hold链接的线程调用了releaseConnection等方法才会被notify。

  

   这个getPoolEntityBlocking方法,就是申请连接池链接的核心代码了。

  1. private E getPoolEntryBlocking(
  2. final T route, final Object state,
  3. final long timeout, final TimeUnit tunit,
  4. final Future<E> future) throws IOException, InterruptedException, TimeoutException {
  5.  
  6. Date deadline = null;
  7. if (timeout > 0) {
  8. deadline = new Date (System.currentTimeMillis() + tunit.toMillis(timeout));
  9. }
  10. this.lock.lock();
  11. try {
  12. final RouteSpecificPool<T, C, E> pool = getPool(route);
  13. E entry;
  14. for (;;) {
  15. Asserts.check(!this.isShutDown, "Connection pool shut down");
  16. for (;;) {
  17. entry = pool.getFree(state);
  18. if (entry == null) {
  19. break;
  20. }
  21. if (entry.isExpired(System.currentTimeMillis())) {
  22. entry.close();
  23. }
  24. if (entry.isClosed()) {
  25. this.available.remove(entry);
  26. pool.free(entry, false);
  27. } else {
  28. break;
  29. }
  30. }
  31. if (entry != null) {
  32. this.available.remove(entry);
  33. this.leased.add(entry);
  34. onReuse(entry);
  35. return entry;
  36. }
  37.  
  38. // New connection is needed
  39. final int maxPerRoute = getMax(route);
  40. // Shrink the pool prior to allocating a new connection
  41. final int excess = Math.max(0, pool.getAllocatedCount() + 1 - maxPerRoute);
  42. if (excess > 0) {
  43. for (int i = 0; i < excess; i++) {
  44. final E lastUsed = pool.getLastUsed();
  45. if (lastUsed == null) {
  46. break;
  47. }
  48. lastUsed.close();
  49. this.available.remove(lastUsed);
  50. pool.remove(lastUsed);
  51. }
  52. }
  53.  
  54. if (pool.getAllocatedCount() < maxPerRoute) {
  55. final int totalUsed = this.leased.size();
  56. final int freeCapacity = Math.max(this.maxTotal - totalUsed, 0);
  57. if (freeCapacity > 0) {
  58. final int totalAvailable = this.available.size();
  59. if (totalAvailable > freeCapacity - 1) {
  60. if (!this.available.isEmpty()) {
  61. final E lastUsed = this.available.removeLast();
  62. lastUsed.close();
  63. final RouteSpecificPool<T, C, E> otherpool = getPool(lastUsed.getRoute());
  64. otherpool.remove(lastUsed);
  65. }
  66. }
  67. final C conn = this.connFactory.create(route);
  68. entry = pool.add(conn);
  69. this.leased.add(entry);
  70. return entry;
  71. }
  72. }
  73.  
  74. boolean success = false;
  75. try {
  76. if (future.isCancelled()) {
  77. throw new InterruptedException("Operation interrupted");
  78. }
  79. pool.queue(future);
  80. this.pending.add(future);
  81. if (deadline != null) {
  82. success = this.condition.awaitUntil(deadline);
  83. } else {
  84. this.condition.await();
  85. success = true;
  86. }
  87. if (future.isCancelled()) {
  88. throw new InterruptedException("Operation interrupted");
  89. }
  90. } finally {
  91. // In case of 'success', we were woken up by the
  92. // connection pool and should now have a connection
  93. // waiting for us, or else we're shutting down.
  94. // Just continue in the loop, both cases are checked.
  95. pool.unqueue(future);
  96. this.pending.remove(future);
  97. }
  98. // check for spurious wakeup vs. timeout
  99. if (!success && (deadline != null && deadline.getTime() <= System.currentTimeMillis())) {
  100. break;
  101. }
  102. }
  103. throw new TimeoutException("Timeout waiting for connection");
  104. } finally {
  105. this.lock.unlock();
  106. }
  107. }

  

  回过头来,在MainClient里边的这一行,是建立Tcp链接的代码。

  

   他们将链接的各种类型作为一种常量也可以说是一种枚举,然后通过while循环全部处理,这里边的状态跳转就不理了。

  1. /** Indicates that the route can not be established at all. */
  2. public final static int UNREACHABLE = -1;
  3.  
  4. /** Indicates that the route is complete. */
  5. public final static int COMPLETE = 0;
  6.  
  7. /** Step: open connection to target. */
  8. public final static int CONNECT_TARGET = 1;
  9.  
  10. /** Step: open connection to proxy. */
  11. public final static int CONNECT_PROXY = 2;
  12.  
  13. /** Step: tunnel through proxy to target. */
  14. public final static int TUNNEL_TARGET = 3;
  15.  
  16. /** Step: tunnel through proxy to other proxy. */
  17. public final static int TUNNEL_PROXY = 4;
  18.  
  19. /** Step: layer protocol (over tunnel). */
  20. public final static int LAYER_PROTOCOL = 5;
  1. /**
  2. * Establishes the target route.
  3. */
  4. void establishRoute(
  5. final AuthState proxyAuthState,
  6. final HttpClientConnection managedConn,
  7. final HttpRoute route,
  8. final HttpRequest request,
  9. final HttpClientContext context) throws HttpException, IOException {
  10. final RequestConfig config = context.getRequestConfig();
  11. final int timeout = config.getConnectTimeout();
  12. final RouteTracker tracker = new RouteTracker(route);
  13. int step;
  14. do {
  15. final HttpRoute fact = tracker.toRoute();
  16. step = this.routeDirector.nextStep(route, fact);
  17.  
  18. switch (step) {
  19.  
  20. case HttpRouteDirector.CONNECT_TARGET:
  21. this.connManager.connect(
  22. managedConn,
  23. route,
  24. timeout > 0 ? timeout : 0,
  25. context);
  26. tracker.connectTarget(route.isSecure());
  27. break;
  28. case HttpRouteDirector.CONNECT_PROXY:
  29. this.connManager.connect(
  30. managedConn,
  31. route,
  32. timeout > 0 ? timeout : 0,
  33. context);
  34. final HttpHost proxy = route.getProxyHost();
  35. tracker.connectProxy(proxy, false);
  36. break;
  37. case HttpRouteDirector.TUNNEL_TARGET: {
  38. final boolean secure = createTunnelToTarget(
  39. proxyAuthState, managedConn, route, request, context);
  40. this.log.debug("Tunnel to target created.");
  41. tracker.tunnelTarget(secure);
  42. } break;
  43.  
  44. case HttpRouteDirector.TUNNEL_PROXY: {
  45. // The most simple example for this case is a proxy chain
  46. // of two proxies, where P1 must be tunnelled to P2.
  47. // route: Source -> P1 -> P2 -> Target (3 hops)
  48. // fact: Source -> P1 -> Target (2 hops)
  49. final int hop = fact.getHopCount()-1; // the hop to establish
  50. final boolean secure = createTunnelToProxy(route, hop, context);
  51. this.log.debug("Tunnel to proxy created.");
  52. tracker.tunnelProxy(route.getHopTarget(hop), secure);
  53. } break;
  54.  
  55. case HttpRouteDirector.LAYER_PROTOCOL:
  56. this.connManager.upgrade(managedConn, route, context);
  57. tracker.layerProtocol(route.isSecure());
  58. break;
  59.  
  60. case HttpRouteDirector.UNREACHABLE:
  61. throw new HttpException("Unable to establish route: " +
  62. "planned = " + route + "; current = " + fact);
  63. case HttpRouteDirector.COMPLETE:
  64. this.connManager.routeComplete(managedConn, route, context);
  65. break;
  66. default:
  67. throw new IllegalStateException("Unknown step indicator "
  68. + step + " from RouteDirector.");
  69. }
  70.  
  71. } while (step > HttpRouteDirector.COMPLETE);
  72. }

   最近遇到了一个问题,设置了最大超时时间为5s,但是由于触发了某种条件,监控显示Http调用有长达12s也能成功的,因为阅读了源码,所以马上想到了是连接池的PerRoute设置得太小了,导致同一域名的请求超过了这个设置,引起了线程的await事件,所以Http层面的超时时间只能保证通信的超时,如果触发了线程排队,那么设置的超时时间并没有生效,而且在源码中也没找到能快速失败的路径,正在寻找这一问题的解决方案。

HttpClient 源码阅读的更多相关文章

  1. 【原】AFNetworking源码阅读(四)

    [原]AFNetworking源码阅读(四) 本文转载请注明出处 —— polobymulberry-博客园 1. 前言 上一篇还遗留了很多问题,包括AFURLSessionManagerTaskDe ...

  2. 【原】FMDB源码阅读(三)

    [原]FMDB源码阅读(三) 本文转载请注明出处 —— polobymulberry-博客园 1. 前言 FMDB比较优秀的地方就在于对多线程的处理.所以这一篇主要是研究FMDB的多线程处理的实现.而 ...

  3. 【原】FMDB源码阅读(二)

    [原]FMDB源码阅读(二) 本文转载请注明出处 -- polobymulberry-博客园 1. 前言 上一篇只是简单地过了一下FMDB一个简单例子的基本流程,并没有涉及到FMDB的所有方方面面,比 ...

  4. 【原】FMDB源码阅读(一)

    [原]FMDB源码阅读(一) 本文转载请注明出处 —— polobymulberry-博客园 1. 前言 说实话,之前的SDWebImage和AFNetworking这两个组件我还是使用过的,但是对于 ...

  5. 【原】AFNetworking源码阅读(六)

    [原]AFNetworking源码阅读(六) 本文转载请注明出处 —— polobymulberry-博客园 1. 前言 这一篇的想讲的,一个就是分析一下AFSecurityPolicy文件,看看AF ...

  6. 【原】AFNetworking源码阅读(五)

    [原]AFNetworking源码阅读(五) 本文转载请注明出处 —— polobymulberry-博客园 1. 前言 上一篇中提及到了Multipart Request的构建方法- [AFHTTP ...

  7. 【原】AFNetworking源码阅读(三)

    [原]AFNetworking源码阅读(三) 本文转载请注明出处 —— polobymulberry-博客园 1. 前言 上一篇的话,主要是讲了如何通过构建一个request来生成一个data tas ...

  8. 【原】AFNetworking源码阅读(二)

    [原]AFNetworking源码阅读(二) 本文转载请注明出处 —— polobymulberry-博客园 1. 前言 上一篇中我们在iOS Example代码中提到了AFHTTPSessionMa ...

  9. 【原】AFNetworking源码阅读(一)

    [原]AFNetworking源码阅读(一) 本文转载请注明出处 —— polobymulberry-博客园 1. 前言 AFNetworking版本:3.0.4 由于我平常并没有经常使用AFNetw ...

随机推荐

  1. java(SSM)上传文件到七牛云(对象存储)

    项目中会用到大量的图片和小视频,为了分担服务器压力,将文件都放在七牛云.这里的思路很简单, 就是移动端.pc端把文件上传到服务器,服务器做一个临时缓存,保存必要的信息到数据库后, 将文件上传到七牛云, ...

  2. spark简单快速学习及打开UI界面---1

    1.远程集群测试 import org.apache.spark.{SparkContext, SparkConf} import scala.math.random /** * 利用spark进行圆 ...

  3. web页面找不到资源文件,报404,但是资源文件存在且路径没错

    如题 , 今天遇到这个问题,maven项目导入本地myeclipse,正常跑起来之后,在web端存在部分页面资源加载不进来. 但是项目资源确实存在,一开始以为是myeclipse开发环境搭建错误导致, ...

  4. Python multiprocess模块(中)

    主要内容: 一. 锁 二. 信号量 三. 事件 通过event来完成红绿灯模型 四. 队列(重点) 队列实现进程间的通信 五. 生产者消费者模型 1. 初始版本(程序会阻塞住) 2. 升级版本一(通过 ...

  5. 浏览器F12功能键对测试工程师的重要性

    F键,功能键,Function键.F12常用于网站界面测试.调试,分析网页所出现的问题,查看html元素.查看响应事件等方面. 打开一个网页,点击F12,弹出一个窗口,其窗口的功能如下: 1 Elem ...

  6. kubeadm安装集群系列-2.Master高可用

    Master高可用安装 VIP负载均衡可以使用haproxy+keepalive实现,云上用户可以使用对应的ULB实现 准备kubeadm-init.yaml文件 apiVersion: kubead ...

  7. python之函数形参、实参、可变长参数整体使用和分类

    形参与实参 '''def fn(形参们): pass fn(实参们)'''# 形参:定义函数,在括号内声明的变量名,用来结束外界传来的值# 实参:调用函数,在括号内传入的实际值,值可以为常量.变量.表 ...

  8. vi操作笔记一

    vi命令  gg 到首行 shift + 4 跳到该行最后一个字符 shift + 6 跳到该行首个字符 shift + g 到尾行 vi 可视 G 全选 = 程序对齐   gg 到首行 vi 可视  ...

  9. firewall-cmd 的简单使用 进行端口转发的使用

    今天本来想用 ssh 做端口转发 但是命令死活想不起来了.. 没办法改用firewall-cmd 来进行处理 方法: 1. 首先不能关防火墙 systemctl enable firewalld sy ...

  10. selenium开发-C#/java/Python

    背景:之前由于自己有编写CefSharp.WinForms 窗体版以及 接口化 WCF+CefSharp.WinForms的网站版本,但是由于某些原因,延伸出Selenium学习与研究 总结:sele ...