爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.packages.urllib3.connection.HTTPConnection object at XXXX>: Failed to establish a new connection: [Errno 99] Cannot assign requested address'
是因为在每次数据传输前客户端要和服务器建立TCP连接,为节省传输消耗,默认为keep-alive,即连接一次,传输多次,然而在多次访问后不能结束并回到连接池中,导致不能产生新的连接
headers中的Connection默认为keep-alive,
将header中的Connection一项置为close

headers = {
    'Connection': 'close',
}
r = requests.get(url, headers=headers)

此时问题解决

HTTPConnectionPool(host:XX)Max retries exceeded with url的更多相关文章

  1. HTTPConnectionPool(host:XX)Max retries exceeded with url 解决方法

    爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.package ...

  2. HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x0000015A25025EB8>...))

    HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTim ...

  3. 关于python3.6上传文件时报错:HTTPSConnectionPool(host='***.org', port=443): Max retries exceeded with url: /post (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAIL解决办法

    第一个报错: 最近在练习post请求中上传文件时遇到了一个奇葩事情,两台电脑上写了一模一样的代码,一个运行正常,另一个一片红. 最后了解了一下原因以及解决办法.先记录下关键代码: files = {& ...

  4. windows环境pip安装时一直报错Could not fetch URL https://pypi.org/simple/xrld/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url:

    最近项目不忙了~~有开始专研的python大业,上来想用pip安装一个第三方的库,就一直报错: Could not fetch URL https://pypi.org/simple/xrld/: T ...

  5. HTTPSConnectionPool(host='xxxxx', port=443): Max retries exceeded with url:xxxxxxxx (Caused by NewConnectionError('<urllib3.connect,Max retries exceeded with ,(Caused by NewConnectionError

    HTTPSConnectionPool(host='f6ws-sha8re-o88k.s3.ama66zaws.com', port=443): Max retries exceeded with u ...

  6. centos7中python3.6报错ModuleNotFoundError: No module named '_ssl' 或者 Max retries exceeded with url: / (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",))

    如果在运行爬虫时报此错:requests.exceptions.SSLError: HTTPSConnectionPool(host='www.baidu.com', port=443): Max r ...

  7. Max retries exceeded with url

    78 Traceback (most recent call last):   File "thread072413.py", line 163, in <module> ...

  8. 解决Max retries exceeded with url的问题

    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max ret ...

  9. requests.exceptions.SSLError……Max retries exceeded with url错误求助!!!

    import requests head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl ...

随机推荐

  1. MicroMsg.SDK.WXApiImplV10: register app failed for wechat app signature check failed

    支付时: IWXAPI wxapi = WXAPIFactory.createWXAPI(this,WXAPPID,true); 替换为 IWXAPI wxapi = WXAPIFactory.cre ...

  2. Noi.ac #309. Mas的童年(贪心)

    /* 用所谓的加法拆分操作得到 x + y = (x ^ y) + 2 * (x & y) 那么我们这两段异或相当于前缀和 + 2 * 分段使左右两块&最大 记当前前缀异或和为S, 那 ...

  3. c#,Model 实体转json,字符串转json

    public class JsonF { #region 字符串转json /// <summary> /// 字符串转json /// </summary> /// < ...

  4. java 日期排序。。。。

    Collections.sort(list, new Comparator<Map<Object, Object>>() { public int compare(Map< ...

  5. 【HQL】常用函数

    CONCAT_WS(separator, str1, str2,...) 多列转1列,以分割符分割 使用场景: 1.多列在一列显示: 2.多列转多行作为辅助,结合split和explode使用 SEL ...

  6. UVA-10020-贪心

    题意:给你一些数轴上的线段,要求寻找出某些线段能够完全覆盖[0,M],并且取的线段数目最小. 解题思路: 贪心思路, 1.每个线段都有一个L和R,代表它的起点和终点,对于所有R <= 0 ,   ...

  7. (1)Linux操作系统的安装

    1.安装Vmware Workstation 如何判断Vmware是否安装成功? 我的电脑控制面板 --> 网络和Internet --> 查看网络状态和任务 --> 更改适配器设置 ...

  8. 2. 修改jsp需要重启tomcat的问题

  9. tomcat中项目后有括号

    引入他人项目时,由于报错,copy本地workspace下其他项目的 .settings和.project到该项目路径下 结果Eclipse 的 Server 中出现了  aaa(bbb)的情况 并且 ...

  10. LinkedHashMap结构get和put源码流程简析及LRU应用

    原理这篇讲得比较透彻Java集合之LinkedHashMap. 本文属于源码阅读笔记,因put,get调用逻辑及链表维护逻辑复杂(至少网上其它文章的逻辑描述及配图,我都没看明白LinkedHashMa ...