爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.packages.urllib3.connection.HTTPConnection object at XXXX>: Failed to establish a new connection: [Errno 99] Cannot assign requested address'

是因为在每次数据传输前客户端要和服务器建立TCP连接,为节省传输消耗,默认为keep-alive,即连接一次,传输多次,然而在多次访问后不能结束并回到连接池中,导致不能产生新的连接

headers中的Connection默认为keep-alive,

将header中的Connection一项置为close

headers = {

    'Connection': 'close',

}

r = requests.get(url, headers=headers)

此时问题解决

HTTPConnectionPool(host:XX)Max retries exceeded with url 解决方法的更多相关文章

  1. HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x0000015A25025EB8>...))

    HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTim ...

  2. scrapy::Max retries exceeded with url

    运行scrapy时出错这个错误:Max retries exceeded with url解决方法: img1=requests.get(url=aa,headers=header1,timeout= ...

  3. HTTPConnectionPool(host:XX)Max retries exceeded with url

    爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.package ...

  4. 关于python3.6上传文件时报错:HTTPSConnectionPool(host='***.org', port=443): Max retries exceeded with url: /post (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAIL解决办法

    第一个报错: 最近在练习post请求中上传文件时遇到了一个奇葩事情,两台电脑上写了一模一样的代码,一个运行正常,另一个一片红. 最后了解了一下原因以及解决办法.先记录下关键代码: files = {& ...

  5. windows环境pip安装时一直报错Could not fetch URL https://pypi.org/simple/xrld/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url:

    最近项目不忙了~~有开始专研的python大业,上来想用pip安装一个第三方的库,就一直报错: Could not fetch URL https://pypi.org/simple/xrld/: T ...

  6. HTTPSConnectionPool(host='xxxxx', port=443): Max retries exceeded with url:xxxxxxxx (Caused by NewConnectionError('<urllib3.connect,Max retries exceeded with ,(Caused by NewConnectionError

    HTTPSConnectionPool(host='f6ws-sha8re-o88k.s3.ama66zaws.com', port=443): Max retries exceeded with u ...

  7. Max retries exceeded with url

    78 Traceback (most recent call last):   File "thread072413.py", line 163, in <module> ...

  8. 解决Max retries exceeded with url的问题

    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max ret ...

  9. requests.exceptions.SSLError……Max retries exceeded with url错误求助!!!

    import requests head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl ...

随机推荐

  1. 费马大定理以及求解a^2+b^2=c^2的奇偶数列法则

    <一>费马大定理:a^n+b^n=c^n 当n大于2时无正整数解. <二>求解a^2+b^=c^2可以使用a值奇偶法则:1.当a=2*n时,b=n^2-1,c=n^2+1   ...

  2. 《DSP using MATLAB》Problem 5.31

    第3小题: 代码: %% ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ %% Out ...

  3. <------------------字节流--------------------->

    字节流: 输入和输出:1.参照物都是java程序来惨遭 2.Input输入,持久化上的数据---->内存 3.Output输出,内存--->硬盘 字节输出流: OutputStream: ...

  4. laya的UI编辑器

    //加载一个图集 Laya.loader.load("res/atlas/comp.json",Handler.create(this,this.onLoaderComp), Ha ...

  5. keil5配置ST Link v2 for STM32F10x

    请务必先安装MDKCM525.EXE,否则会找不到闪存编程算法

  6. day9大纲

    01 作业内容回顾 函数的初识: 封装一个功能. def 函数名(): 函数体 函数的返回值:return 1,结束函数. 2,返回给执行者(函数名())值. return ----> None ...

  7. jmap获取内存排名靠前的对象

    docker部署,jvvm通过jmx访问失败,只能永远是的jdk自带工具查看情况: jmap -histo pid | sort -n -r -k 2 | head -10

  8. php调用c#的dll(转)

    这几天,一直在做DES ecb模式的加解密,刚用.net实现了加解密,完了由于需要又要转型成PHP代码,费了九牛二虎之力单独用PHP没能实现,结构看到一篇php直接调用c#里生成的.dll文件的方法, ...

  9. Linux crontab配置

    crontab配置 1.命令功能 通过crontab 命令,我们可以在固定的间隔时间执行指定的系统指令或 shell script脚本.时间间隔的单位可以是分钟.小时.日.月.周及以上的任意组合.这个 ...

  10. 多线程中的Lock小结

    出处:http://www.cnblogs.com/DarrenChan/p/6528578.html#undefined 1.lock和synchronized的区别 1)Lock不是Java语言内 ...