爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.packages.urllib3.connection.HTTPConnection object at XXXX>: Failed to establish a new connection: [Errno 99] Cannot assign requested address'

是因为在每次数据传输前客户端要和服务器建立TCP连接,为节省传输消耗,默认为keep-alive,即连接一次,传输多次,然而在多次访问后不能结束并回到连接池中,导致不能产生新的连接

headers中的Connection默认为keep-alive,

将header中的Connection一项置为close

headers = {

    'Connection': 'close',

}

r = requests.get(url, headers=headers)

此时问题解决

HTTPConnectionPool(host:XX)Max retries exceeded with url 解决方法的更多相关文章

  1. HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x0000015A25025EB8>...))

    HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTim ...

  2. scrapy::Max retries exceeded with url

    运行scrapy时出错这个错误:Max retries exceeded with url解决方法: img1=requests.get(url=aa,headers=header1,timeout= ...

  3. HTTPConnectionPool(host:XX)Max retries exceeded with url

    爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.package ...

  4. 关于python3.6上传文件时报错:HTTPSConnectionPool(host='***.org', port=443): Max retries exceeded with url: /post (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAIL解决办法

    第一个报错: 最近在练习post请求中上传文件时遇到了一个奇葩事情,两台电脑上写了一模一样的代码,一个运行正常,另一个一片红. 最后了解了一下原因以及解决办法.先记录下关键代码: files = {& ...

  5. windows环境pip安装时一直报错Could not fetch URL https://pypi.org/simple/xrld/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url:

    最近项目不忙了~~有开始专研的python大业,上来想用pip安装一个第三方的库,就一直报错: Could not fetch URL https://pypi.org/simple/xrld/: T ...

  6. HTTPSConnectionPool(host='xxxxx', port=443): Max retries exceeded with url:xxxxxxxx (Caused by NewConnectionError('<urllib3.connect,Max retries exceeded with ,(Caused by NewConnectionError

    HTTPSConnectionPool(host='f6ws-sha8re-o88k.s3.ama66zaws.com', port=443): Max retries exceeded with u ...

  7. Max retries exceeded with url

    78 Traceback (most recent call last):   File "thread072413.py", line 163, in <module> ...

  8. 解决Max retries exceeded with url的问题

    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max ret ...

  9. requests.exceptions.SSLError……Max retries exceeded with url错误求助!!!

    import requests head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl ...

随机推荐

  1. Linux下安装搜狗拼音输入法

    1.安装 下面命令即可完成安装: sudo apt-add-repository ppa:fcitx-team/nightly sudo apt-get update sudo apt-get ins ...

  2. LeetCode - Top K Frequent Words

    Given a non-empty list of words, return the k most frequent elements. Your answer should be sorted b ...

  3. vue全家桶+Koa2开发笔记(3)--mongodb

    1. 安装 momgodb brew install mongodb安装成功后执行 which mongod启动:mongod 2. 下载可视化操作数据库的软件 https://robomongo.o ...

  4. webpack执行命令参数

    在webpack执行命令之后可以添加一些参数,这些参数都有自己的作用,下面是参数列表: $ webpack --config XXX.js //使用另一份配置文件(比如webpack.config2. ...

  5. centos7安装部署mysql5.7服务器

    因为自带源没有最新版的mysql,所以我们需要自己下载rpm包,先下载下面的rpm包源 https://repo.mysql.com//mysql57-community-release-el7-11 ...

  6. adb command

  7. ML(5)——神经网络2(BP反向传播)

    上一章的神经网络实际上是前馈神经网络(feedforward neural network),也叫多层感知机(multilayer perceptron,MLP).具体来说,每层神经元与下一层神经元全 ...

  8. 艰苦的编译boost python (失败)

    1.下载 boost_1_67_0 2.在目录下执行 bootstrap 3.将python36添加到path环境变量 4.执行 b2 --with-python,将会声场如下dll 2018/04/ ...

  9. JMeterPlugins插件监听器学习-监听器

    JMeterPlugins插件监听器学习-监听器 1.jp@gc - Actiive Threads Over Time:不同时间活动用户数量展示(图表)2.jp@gc - AutoStop List ...

  10. 基于结构化平均感知机的分词器Java实现

    基于结构化平均感知机的分词器Java实现 作者:hankcs 最近高产似母猪,写了个基于AP的中文分词器,在Bakeoff-05的MSR语料上F值有96.11%.最重要的是,只训练了5个迭代:包含语料 ...