爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.packages.urllib3.connection.HTTPConnection object at XXXX>: Failed to establish a new connection: [Errno 99] Cannot assign requested address'
是因为在每次数据传输前客户端要和服务器建立TCP连接,为节省传输消耗,默认为keep-alive,即连接一次,传输多次,然而在多次访问后不能结束并回到连接池中,导致不能产生新的连接
headers中的Connection默认为keep-alive,
将header中的Connection一项置为close

headers = {
    'Connection': 'close',
}
r = requests.get(url, headers=headers)

此时问题解决

HTTPConnectionPool(host:XX)Max retries exceeded with url的更多相关文章

  1. HTTPConnectionPool(host:XX)Max retries exceeded with url 解决方法

    爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.package ...

  2. HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x0000015A25025EB8>...))

    HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTim ...

  3. 关于python3.6上传文件时报错:HTTPSConnectionPool(host='***.org', port=443): Max retries exceeded with url: /post (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAIL解决办法

    第一个报错: 最近在练习post请求中上传文件时遇到了一个奇葩事情,两台电脑上写了一模一样的代码,一个运行正常,另一个一片红. 最后了解了一下原因以及解决办法.先记录下关键代码: files = {& ...

  4. windows环境pip安装时一直报错Could not fetch URL https://pypi.org/simple/xrld/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url:

    最近项目不忙了~~有开始专研的python大业,上来想用pip安装一个第三方的库,就一直报错: Could not fetch URL https://pypi.org/simple/xrld/: T ...

  5. HTTPSConnectionPool(host='xxxxx', port=443): Max retries exceeded with url:xxxxxxxx (Caused by NewConnectionError('<urllib3.connect,Max retries exceeded with ,(Caused by NewConnectionError

    HTTPSConnectionPool(host='f6ws-sha8re-o88k.s3.ama66zaws.com', port=443): Max retries exceeded with u ...

  6. centos7中python3.6报错ModuleNotFoundError: No module named '_ssl' 或者 Max retries exceeded with url: / (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",))

    如果在运行爬虫时报此错:requests.exceptions.SSLError: HTTPSConnectionPool(host='www.baidu.com', port=443): Max r ...

  7. Max retries exceeded with url

    78 Traceback (most recent call last):   File "thread072413.py", line 163, in <module> ...

  8. 解决Max retries exceeded with url的问题

    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max ret ...

  9. requests.exceptions.SSLError……Max retries exceeded with url错误求助!!!

    import requests head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl ...

随机推荐

  1. SQL with(unlock)与with(readpast)

    所有Select加 With (NoLock)解决阻塞死锁,在查询语句中使用 NOLOCK 和 READPAST 处理一个数据库死锁的异常时候,其中一个建议就是使用 NOLOCK 或者 READPAS ...

  2. Echarts动态加载柱状图和折线图混合展示的实例

    一.引入echarts文件: <script type="text/javascript" src="echarts.js"></script ...

  3. python hashillb 模块

    hashlib主要提供字符加密功能,将md5和sha模块整合到了一起,支持md5,sha1, sha224, sha256, sha384, sha512等算法 import hashlib # ## ...

  4. @Autowired与@Resource 详细诠释和区别(附带例子)

    @Autowired 与@Resource:1.@Autowired与@Resource都可以用来装配bean. 都可以写在字段上,或写在setter方法上. 2.@Autowired默认按类型装配( ...

  5. day5作业(基本数据类型字符串,列表)

    #coding:utf-8'''默写99乘法标 金字塔 必做: 1.昨日选做题 博客中有 http://www.cnblogs.com/linhaifeng/articles/7133357.html ...

  6. GNU tar

    2.tar教程 2.4.常用选项 2.5.两个选项 2.6.创建档案文档 2.7.查看档案文档内容 4.tar操作 4.1.基本操作 4.2.高级操作 4.3.“-c”的选项 链接到压缩命令 2.ta ...

  7. ace admin

    .svg             image/svg+xml.woff            application/x-font-woff.woff2          application/x- ...

  8. js:浏览器插件

    1.chrome background.js //chrome.webRequest.onBeforeRequest.addListener(function(info) { // chrome.ta ...

  9. 查SQLserver性能瓶颈

    https://www.cnblogs.com/wayne-ivan/p/3821750.html

  10. 使用MediaPlayer类和SurfaceView来播放视频

    MediaPlayer可以播放视频,只需需要SurfaceView的配合,SurfaceView主要用于显示MediaPlayer播放的视频流媒体的画面渲染. SurfaceView是配合MediaP ...