爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.packages.urllib3.connection.HTTPConnection object at XXXX>: Failed to establish a new connection: [Errno 99] Cannot assign requested address'

是因为在每次数据传输前客户端要和服务器建立TCP连接,为节省传输消耗,默认为keep-alive,即连接一次,传输多次,然而在多次访问后不能结束并回到连接池中,导致不能产生新的连接

headers中的Connection默认为keep-alive,

将header中的Connection一项置为close

headers = {

    'Connection': 'close',

}

r = requests.get(url, headers=headers)

此时问题解决

HTTPConnectionPool(host:XX)Max retries exceeded with url 解决方法的更多相关文章

  1. HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x0000015A25025EB8>...))

    HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTim ...

  2. scrapy::Max retries exceeded with url

    运行scrapy时出错这个错误:Max retries exceeded with url解决方法: img1=requests.get(url=aa,headers=header1,timeout= ...

  3. HTTPConnectionPool(host:XX)Max retries exceeded with url

    爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.package ...

  4. 关于python3.6上传文件时报错:HTTPSConnectionPool(host='***.org', port=443): Max retries exceeded with url: /post (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAIL解决办法

    第一个报错: 最近在练习post请求中上传文件时遇到了一个奇葩事情,两台电脑上写了一模一样的代码,一个运行正常,另一个一片红. 最后了解了一下原因以及解决办法.先记录下关键代码: files = {& ...

  5. windows环境pip安装时一直报错Could not fetch URL https://pypi.org/simple/xrld/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url:

    最近项目不忙了~~有开始专研的python大业,上来想用pip安装一个第三方的库,就一直报错: Could not fetch URL https://pypi.org/simple/xrld/: T ...

  6. HTTPSConnectionPool(host='xxxxx', port=443): Max retries exceeded with url:xxxxxxxx (Caused by NewConnectionError('<urllib3.connect,Max retries exceeded with ,(Caused by NewConnectionError

    HTTPSConnectionPool(host='f6ws-sha8re-o88k.s3.ama66zaws.com', port=443): Max retries exceeded with u ...

  7. Max retries exceeded with url

    78 Traceback (most recent call last):   File "thread072413.py", line 163, in <module> ...

  8. 解决Max retries exceeded with url的问题

    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max ret ...

  9. requests.exceptions.SSLError……Max retries exceeded with url错误求助!!!

    import requests head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl ...

随机推荐

  1. (11)线程池(最新的concurrent.futures包去开启)

    '''concurrent.futures是最新的开启线程池的包'''import timefrom concurrent.futures import ThreadPoolExecutor #开启线 ...

  2. Spring源码学习(总)

    前文: ------------------------------------------------------------------------------------------------ ...

  3. 【NOI2005】 聪聪可可

    树分治劲啊 原题: 聪聪和可可是兄弟俩,他们俩经常为了一些琐事打起来,例如家中只剩下最后一根冰棍而两人都想吃.两个人都想玩儿电脑(可是他们家只有一台电脑)……遇到这种问题,一般情况下石头剪刀布就好了, ...

  4. linux系统调用的三种方法

    通过glibc提供的库函数 [23:02:14] gcc chmodtest.c [23:02:17] ls -l kali //记得先创建这个文件 -rwxrwxrwx. 1 root root 0 ...

  5. 用flask和长轮询实现对帅哥投票和实时查看票数

    flask中的代码 from flask import Flask,request,render_template,redirect,session,jsonify import uuid impor ...

  6. Singer 学习四 可视化数据操作工具

    knots 是一款基于electron 开发的可视化UI界面,我们可以此工具进行方便的数据处理,注意工具使用了 docker 运行,需要安装docker 下载地址   https://github.c ...

  7. YAML Class ID Reference

    Classes Ordered by ID Number ID Class 1 GameObject 2 Component 3 LevelGameManager 4 Transform 5 Time ...

  8. Hi3518EV200音频相关

    1.sample程序可以录音,音频格式为G711A.G711U.G726.ADPCM: 2.ADPCM找不到音频播放器 3.G711格式海思添加了4字节头0x00 0x01 0x0a 0x00,普通播 ...

  9. webpack报错:Module build failed (from ./node_modules/babel-loader/lib/index.js): Error: Cannot find module 'babel-preset-env' from '...' - Did you mean "@babel/env"?

    webpack报错:Module build failed (from ./node_modules/babel-loader/lib/index.js): Error: Cannot find mo ...

  10. egg 知识点

    egg 的约定 约定 使用方法 路由对应controller中的方法 举例:router.get('/', controller.home.index);,此时对应app/controller/hom ...