0.

requests不设置UA 访问baidu 得到 r.headers['Content-Type'] 是text/html  使用chrome UA: Content-Type:text/html; charset=utf-8

1.参考

代码分析Python requests库中文编码问题

iso-8859是什么?  他又被叫做Latin-1或“西欧语言”

补丁:

import requests
def monkey_patch():
prop = requests.models.Response.content
def content(self):
_content = prop.fget(self)
if self.encoding == 'ISO-8859-1':
encodings = requests.utils.get_encodings_from_content(_content)
if encodings:
self.encoding = encodings[0]
else:
self.encoding = self.apparent_encoding
_content = _content.decode(self.encoding, 'replace').encode('utf8', 'replace')
self._content = _content
return _content
requests.models.Response.content = property(content)
monkey_patch()

2.原因

In [291]: r = requests.get('http://cn.python-requests.org/en/latest/')

In [292]: r.headers.get('content-type')
Out[292]: 'text/html; charset=utf-8' In [293]: r.encoding
Out[293]: 'utf-8' In [294]: rc = requests.get('http://python3-cookbook.readthedocs.io/zh_CN/latest/index.html') In [296]: rc.headers.get('content-type')
Out[296]: 'text/html' In [298]: rc.encoding
Out[298]: 'ISO-8859-1'

response text 异常

In [312]: rc.text
Out[312]: u'\n\n<!DOCTYPE html>\n<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->\n<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->\n<head>\n <meta charset="ut
f-8">\n \n <meta name="viewport" content="width=device-width, initial-scale=1.0">\n \n <title>Python Cookbook 3rd Edition Documentation &mdash; python3-cookbook 2.0.0 \xe6\x96\x87\xe6\xa1\xa3</tit
le>\n \n\n \n \n \n \n\n \n\n \n \n \n\n \n\n \n \n\n \n <link rel="stylesheet" href="https://media.readthedocs.org/css/sphinx_rtd_theme.css" type="text/css" />\n \n\n \n <l
ink rel="index" title="\xe7\xb4\xa2\xe5\xbc\x95"\n href="genindex.html"/>\n <link rel="search" title="\xe6\x90\x9c\xe7\xb4\xa2" href="search.html"/>\n <link rel="copyright"
title="\xe7\x89\x88\xe6\x9d\x83\xe6\x89\x80\xe6\x9c\x89" href="copyright.html"/>\n <link rel="top" title="python3-cookbook 2.0.0 \xe6\x96\x87\xe6\xa1\xa3" href="#"/>\n <link rel="next" title In [313]: rc.content
Out[313]: '\n\n<!DOCTYPE html>\n<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->\n<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->\n<head>\n <meta charset="utf
-8">\n \n <meta name="viewport" content="width=device-width, initial-scale=1.0">\n \n <title>Python Cookbook 3rd Edition Documentation &mdash; python3-cookbook 2.0.0 \xe6\x96\x87\xe6\xa1\xa3</titl
e>\n \n\n \n \n \n \n\n \n\n \n \n \n\n \n\n \n \n\n \n <link rel="stylesheet" href="https://media.readthedocs.org/css/sphinx_rtd_theme.css" type="text/css" />\n \n\n \n <li
nk rel="index" title="\xe7\xb4\xa2\xe5\xbc\x95"\n href="genindex.html"/>\n <link rel="search" title="\xe6\x90\x9c\xe7\xb4\xa2" href="search.html"/>\n <link rel="copyright" t
itle="\xe7\x89\x88\xe6\x9d\x83\xe6\x89\x80\xe6\x9c\x89" href="copyright.html"/>\n <link rel="top" title="python3-cookbook 2.0.0 \xe6\x96\x87\xe6\xa1\xa3" href="#"/>\n <link rel="next" title=

response headers有'content-type'而且没有charset而且有'text',同时满足三个条件导致判定'ISO-8859-1'

参考文章说 python3 没有问题,实测有。

C:\Program Files\Anaconda2\Lib\site-packages\requests\utils.py

20180102 补充:# "Content-Type": "application/json" 对应 r.encoding 为 None

def get_encoding_from_headers(headers):
"""Returns encodings from given HTTP Header Dict. :param headers: dictionary to extract encoding from.
:rtype: str
""" content_type = headers.get('content-type') if not content_type:
return None content_type, params = cgi.parse_header(content_type) if 'charset' in params:
return params['charset'].strip("'\"") if 'text' in content_type:
return 'ISO-8859-1'

C:\Program Files\Anaconda2\Lib\site-packages\requests\adapters.py

class HTTPAdapter(BaseAdapter):
def build_response(self, req, resp):
# Set encoding.
response.encoding = get_encoding_from_headers(response.headers)

3.解决办法

参考文章打补丁或:

20180102 补充: if resp.encoding == 'ISO-8859-1':   修改为 if r.encoding == 'ISO-8859-1' and not 'ISO-8859-1' in headers.get('content-type', ''):  即只处理按照协议最后返回的 'ISO-8859-1'

    if r.encoding == 'ISO-8859-1' and not 'ISO-8859-1' in headers.get('content-type', ''):
encodings = requests.utils.get_encodings_from_content(resp.content) #re.compile(r'<meta.*?charset #源代码没有利用这个方法
if encodings:
resp.encoding = encodings[0]
else:
resp.encoding = resp.apparent_encoding #models.py chardet.detect(self.content)['encoding'] 消耗计算 # resp.text >>> if self.encoding is None: encoding = self.apparent_encoding
print 'ISO-8859-1 changed to %s'%resp.encoding

requests之headers 'Content-Type': 'text/html'误判encoding为'ISO-8859-1'导致中文text解码错误的更多相关文章

  1. Jsoup问题---获取http协议请求失败 org.jsoup.UnsupportedMimeTypeException: Unhandled content type. Must be text/*, application/xml, or application/xhtml+xml.

    Jsoup问题---获取http协议请求失败 1.问题:用Jsoup在获取一些网站的数据时,起初获取很顺利,但是在访问某浪的数据是Jsoup报错,应该是请求头里面的请求类型(ContextType)不 ...

  2. Jsoup获取部分页面数据失败 org.jsoup.UnsupportedMimeTypeException: Unhandled content type. Must be text/*, application/xml, or application/xhtml+xml.

    用Jsoup在获取一些网站的数据时,起初获取很顺利,但是在访问某浪的数据是Jsoup报错,应该是请求头里面的请求类型(ContextType)不符合要求. 请求代码如下: private static ...

  3. Jsoup获取部分页面数据失败 Unhandled content type. Must be text/*, application/xml, or application/xhtml+xml

    用Jsoup在获取一些网站的数据时,起初获取很顺利,但是在访问某浪的数据是Jsoup报错,应该是请求头里面的请求类型(ContextType)不符合要求. 请求代码如下: private static ...

  4. {"timestamp":"2019-11-12T02:39:28.949+0000","status":415,"error":"Unsupported Media Type","message":"Content type 'text/plain;charset=UTF-8' not supported","path":&quo

    在Jmeter运行http请求时报错: {"timestamp":"2019-11-12T02:39:28.949+0000","status&quo ...

  5. 遇到问题之“postman报Unsupported Media Type: Content type 'text/plain;charset=UTF-8' not supported”

    postman报Unsupported Media Type: Content type 'text/plain;charset=UTF-8' not supported postman之所以报Uns ...

  6. the request doesn't contain a multipart/form-data or multipart/form-data stream, content type header

    the request doesn't contain a multipart/form-data or multipart/form-data stream, content type header ...

  7. Jmeter发送post请求报错Content type 'application/x-www-form-urlencoded;charset=UTF-8' not supported

    常识普及: Content-type,在Request Headers里面,告诉服务器,我们发送的请求信息格式,在JMeter中,信息头存储在信息头管理器中,所以在做接口测试的时候,我们维护Conte ...

  8. Content type 'application/x-www-form-urlencoded;charset=UTF-8' not supported

    Content type 'application/x-www-form-urlencoded;charset=UTF-8' not supported告诉你,你的请求头是application/x- ...

  9. SharePoint自动化系列——Add content type to list.

    转载请注明出自天外归云的博客园:http://www.cnblogs.com/LanTianYou/ 将创建好的content type(若是跨web application需要事先publish c ...

随机推荐

  1. LODOP、C-Lodop简短排查语句

    https使用,故障:1.是https网站吗,https扩展版C-Lodop如何使用 参考http://www.c-lodop.com/faq/pp32.html2.双击桌面上的c-lodop快捷方式 ...

  2. MyBatis:二级缓存原理分析

    MyBatis从入门到放弃七:二级缓存原理分析 前言 说起mybatis的一级缓存和二级缓存我特意问了几个身边的朋友他们平时会不会用,结果没有一个人平时业务场景中用. 好吧,那我暂且用来学习源码吧.一 ...

  3. [bilibili]弹幕屏蔽列表

    <filters> <item enabled="true">t=定单身</item> <item enabled="true& ...

  4. CentOS Linux安装python3

    本文的方法是在CentOS上新装了python3,如果本机安装了python2则保留,因为可能有程序依赖目前的python2环境,比如yum!!!!! 一.安装python3.7 1. 安装依赖环境 ...

  5. VS2012/2013/2015/Visual Studio 2017 关闭单击文件进行预览的功能

    Visual Studio在2010版本后推出了点击项目管理器预览文件的功能,但是对于配置不咋地的旧电脑总是觉得有点卡,下面是解决方案. 英文版方法:Tools->Options->Env ...

  6. ajax扩展、jsonp、

    偷偷上传消息,首先想到ajax 而ajax是jquery是我们封装的功能. 方式一:ajax发消息 $.ajax({ url: '/ajax3.html', type: 'GET', //或者是POS ...

  7. Mathematica 求出解后代入变量

    Solve[2 x - 3 == 0, x] x = x //. %[[1]]

  8. [物理学与PDEs]第1章第6节 电磁场的标势与矢势 6.1 预备知识

    1.  若 ${\bf B}$ 为横场 ($\Div{\bf B}=0\ra {\bf k}\cdot {\bf B}=0\ra $ 波的振动方向与传播方向平行), 则 $$\bex \exists\ ...

  9. 选择性搜索(SS)算法

    一.目标检测和目标识别 目标识别(object recognition)是要指明一张图像中包含哪类目标.输入是图像,输出是图像中的目标属于的类别(class probability).目标检测是识别出 ...

  10. JAVA写JSON的三种方法,java对象转json数据

    JAVA写JSON的三种方法,java对象转json数据 转自:http://www.xdx97.com/#/single?bid=5afe2ff9-8cd1-67cf-e7bc-437b74c07a ...