Python 3.X 要使用urllib.request 来抓取网络资源。转
Python 3.X 要使用urllib.request 来抓取网络资源。
最简单的方式:
#coding=utf-8
import urllib.request
response = urllib.request.urlopen('http://python.org/')
buff = response.read()
#显示
html = buff.decode("utf8")
response.close()
print(html)
使用Request的方式:
#coding=utf-8
import urllib.request
req = urllib.request.Request('http://www.voidspace.org.uk')
response = urllib.request.urlopen(req)
buff = response.read()
#显示
the_page = buff.decode("utf8")
response.close()
print(the_page)
这种方式同样可以用来处理其他URL,例如FTP:
#coding=utf-8
import urllib.request
req = urllib.request.Request('ftp://ftp.pku.edu.cn/')
response = urllib.request.urlopen(req)
buff = response.read()
#显示
the_page = buff.decode("utf8")
response.close()
print(the_page)
使用POST请求:
import urllib.parseimport
urllib.requesturl = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
'location' : 'Northampton',
'language' : 'Python' }
data = urllib.parse.urlencode(values)
req = urllib.request.Request(url, data)
response = urllib.request.urlopen(req)
the_page = response.read()
使用GET请求:
import urllib.request
import urllib.parse
data = {}
data['name'] = 'Somebody Here'
data['location'] = 'Northampton'
data['language'] = 'Python'
url_values = urllib.parse.urlencode(data)
print(url_values)
name=Somebody+Here&language=Python&location=Northampton
url = 'http://www.example.com/example.cgi'
full_url = url + '?' + url_values
data = urllib.request.open(full_url)
添加header:
import urllib.parse
import urllib.request url = 'http://www.someserver.com/cgi-bin/register.cgi'
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
values = {'name' : 'Michael Foord',
'location' : 'Northampton',
'language' : 'Python' }
headers = { 'User-Agent' : user_agent } data = urllib.parse.urlencode(values)
req = urllib.request.Request(url, data, headers)
response = urllib.request.urlopen(req)
the_page = response.read()
错误处理:
req = urllib.request.Request('http://www.pretend_server.org')
try: urllib.request.urlopen(req)
except urllib.error.URLError as e:
print(e.reason)
返回的错误代码:
# Table mapping response codes to messages; entries have the
# form {code: (shortmessage, longmessage)}.
responses = {
100: ('Continue', 'Request received, please continue'),
101: ('Switching Protocols',
'Switching to new protocol; obey Upgrade header'), 200: ('OK', 'Request fulfilled, document follows'),
201: ('Created', 'Document created, URL follows'),
202: ('Accepted',
'Request accepted, processing continues off-line'),
203: ('Non-Authoritative Information', 'Request fulfilled from cache'),
204: ('No Content', 'Request fulfilled, nothing follows'),
205: ('Reset Content', 'Clear input form for further input.'),
206: ('Partial Content', 'Partial content follows.'), 300: ('Multiple Choices',
'Object has several resources -- see URI list'),
301: ('Moved Permanently', 'Object moved permanently -- see URI list'),
302: ('Found', 'Object moved temporarily -- see URI list'),
303: ('See Other', 'Object moved -- see Method and URL list'),
304: ('Not Modified',
'Document has not changed since given time'),
305: ('Use Proxy',
'You must use proxy specified in Location to access this '
'resource.'),
307: ('Temporary Redirect',
'Object moved temporarily -- see URI list'), 400: ('Bad Request',
'Bad request syntax or unsupported method'),
401: ('Unauthorized',
'No permission -- see authorization schemes'),
402: ('Payment Required',
'No payment -- see charging schemes'),
403: ('Forbidden',
'Request forbidden -- authorization will not help'),
404: ('Not Found', 'Nothing matches the given URI'),
405: ('Method Not Allowed',
'Specified method is invalid for this server.'),
406: ('Not Acceptable', 'URI not available in preferred format.'),
407: ('Proxy Authentication Required', 'You must authenticate with '
'this proxy before proceeding.'),
408: ('Request Timeout', 'Request timed out; try again later.'),
409: ('Conflict', 'Request conflict.'),
410: ('Gone',
'URI no longer exists and has been permanently removed.'),
411: ('Length Required', 'Client must specify Content-Length.'),
412: ('Precondition Failed', 'Precondition in headers is false.'),
413: ('Request Entity Too Large', 'Entity is too large.'),
414: ('Request-URI Too Long', 'URI is too long.'),
415: ('Unsupported Media Type', 'Entity body in unsupported format.'),
416: ('Requested Range Not Satisfiable',
'Cannot satisfy request range.'),
417: ('Expectation Failed',
'Expect condition could not be satisfied.'), 500: ('Internal Server Error', 'Server got itself in trouble'),
501: ('Not Implemented',
'Server does not support this operation'),
502: ('Bad Gateway', 'Invalid responses from another server/proxy.'),
503: ('Service Unavailable',
'The server cannot process the request due to a high load'),
504: ('Gateway Timeout',
'The gateway server did not receive a timely response'),
505: ('HTTP Version Not Supported', 'Cannot fulfill request.'),
}
Python 3.X 要使用urllib.request 来抓取网络资源。转的更多相关文章
- Python做简单爬虫(urllib.request怎么抓取https以及伪装浏览器访问的方法)
一:抓取简单的页面: 用Python来做爬虫抓取网站这个功能很强大,今天试着抓取了一下百度的首页,很成功,来看一下步骤吧 首先需要准备工具: 1.python:自己比较喜欢用新的东西,所以用的是Pyt ...
- 使用python/casperjs编写终极爬虫-客户端App的抓取-ZOL技术频道
使用python/casperjs编写终极爬虫-客户端App的抓取-ZOL技术频道 使用python/casperjs编写终极爬虫-客户端App的抓取
- [Python爬虫] 之十四:Selenium +phantomjs抓取媒介360数据
具体代码如下: # coding=utf-8import osimport refrom selenium import webdriverimport selenium.webdriver.supp ...
- 使用Request+正则抓取猫眼电影(常见问题)
目前使用Request+正则表达式,爬取猫眼电影top100的例子很多,就不再具体阐述过程! 完整代码github:https://github.com/connordb/Top-100 总结一下,容 ...
- python网络爬虫 - 设定重试次数内反复抓取
import urllib.request def download(url, num_retries=2): print('Downloading:', url) try: html = urlli ...
- python爬虫(一)_爬虫原理和数据抓取
本篇将开始介绍Python原理,更多内容请参考:Python学习指南 为什么要做爬虫 著名的革命家.思想家.政治家.战略家.社会改革的主要领导人物马云曾经在2015年提到由IT转到DT,何谓DT,DT ...
- Python爬虫入门教程 29-100 手机APP数据抓取 pyspider
1. 手机APP数据----写在前面 继续练习pyspider的使用,最近搜索了一些这个框架的一些使用技巧,发现文档竟然挺难理解的,不过使用起来暂时没有障碍,估摸着,要在写个5篇左右关于这个框架的教程 ...
- Python爬虫【四】Scrapy+Cookies池抓取新浪微博
1.设置ROBOTSTXT_OBEY,由true变为false 2.设置DEFAULT_REQUEST_HEADERS,将其改为request headers 3.根据请求链接,发出第一个请求,设置一 ...
- Python大数据:外部数据获取(网页抓取)
import urllib2 as url import cookielib,StringIO,gzip,json import pandas as pd import numpy as np #定义 ...
随机推荐
- [ECharts] - ECharts使用中国地图
格式1: https://www.cnblogs.com/luna666/p/9007263.html (非官方) <!DOCTYPE html> <html lang=" ...
- Docker run 出现问题如何调试?
docker run -ti 3f5dd697cc83 /bin/bash #进入image的目录 ls -l #列出所有目录 dotnet run WestWin.Ads.Crawler.WebAp ...
- 使用MapperScannerConfigurer简化MyBatis配置
MyBatis的一大亮点就是可以不用DAO的实现类.如果没有实现类,Spring如何为Service注入DAO的实例呢?MyBatis-Spring提供了一个MapperFactoryBean,可以将 ...
- Latex 左右引号
参考: LaTeX技巧218:LaTeX如何正确输入引号:双引号""单引号'' Latex 左右引号 在latex中加引号时,使用""的输出为两个同向的引号: ...
- jqGrid API (转)
来源:https://www.cnblogs.com/MonaSong/p/5109991.html JQGrid是一个在jquery基础上做的一个表格控件,以ajax的方式和服务器端通信. JQGr ...
- Centos6.8安装redis(一)
最近有在学习会话共享的配置,其中一种呢是 nginx+redis+tomcat 的会话共享配置,在记录此会话共享配置之前呢先记录下redis等的安装.这篇先简单记录下redis的安装,是其中一种方式, ...
- CentOS6.5下搭建SVN服务器
1.检查是否已安装 rpm -qa | grep subversion 如果要卸载旧版本: yum remove subversion 2.安装 yum install subversion PS:y ...
- centos7安装tomcat8 新手入门 图文教程
系统环境 操作系统:64位CentOS Linux release 7.2.1511 (Core) JDK版本:1.8.0_121 下载tomcat8压缩包 访问官网:http://tomcat.ap ...
- ubuntu16系统磁盘空间/dev/vda1占用满的问题
参考文档: https://www.cnblogs.com/moonandstar08/p/6091507.html (系统磁盘空间/dev/xvda1占满原因分析) https://blog.csd ...
- bubbo调用Failed to invoke remote method异常解决
bubbo调用服务异常: com.alibaba.dubbo.rpc.RpcException: Failed to invoke remote method: getPlanFlowInfo, pr ...