《python网络数据采集》笔记2

1.网页表单与登陆窗口

Requests 库擅长处理那些复杂的 HTTP 请求、cookie、header（响应头和请求头）等内容。

1)表单提交

import requests

#字段

params = {'firstname': 'Ryan', 'lastname': 'Mitchell'}

#请求方式:post    表单提交页面

r = requests.post("http://pythonscraping.com/pages/files/processing.php", data=params)

print(r.text)

2)提交文件和图像

#表单

<form action="processing2.php" method="post" enctype="multipart/form-data">

Submit a jpg, png, or gif: <input type="file" name="image"><br>

<input type="submit" value="Upload File">

</form>

#提交文件

import requests

files = {'uploadFile': open('../files/Python-logo.png', 'rb')}

r = requests.post("http://pythonscraping.com/pages/processing2.php",

files=files)

print(r.text)

3)处理登陆和cookie

#从请求结果中获取 cookie，打印登录状态的验证结果,然后再通过 cookies 参数把 cookie 发送到简介页面。

import requests

params = {'username': 'Ryan', 'password': 'password'}

r = requests.post("http://pythonscraping.com/pages/cookies/welcome.php", params)

print(r.cookies.get_dict())

r = requests.get("http://pythonscraping.com/pages/cookies/profile.php",cookies=r.cookies)

print(r.text)

#会话（session）对象（调用 requests.Session() 获取）会持续跟踪会话信息，像 cookie、header、运行 HTTP 协议的信息

import requests

session = requests.Session()

params = {'username': 'username', 'password': 'password'}

s = session.post("http://pythonscraping.com/pages/cookies/welcome.php", params)

print(s.cookies.get_dict())

s = session.get("http://pythonscraping.com/pages/cookies/profile.php")

print(s.text)

#Requests 库有一个 auth 模块专门用来处理 HTTP 认证：

import requests

from requests.auth import HTTPBasicAuth

auth = HTTPBasicAuth('ryan', 'password')

r = requests.post(url="http://pythonscraping.com/pages/auth/login.php", auth=auth)

print(r.text)