Python 动态网页Fetch/XHR爬虫——以获取NBA球员信息为例

Python 动态网页Fetch/XHR爬虫——以获取NBA球员信息为例

动态网页抓取信息，一般利用F12开发者工具-网络-Fetch/XHR获取信息，实现难点有：

1. 1. 动态网页的加载方式

   2. 获取请求Url

   3. 编排处理Headers

   4. 分析返回的数据Json

   5. pandas DataFrame的处理

我们本次想获取的信息如下：

成功获取到的csv一共506位球员，具体如下：

实现代码：

 import requests
 import pandas as pd
 ​
 def get_headers(header_raw):
    return dict(line.split(": ", 1) for line in header_raw.split("\n") if line != '')
 ​
 # 设置headers
 headers_str = '''
 accept: application/json, text/plain, */*
 accept-encoding: gzip, deflate, br
 accept-language: zh-CN,zh;q=0.9
 referer: https://china.nba.cn/playerindex/
 sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"
 sec-ch-ua-mobile: ?0
 sec-ch-ua-platform: "Windows"
 sec-fetch-dest: empty
 sec-fetch-mode: cors
 sec-fetch-site: same-origin
 cookie: sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%22182d0029f842fc-0d281a685dd4e08-4303066-2400692-182d0029f85406%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22identities%22%3A%22eyIkaWRlbnRpdHlfY29va2llX2lkIjoiMTgyZDAwMjlmODQyZmMtMGQyODFhNjg1ZGQ0ZTA4LTQzMDMwNjYtMjQwMDY5Mi0xODJkMDAyOWY4NTQwNiJ9%22%2C%22history_login_id%22%3A%7B%22name%22%3A%22%22%2C%22value%22%3A%22%22%7D%2C%22%24device_id%22%3A%22182d0029f842fc-0d281a685dd4e08-4303066-2400692-182d0029f85406%22%7D; privacyV2=true; i18next=zh_CN; locale=zh_CN
 user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
 '''
 headers = get_headers(headers_str)
 # print(headers)
 ​
 # requests请求
 param = {'locale': 'zh_CN'}
 url = 'https://china.nba.cn/stats2/league/playerlist.json'
 response = requests.get(url=url, headers=headers, params=param)
 ​
 print('返回状态码：', response.status_code)
 print('编码：', response.encoding)
 ​
 # json解码成字典
 myjson = response.json()
 ​
 # 保存为pandas DataFrame
 # print(players_dicts['playerProfile'])
 # print(players_dicts['teamProfile'])
 ​
 # 遍历选手信息
 players_info = []
 for players_dicts in myjson['payload']['players']:
     players_info.append(pd.DataFrame([players_dicts['playerProfile']]))
 ​
 # 遍历队伍简介信息
 teams_info = []
 for players_dicts in myjson['payload']['players']:
     teams_info.append(pd.DataFrame([players_dicts['teamProfile']]))
 ​
 # 得到两个DataFrame
 players_pandas = pd.concat(players_info)
 teams_pandas = pd.concat(teams_info)
 ​
 # 合并得到最终DataFrame
 result = pd.concat([players_pandas, teams_pandas], axis=1)
 result.to_csv(r'C:\Users\WeiRonbbin\Desktop\NBA_Players1.csv')