etree和Beautiful Soup的使用

1.lxml 是一种使用 Python 编写的库,可以迅速、灵活地处理 XML ，支持 XPath (XML Path Language)，使用 lxml 的 etree 库来进行爬取网站信息

2.Beautiful Soup支持从HTML或XML文件中提取数据的Python库；支持Python标准库中的HTML解析器；还支持一些第三方的解析器lxml, 使用的是 Xpath 语法

Beautiful Soup自动将输入文档转换为Unicode编码，输出文档转换为utf-8编码。

我们爬取腾讯招聘网站的链接为https://hr.tencent.com/position.php?&start=10#a

需要获取职位名称、职位类别、招聘人数、工作地点、发布时间等信息

一、使用etree爬取信息

1.导入库

 from lxml import etree
 from urllib import request#进一步了解urllib和requests的区别
 import json

在python.3中使用urllib库中的request模块,保存输出为json文件

2.获取网站并写到json文件中

 response=request.urlopen('https://hr.tencent.com/position.php?&start=10#a')#获取网站链接
 resHtml=response.read()
 output=open('tencent1.json','wb+')#使用二进制方式打开，写入到json文件

如果只使用w来写入文件会报错：

write() argument must be str, not bytes

我们需要用二进制来打开改为wb+

3.获取我们需要得到的标签

 html=etree.HTML(resHtml)
 result=html.xpath('//tr[@class="odd"] | //tr[@class="even"]')#获取tr标签下的所有class只有odd和even,用|并列
 for site in result:
     item={ }

必须是字典形式，先定义一个空字典

     name=site.xpath('./td[1]/a')[0].text
     detailLink=site.xpath('./td[1]/a')[0].attrib['href']
     catalog=site.xpath('./td[2]')[0].text
     recruitNumber=site.xpath('./td[3]')[0].text
     workLocation=site.xpath('./td[4]')[0].text
     publishTime=site.xpath('./td[5]')[0].text

找到我们需要的字段

4.规范输出形式

     print(type(name))
     print(name,detailLink,catalog,recruitNumber,workLocation,publishTime)
     item['name']=name
     item['detailLink']=detailLink
     item['catalog']=catalog
     item['recruitNumber']=recruitNumber
     item['publishTime']=publishTime

     line = json.dumps(item,ensure_ascii=False) + '\n'
     print(line)
     output.write(line.encode('utf-8'))#编码格式

 output.close()

运行后结果如下：

<class 'str'>
23677-互娱服务采购经理 position_detail.php?id=44802&keywords=&tid=0&lid=0 职能类 1 深圳 2018-10-16
{"catalog": "职能类", "name": "23677-互娱服务采购经理", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44802&keywords=&tid=0&lid=0"}

<class 'str'>
22989-腾讯云块存储底层开发工程师（深圳） position_detail.php?id=44803&keywords=&tid=0&lid=0 技术类 2 深圳 2018-10-16
{"catalog": "技术类", "name": "22989-腾讯云块存储底层开发工程师（深圳）", "recruitNumber": "2", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44803&keywords=&tid=0&lid=0"}

<class 'str'>
24549-渠道管理经理（政策管理方向-上海） position_detail.php?id=44804&keywords=&tid=0&lid=0 市场类 1 上海 2018-10-16
{"catalog": "市场类", "name": "24549-渠道管理经理（政策管理方向-上海）", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44804&keywords=&tid=0&lid=0"}

<class 'str'>
24549-渠道管理经理（ROC管理方向-上海） position_detail.php?id=44805&keywords=&tid=0&lid=0 市场类 1 上海 2018-10-16
{"catalog": "市场类", "name": "24549-渠道管理经理（ROC管理方向-上海）", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44805&keywords=&tid=0&lid=0"}

<class 'str'>
24549-广告营销业务分析师（上海） position_detail.php?id=44806&keywords=&tid=0&lid=0 市场类 1 上海 2018-10-16
{"catalog": "市场类", "name": "24549-广告营销业务分析师（上海）", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44806&keywords=&tid=0&lid=0"}

<class 'str'>
28297-RPG手游—市场和平台渠道推广（深圳） position_detail.php?id=44809&keywords=&tid=0&lid=0 产品/项目类 1 深圳 2018-10-16
{"catalog": "产品/项目类", "name": "28297-RPG手游—市场和平台渠道推广（深圳）", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44809&keywords=&tid=0&lid=0"}

<class 'str'>
21309-在线教育-运营视觉设计师（深圳） position_detail.php?id=44800&keywords=&tid=0&lid=0 设计类 2 深圳 2018-10-16
{"catalog": "设计类", "name": "21309-在线教育-运营视觉设计师（深圳）", "recruitNumber": "2", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44800&keywords=&tid=0&lid=0"}

<class 'str'>
21309-在线教育-UI设计师（深圳） position_detail.php?id=44801&keywords=&tid=0&lid=0 设计类 2 深圳 2018-10-16
{"catalog": "设计类", "name": "21309-在线教育-UI设计师（深圳）", "recruitNumber": "2", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44801&keywords=&tid=0&lid=0"}

<class 'str'>
22989-数据库高级产品运营经理 position_detail.php?id=44795&keywords=&tid=0&lid=0 产品/项目类 1 北京 2018-10-16
{"catalog": "产品/项目类", "name": "22989-数据库高级产品运营经理", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44795&keywords=&tid=0&lid=0"}

<class 'str'>
27087-海外区域中心空间运营经理（深圳） position_detail.php?id=44797&keywords=&tid=0&lid=0 市场类 1 深圳 2018-10-16
{"catalog": "市场类", "name": "27087-海外区域中心空间运营经理（深圳）", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44797&keywords=&tid=0&lid=0"}

导出的json文件如下：

{"catalog": "职能类", "name": "23677-互娱服务采购经理", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44802&keywords=&tid=0&lid=0"}
{"catalog": "技术类", "name": "22989-腾讯云块存储底层开发工程师（深圳）", "recruitNumber": "2", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44803&keywords=&tid=0&lid=0"}
{"catalog": "市场类", "name": "24549-渠道管理经理（政策管理方向-上海）", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44804&keywords=&tid=0&lid=0"}
{"catalog": "市场类", "name": "24549-渠道管理经理（ROC管理方向-上海）", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44805&keywords=&tid=0&lid=0"}
{"catalog": "市场类", "name": "24549-广告营销业务分析师（上海）", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44806&keywords=&tid=0&lid=0"}
{"catalog": "产品/项目类", "name": "28297-RPG手游—市场和平台渠道推广（深圳）", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44809&keywords=&tid=0&lid=0"}
{"catalog": "设计类", "name": "21309-在线教育-运营视觉设计师（深圳）", "recruitNumber": "2", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44800&keywords=&tid=0&lid=0"}
{"catalog": "设计类", "name": "21309-在线教育-UI设计师（深圳）", "recruitNumber": "2", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44801&keywords=&tid=0&lid=0"}
{"catalog": "产品/项目类", "name": "22989-数据库高级产品运营经理", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44795&keywords=&tid=0&lid=0"}
{"catalog": "市场类", "name": "27087-海外区域中心空间运营经理（深圳）", "recruitNumber": "1", "publishTime": "2018-10-16", "detailLink": "position_detail.php?id=44797&keywords=&tid=0&lid=0"}

二、使用Beautiful Soup爬取信息

1.导入库

  from bs4 import BeautifulSoup from urllib import request import json 

2.获取网站并写到json文件中

 response=request.urlopen('https://hr.tencent.com/position.php?&start=10#a')
 resHtml=response.read()
 output=open('tencent2.json','wb+')

3.获取我们需要得到的标签

 html = BeautifulSoup(resHtml,'lxml')
 result = html.select('tr[class="even"]')
 result2= html.select('tr[class="odd"]')
 result+=result2
 print(len(result))

 for site in result:
     item = {}

     name = site.select('td a')[0].get_text()
     detailLink = site.select('td a')[0].attrs['href']#Tag就是 HTML 中的一个个标签，它的两个属性是name和attrs
     catalog = site.select('td ')[1].get_text()
     recruitNumber = site.select('td ')[2].get_text()
     workLocation = site.select('td ')[3].get_text()
     publishTime = site.select('td ')[4].get_text()

4.规范输出形式

  item['name']=name
     item['detailLink'] = detailLink
     item['catalog'] = catalog
     item['recruitNumber'] = recruitNumber
     item['workLocation'] = workLocation
     item['publishTime'] = publishTime

     line = json.dumps(item,ensure_ascii=False)
     print(line)

     output.write(line.encode('utf-8'))

 output.close()

运行结果如下：

 10
 {"detailLink": "position_detail.php?id=44802&keywords=&tid=0&lid=0", "catalog": "职能类", "publishTime": "2018-10-16", "recruitNumber": "1", "name": "23677-互娱服务采购经理", "workLocation": "深圳"}
 {"detailLink": "position_detail.php?id=44804&keywords=&tid=0&lid=0", "catalog": "市场类", "publishTime": "2018-10-16", "recruitNumber": "1", "name": "24549-渠道管理经理（政策管理方向-上海）", "workLocation": "上海"}
 {"detailLink": "position_detail.php?id=44806&keywords=&tid=0&lid=0", "catalog": "市场类", "publishTime": "2018-10-16", "recruitNumber": "1", "name": "24549-广告营销业务分析师（上海）", "workLocation": "上海"}
 {"detailLink": "position_detail.php?id=44800&keywords=&tid=0&lid=0", "catalog": "设计类", "publishTime": "2018-10-16", "recruitNumber": "2", "name": "21309-在线教育-运营视觉设计师（深圳）", "workLocation": "深圳"}
 {"detailLink": "position_detail.php?id=44795&keywords=&tid=0&lid=0", "catalog": "产品/项目类", "publishTime": "2018-10-16", "recruitNumber": "1", "name": "22989-数据库高级产品运营经理", "workLocation": "北京"}
 {"detailLink": "position_detail.php?id=44803&keywords=&tid=0&lid=0", "catalog": "技术类", "publishTime": "2018-10-16", "recruitNumber": "2", "name": "22989-腾讯云块存储底层开发工程师（深圳）", "workLocation": "深圳"}
 {"detailLink": "position_detail.php?id=44805&keywords=&tid=0&lid=0", "catalog": "市场类", "publishTime": "2018-10-16", "recruitNumber": "1", "name": "24549-渠道管理经理（ROC管理方向-上海）", "workLocation": "上海"}
 {"detailLink": "position_detail.php?id=44809&keywords=&tid=0&lid=0", "catalog": "产品/项目类", "publishTime": "2018-10-16", "recruitNumber": "1", "name": "28297-RPG手游—市场和平台渠道推广（深圳）", "workLocation": "深圳"}
 {"detailLink": "position_detail.php?id=44801&keywords=&tid=0&lid=0", "catalog": "设计类", "publishTime": "2018-10-16", "recruitNumber": "2", "name": "21309-在线教育-UI设计师（深圳）", "workLocation": "深圳"}
 {"detailLink": "position_detail.php?id=44797&keywords=&tid=0&lid=0", "catalog": "市场类", "publishTime": "2018-10-16", "recruitNumber": "1", "name": "27087-海外区域中心空间运营经理（深圳）", "workLocation": "深圳"}

以上为两种方法爬取网站信息，个人觉得用Beautiful Soup爬取比较方便