BeautifulSoup解析库的介绍和使用
### BeautifulSoup解析库的介绍和使用
### 三大选择器:节点选择器,方法选择器,CSS选择器
### 使用建议:方法选择器 > CSS选择器 > 节点选择器 ## 测试文本
text = '''
<html><head><title>there is money</title></head>
<body>
<p class="title" name="dmr"><b>there is money</b></p>
<p class="money">good good study, day day up
<a href="https://www.baidu.com/1" class="error" id="l1"><span><!-- 1 --></span></a>,
<a href="https://www.baidu.com/2" class="error" id="l2"><span>2</span></a> and
<a href="https://www.baidu.com/3" class="error" id="l3">3</a>;
66666666666
</p>
<p class='body'>...</p>
'''
1. 基本用法
## 基本用法
from bs4 import BeautifulSoup # 初始化BeautifulSoup对象,选择lxml类型
soup = BeautifulSoup(text, 'lxml')
# 以标准的缩进格式输出
print(soup.prettify())
# 提取title节点的文本内容
print(soup.title.string) '''
输出内容:
<html>
<head>
<title>
there is money
</title>
</head>
<body>
<p class="title" name="dmr">
<b>
there is money
</b>
</p>
<p class="money">
good good study, day day up
<a class="error" href="https://www.baidu.com/1" id="l1">
<!-- 1 -->
</a>
,
<a class="error" href="https://www.baidu.com/2" id="l2">
2
</a>
and
<a class="error" href="https://www.baidu.com/3" id="l3">
3
</a>
;
66666666666
</p>
<p class="body">
...
</p>
</body>
</html>
there is money
'''
2. 节点选择器
### 节点选择器
from bs4 import BeautifulSoup soup = BeautifulSoup(text, 'lxml')
print(type(soup))
print(soup.title)
print(type(soup.title))
print(soup.p)
print(soup.head) '''
输出结果:
<class 'bs4.BeautifulSoup'>
<title>there is money</title>
<class 'bs4.element.Tag'>
<p class="title" name="dmr"><b>there is money</b></p>
<head><title>there is money</title></head>
''' ## 提取信息
from bs4 import BeautifulSoup soup = BeautifulSoup(text, 'lxml')
# 提取title标签的文本内容
print(soup.title.string)
# p表情的名称
print(soup.p.name)
# p标签的属性,字典格式
print(soup.p.attrs)
print(soup.p.attrs.get('name'))
# attrs可省略,直接以字典的提取方式进行信息提取
print(soup.p['class'])
print(soup.p.get('class'))
print(soup.p.string) '''
输出内容:
there is money
p
{'class': ['title'], 'name': 'dmr'}
dmr
['title']
['title']
there is money
''' ## 嵌套选择,套中套 from bs4 import BeautifulSoup soup = BeautifulSoup(text, 'lxml')
print(soup.body.p.string) '''
输出内容:
there is money
''' ## 关联选择
## 子节点和子孙节点
from bs4 import BeautifulSoup soup = BeautifulSoup(text, 'lxml')
# 直接子节点,包含换行符文本内容等;contents获取到一个list, children生成一个迭代器(建议使用)
print(soup.body.contents)
print(len(soup.body.contents))
print(soup.body.children)
for i, child in enumerate(soup.body.children):
print(i, child)
print(soup.body.descendants)
for j, item in enumerate(soup.body.descendants):
print(j, item) '''
输出结果:
['\n', <p class="title" name="dmr"><b>there is money</b></p>, '\n', <p class="money">good good study, day day up
<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>,
<a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a> and
<a class="error" href="https://www.baidu.com/3" id="l3">3</a>;
66666666666
</p>, '\n', <p class="body">...</p>, '\n']
7
<list_iterator object at 0x0000000002DAD320>
0 1 <p class="title" name="dmr"><b>there is money</b></p>
2 3 <p class="money">good good study, day day up
<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>,
<a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a> and
<a class="error" href="https://www.baidu.com/3" id="l3">3</a>;
66666666666
</p>
4 5 <p class="body">...</p>
6 <generator object Tag.descendants at 0x0000000002D67E58>
0 1 <p class="title" name="dmr"><b>there is money</b></p>
2 <b>there is money</b>
3 there is money
4 5 <p class="money">good good study, day day up
<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>,
<a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a> and
<a class="error" href="https://www.baidu.com/3" id="l3">3</a>;
66666666666
</p>
6 good good study, day day up 7 <a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>
8 <span><!-- 1 --></span>
9 1
10 , 11 <a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a>
12 <span>2</span>
13 2
14 and 15 <a class="error" href="https://www.baidu.com/3" id="l3">3</a>
16 3
17 ;
66666666666 18 19 <p class="body">...</p>
20 ...
21
''' ## 父节点和祖先节点
from bs4 import BeautifulSoup soup = BeautifulSoup(text, 'lxml')
print(soup.a.parent)
print(soup.a.parents)
for i, parent in enumerate(soup.a.parents):
print(i, parent) '''
输出结果:
<p class="money">good good study, day day up
<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>,
<a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a> and
<a class="error" href="https://www.baidu.com/3" id="l3">3</a>;
66666666666
</p>
<generator object PageElement.parents at 0x0000000002D68E58>
0 <p class="money">good good study, day day up
<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>,
<a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a> and
<a class="error" href="https://www.baidu.com/3" id="l3">3</a>;
66666666666
</p>
1 <body>
<p class="title" name="dmr"><b>there is money</b></p>
<p class="money">good good study, day day up
<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>,
<a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a> and
<a class="error" href="https://www.baidu.com/3" id="l3">3</a>;
66666666666
</p>
<p class="body">...</p>
</body>
2 <html><head><title>there is money</title></head>
<body>
<p class="title" name="dmr"><b>there is money</b></p>
<p class="money">good good study, day day up
<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>,
<a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a> and
<a class="error" href="https://www.baidu.com/3" id="l3">3</a>;
66666666666
</p>
<p class="body">...</p>
</body></html>
3 <html><head><title>there is money</title></head>
<body>
<p class="title" name="dmr"><b>there is money</b></p>
<p class="money">good good study, day day up
<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>,
<a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a> and
<a class="error" href="https://www.baidu.com/3" id="l3">3</a>;
66666666666
</p>
<p class="body">...</p>
</body></html>
''' ## 兄弟节点
from bs4 import BeautifulSoup soup = BeautifulSoup(text, 'lxml')
print('Next sibling: ', soup.a.next_sibling)
print('Previous sibling: ', soup.a.previous_sibling)
print('Next siblings: ', soup.a.next_siblings)
print('Previous siblings: ', soup.a.previous_sibling) '''
输出结果:
Next sibling: , Previous sibling: good good study, day day up Next siblings: <generator object PageElement.next_siblings at 0x0000000002D67E58>
Previous siblings: good good study, day day up
'''
3. 方法选择器
### 方法选择器,较为灵活
## find_all方法,查询所有符合条件的,返回一个列表,元素类型为tag
## find方法,查询符合条件的第一个元素,返回一个tag类型对象
## 同理,find_parents和find_parent
## find_next_siblings和find_next_sibling
## find_previous_siblings和find_previous_sibling
## find_all_next和find_next
## find_all_previous和find_previous
from bs4 import BeautifulSoup
import re soup = BeautifulSoup(text, 'lxml')
# 找到节点名为a的节点,为一个列表
print(soup.find_all(name='a'))
print(soup.find_all(name='a')[0])
# 找到id属性为l1, class属性为error的节点
print(soup.find_all(attrs={'id': 'l1'}))
print(soup.find_all(class_='error'))
# 通过文本关键字来进行匹配文本内容
print(soup.find_all(text=re.compile('money'))) '''
输出内容:
[<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>, <a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a>, <a class="error" href="https://www.baidu.com/3" id="l3">3</a>]
<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>
[<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>]
[<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>, <a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a>, <a class="error" href="https://www.baidu.com/3" id="l3">3</a>]
['there is money', 'there is money']
'''
4. CSS选择器
### CSS选择器,select方法,返回一个列表
from bs4 import BeautifulSoup soup = BeautifulSoup(text, 'lxml')
print(soup.select('p a'))
print(soup.select('.error'))
print(soup.select('#l1 span'))
print(soup.select('a'))
print(type(soup.select('a'))) '''
输出内容:
[<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>, <a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a>, <a class="error" href="https://www.baidu.com/3" id="l3">3</a>]
[<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>, <a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a>, <a class="error" href="https://www.baidu.com/3" id="l3">3</a>]
[<span><!-- 1 --></span>]
[<a class="error" href="https://www.baidu.com/1" id="l1"><span><!-- 1 --></span></a>, <a class="error" href="https://www.baidu.com/2" id="l2"><span>2</span></a>, <a class="error" href="https://www.baidu.com/3" id="l3">3</a>]
<class 'bs4.element.ResultSet'>
''' ## 嵌套选择,获取属性,获取文本
from bs4 import BeautifulSoup soup = BeautifulSoup(text, 'lxml')
# 嵌套选择
for i in soup.select('a'):
print(i.select('span'))
# 获取属性
print(soup.select('a')[0].attrs)
print(soup.select('a')[0].get('class'))
# 获取文本
print(soup.select('a')[1].string)
print(soup.select('a')[2].get_text()) '''
输出结果:
[<span><!-- 1 --></span>]
[<span>2</span>]
[]
{'href': 'https://www.baidu.com/1', 'class': ['error'], 'id': 'l1'}
['error']
2
3
'''
BeautifulSoup解析库的介绍和使用的更多相关文章
- BeautifulSoup解析库
解析库 解析器 使用方法 优势 劣势 Python标准库 BeautifulSoup(html, 'html.parser') 速度适中,容错能力强 老版本python容错能力差 lxml HTML解 ...
- 第三节:Web爬虫之BeautifulSoup解析库
Beautiful Soup官方说明: Beautiful Soup提供一些简单的.python式的函数用来处理导航.搜索.修改分析树等功能.它是一个工具箱,通过解析文档为用户提供需要抓取的数据,因为 ...
- pyquery解析库的介绍和使用
### pyquery的介绍和使用 ## 测试文本 text = ''' <html><head><title>there is money</title&g ...
- BeautifulSoup与Xpath解析库总结
一.BeautifulSoup解析库 1.快速开始 html_doc = """ <html><head><title>The Dor ...
- xpath beautiful pyquery三种解析库
这两天看了一下python常用的三种解析库,写篇随笔,整理一下思路.太菜了,若有错误的地方,欢迎大家随时指正.......(conme on.......) 爬取网页数据一般会经过 获取信息-> ...
- Python爬虫3大解析库使用导航
1. Xpath解析库 2. BeautifulSoup解析库 3. PyQuery解析库
- 爬虫模块介绍--Beautifulsoup (解析库模块,正则)
Beautiful Soup 是一个可以从HTML或XML文件中提取数据的Python库.它能够通过你喜欢的转换器实现惯用的文档导航,查找,修改文档的方式.Beautiful Soup会帮你节省数小时 ...
- 爬虫 解析库re,Beautifulsoup,
re模块 点我回顾 Beautifulsoup模块 #安装 Beautiful Soup pip install beautifulsoup4 #安装解析器 Beautiful Soup支持Pytho ...
- 解析库之re,Beautifulsoup
本篇导航: 介绍 基本使用 遍历文档树 搜索文档树 总结 re模块在之前的python进阶中有讲过不再做过多的阐述,本篇为BeautifulSoup库的分析 20.collections模块和 ...
随机推荐
- NorFlash、NandFlash在技术和应用上有些什么区别?
首先你要搞懂什么是Flash Memory? Flash Memory(快闪存储器),是一种电子式可清除程序化只读存储器的形式,允许在操作中被多次擦或写的存储器.这种科技主要用于一般性数据存储,以及在 ...
- Jquery取值方法汇总
一.下拉框 1.jquery获取当前选中select的text值 var a = $("#ShareMoneyType").find("option:selected&q ...
- Python matplotlib numpy 曼德尔布罗特集合 曼德尔布罗特 B.Madelbrot
import numpy as np import matplotlib.pyplot as plt def mandelbrot(h,w,maxit=20): y,x = np.ogrid[-1.4 ...
- APP 自动化之appium元素定位(三)
APP自动化测试关键环节--元素定位,以下我们来了解appium提供的元素定位方法! 1. id定位,id一个控件的唯一标识,由开发人员在项目中指定,如果一个元素有对应的resource-id,我们就 ...
- vue禁用浏览器回退
解决方案 mounted() { history.pushState(null, null, document.URL) window.addEventListener('popstate', () ...
- Cannot load module file xxx.iml的两种解决方法
一. 一种是点击左上角File,然后点击Invalidate Caches / Restart...,弹出对话框再点击Invalidate and Restart等待工程重新加载,问题就解决了. 二. ...
- PTA 6-1 求采用邻接矩阵作为存储结构的无向图各顶点的度 (6分)
PTA 6-1 求采用邻接矩阵作为存储结构的无向图各顶点的度 (6分) 函数接口定义: 函数接口为: void dgree(MGraph G); G为采用邻接矩阵作为存储结构的有向图 裁判测试程序样例 ...
- 『学了就忘』Linux基础命令 — 37、Linux中挂载操作的相关命令
目录 1.mount命令介绍 (1)mount命令说明 (2)mount命令格式 2.mount命令示例 3.mount -a命令说明 4.-o特殊选项说明 5.exec/noexec选项说明 挂载就 ...
- git修改用户和邮箱
GIT查看当前用户以及邮箱 $ git config user.name $ git config user.email GIT修改用户以及邮箱 $ git config --global user. ...
- SpringCloud 2020.0.4 系列之 Stream 消息出错重试 与 死信队列 的实现
1. 概述 老话说的好:出错不怕,怕的是出了错,却不去改正.如果屡次出错,无法改对,就先记下了,然后找援军解决. 言归正传,今天来聊一下 Stream 组件的 出错重试 和 死信队列. RabbitM ...