爬虫--PyQuery
什么是PyQuery？
PyQuery
初始化
字符串初始化
from pyquery import PyQuery as pq html="""
<div>
<ul>
<li class="item-0">first item</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-0 active">
<a href="link3.html>
<span class="bold">third item</span>
</a>
</li>
<li class="item-1 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-0">
<a href="link5.html">
fifth item
</a>
</li>
</ul>
</div>
""" doc=pq(html)
print(doc("li"))
<li class="item-0">first item</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-0 active">
<a href="link3.html> <span class=" bold="">third item
</a>
</li>
<li class="item-1 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-0">
<a href="link5.html">
fifth item
</a>
</li>
打印后的结果为：
URL初始化
from pyquery import PyQuery as pq doc = pq(url="http://www.baidu.com")
print(doc("head"))
<head><meta http-equiv="content-type" content="text/html;charset=utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=Edge"/><meta content="always" name="referrer"/><link rel="stylesheet" type="text/css" href="http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css"/><title>ç¾åº¦ä¸ä¸ï¼ä½ å°±ç¥é</title></head>
打印后的结果为：
文件初始化
from pyquery import PyQuery as pq doc = pq(filename="demo.html")
print(doc("li"))
基本CSS选择器
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul>
<li class="item-0">
first item
</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-2 active">
<a href="link3.html>
<span class="bold">third item</span>
</a>
</li>
<li class="item-3 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-4">
<a href="link5.html">
fifth item
</a>
</li>
</ul>
</div>
"""
doc = pq(html)
print(doc("#container .item-0"))
<li class="item-0">
first item
</li>
打印的结果为：
查找元素
子元素
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list>
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
print(type(items))
docs = doc.find("li")
print(type(docs))
print(docs)
<class 'pyquery.pyquery.PyQuery'>
<class 'pyquery.pyquery.PyQuery'>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
docs = items.children() # 查找所有的直接子元素
docs1 = items.children(".active")
print(docs)
print(docs1)
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
打印后的结果为：
父元素
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
docs = items.parent() # 查找所有的直接父元素
print(docs)
print(type(docs))
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div> <class 'pyquery.pyquery.PyQuery'>
打印后的结果：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".list")
docs = items.parents() # 查找所有的直接子元素
print(type(docs))
print(docs)
<class 'pyquery.pyquery.PyQuery'>
<html><body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div></body></html><body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div></body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div><div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
打印后的结果为：
兄弟元素
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0")
print(type(items.siblings()))
print(items.siblings())
<class 'pyquery.pyquery.PyQuery'>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0active") # 同时匹配.item-0和active，若都包含item-0 + active则打印，否则不打印
item = doc(".item-0")
print(item.siblings())
print(type(items.siblings()))
print(items.siblings())
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li> <class 'pyquery.pyquery.PyQuery'>
打印后的结果为：
遍历
单个元素
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0.active") # 同时匹配.item-0和active，若都包含item-0 + active则打印，否则不打印
print(type(items))
print(items)
<class 'pyquery.pyquery.PyQuery'>
<li class="item-0 active">first item</li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc("li").items() # 遍历items ,此时的items是一个迭代器
print(type(items))
for li in items:
print(li)
<class 'generator'>
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
获取信息
获取属性
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1 a")
print(items.attr("href"))
print(items.attr.href)
link2.html
link2.html
打印后的结果为：
获取文本
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.text())
second item
打印后的结果为：
获取html
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items)
print(items.html())
<li class="item-1"><a href="link2.html">second item</a></li> <a href="link2.html">second item</a>
打印后的结果为：
DOM操作
addClass , removeClass
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.remove_class("active"))
print(items.add_class("actives"))
<li class="item-1"><a href="link2.html">second item</a></li> <li class="item-1 actives"><a href="link2.html">second item</a></li>
打印后的结果为：
attr , css
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.attr("name","names"))
print(items.css("font-size","14px"))
<li class="item-1 active" name="names"><a href="link2.html">second item</a></li> <li class="item-1 active" name="names" style="font: 14px"><a href="link2.html">second item</a></li>
打印后的结果为：
remove
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.text())
print("---------------")
items.find('p').remove()
print(items.text())
second item
Third times
---------------
second item
打印后的结果为：
其他DOM方法
https://pythonhosted.org/pyquery/api.html
伪类选择器
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
li = doc("li:first-child")
print(li)
print("------------------------------------------------------------")
li = doc("li:last-child")
print(li)
print("------------------------------------------------------------")
li = doc("li:gt(2)")
print(li)
print("------------------------------------------------------------")
li = doc("li:nth-child(2)") # 指定一个索引顺序，获取第二个li标签
print(li)
print("------------------------------------------------------------")
li = doc("li:nth-child(2n)") # 指定一个索引顺序，获取偶数的li标签
print(li)
print("------------------------------------------------------------")
li = doc("li:contains(second)") # 查找包含second的文本标签
print(li)
<li class="item-0 active">first item</li>
------------------------------------------------------------
<li class="item-4"><a href="link5.html">fifth item</a></li>
------------------------------------------------------------
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
打印后的结果为：
更多的CSS选择器可以查看：http://www.w3school.com.cn/css/index.asp
Pyquery的官方文档可以查看：https://pythonhosted.org/pyquery/api.html
爬虫--PyQuery的更多相关文章
- 一起学爬虫——PyQuery常用用法总结
什么是PyQuery PyQuery是一个类似于jQuery的解析网页工具,使用lxml操作xml和html文档,它的语法和jQuery很像.和XPATH,Beautiful Soup比起来,PyQu ...
- Python爬虫-- PyQuery库
PyQuery库 PyQuery库也是一个非常强大又灵活的网页解析库,PyQuery 是 Python 仿照 jQuery 的严格实现.语法与 jQuery 几乎完全相同,所以不用再去费心去记一些奇怪 ...
- 爬虫--pyquery使用
强大又灵活的网页解析库. 初始化 字符串初始化 html = ''' <div> <ul> <li class="item-0">first ...
- Python新手需要掌握的知识点
一.基础语法 1 变量 2 逻辑判断 3 循环 4 函数 二.数据结构 1 数字(加减乘除) 2 字符串(一串字符) 3 布尔 (真假) 4 元组 (不能修改的列表) 5 列表(Python的苦力,最 ...
- 一起学爬虫——使用selenium和pyquery爬取京东商品列表
layout: article title: 一起学爬虫--使用selenium和pyquery爬取京东商品列表 mathjax: true --- 今天一起学起使用selenium和pyquery爬 ...
- Python爬虫之PyQuery使用（六）
Python爬虫之PyQuery使用 PyQuery简介 pyquery能够通过选择器精确定位 DOM 树中的目标并进行操作.pyquery相当于jQuery的python实现,可以用于解析HTML网 ...
- Python爬虫利器六之PyQuery的用法
前言 你是否觉得 XPath 的用法多少有点晦涩难记呢? 你是否觉得 BeautifulSoup 的语法多少有些悭吝难懂呢? 你是否甚至还在苦苦研究正则表达式却因为少些了一个点而抓狂呢? 你是否已经有 ...
- PYTHON 爬虫笔记十:利用selenium+PyQuery实现淘宝美食数据搜集并保存至MongeDB（实战项目三）
利用selenium+PyQuery实现淘宝美食数据搜集并保存至MongeDB 目标站点分析 淘宝页面信息很复杂的,含有各种请求参数和加密参数,如果直接请求或者分析Ajax请求的话会很繁琐.所以我们可 ...
- 小白学 Python 爬虫（23）：解析库 pyquery 入门
人生苦短,我用 Python 前文传送门: 小白学 Python 爬虫(1):开篇 小白学 Python 爬虫(2):前置准备(一)基本类库的安装 小白学 Python 爬虫(3):前置准备(二)Li ...
随机推荐
- 【第九周】beta-review阶段贡献分分配
组名: 新蜂 组长: 武志远 组员: 宫成荣 谢孝淼 杨柳 李峤 项目名称: java俄罗斯方块NEO 武志远 武志远 武志远 武志远 武志远 宫成荣 宫成荣 杨柳 宫成荣 宫成荣 李峤 杨柳 李峤 ...
- ci上传图片
o_upload.php <?php /** * Created by PhpStorm. * User: brady * Date: 2018/3/15 * Time: 14:10 */ cl ...
- Mysql中关键词执行顺序
MySQL的语句执行顺序 MySQL的语句一共分为11步,最先执行的总是FROM操作,最后执行的是LIMIT操作.其中每一个操作都会产生一张虚拟的表,这个虚拟的表作为一个处理的输入,只是这些虚拟的表对 ...
- postman 断言学习
请求 url :https://www.v2ex.com/api/nodes/show.json?name=python get请求 postman发起请求并做断言 断言: tests["B ...
- display:table的几个妙用：垂直居中、浮动……
一.为什么不用table系表格元素? 目前,在大多数开发环境中,已经基本不用table元素来做网页布局了,取而代之的是div+css,那么为什么不用table系表格元素呢? 1.用DIV+CSS编写出 ...
- 【bzoj3774】最优选择 网络流最小割
题目描述 小N手上有一个N*M的方格图,控制某一个点要付出Aij的代价,然后某个点如果被控制了,或者他周围的所有点(上下左右)都被控制了,那么他就算是被选择了的.一个点如果被选择了,那么可以得到Bij ...
- [AT2172] [agc007_e] Shik and Travel
题目链接 AtCoder:https://agc007.contest.atcoder.jp/tasks/agc007_e 洛谷:https://www.luogu.org/problemnew/sh ...
- CF623D birthday 贪心 概率期望
题意:n个人,玩抓人游戏,每抓住一个人都要猜这个人是谁.对于每一局,第i个人有$p_{i}$的概率被抓到.游戏结束当且仅当每个人都在某局中被抓到并且猜中自己的名字,求一个合适的策略来使得期望游戏局数最 ...
- 【BZOJ3203】保护出题人（动态规划，斜率优化）
[BZOJ3203]保护出题人(动态规划,斜率优化) 题面 BZOJ 洛谷 题解 在最优情况下,肯定是存在某只僵尸在到达重点的那一瞬间将其打死 我们现在知道了每只僵尸到达终点的时间,因为僵尸要依次打死 ...
- 【BZOJ4889】不勤劳的图书管理员（树套树）
[BZOJ4889]不勤劳的图书管理员(树套树) 题面 又是权限题,烦死了 洛谷真好 题解 分开考虑每一次交换产生的贡献. 假设交换\((x,y)\) 检查\(x\)与\(y\)对于区间\([x+1, ...