爬虫--PyQuery
什么是PyQuery？
PyQuery
初始化
字符串初始化
from pyquery import PyQuery as pq html="""
<div>
<ul>
<li class="item-0">first item</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-0 active">
<a href="link3.html>
<span class="bold">third item</span>
</a>
</li>
<li class="item-1 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-0">
<a href="link5.html">
fifth item
</a>
</li>
</ul>
</div>
""" doc=pq(html)
print(doc("li"))
<li class="item-0">first item</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-0 active">
<a href="link3.html> <span class=" bold="">third item
</a>
</li>
<li class="item-1 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-0">
<a href="link5.html">
fifth item
</a>
</li>
打印后的结果为：
URL初始化
from pyquery import PyQuery as pq doc = pq(url="http://www.baidu.com")
print(doc("head"))
<head><meta http-equiv="content-type" content="text/html;charset=utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=Edge"/><meta content="always" name="referrer"/><link rel="stylesheet" type="text/css" href="http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css"/><title>ç¾åº¦ä¸ä¸ï¼ä½ å°±ç¥é</title></head>
打印后的结果为：
文件初始化
from pyquery import PyQuery as pq doc = pq(filename="demo.html")
print(doc("li"))
基本CSS选择器
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul>
<li class="item-0">
first item
</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-2 active">
<a href="link3.html>
<span class="bold">third item</span>
</a>
</li>
<li class="item-3 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-4">
<a href="link5.html">
fifth item
</a>
</li>
</ul>
</div>
"""
doc = pq(html)
print(doc("#container .item-0"))
<li class="item-0">
first item
</li>
打印的结果为：
查找元素
子元素
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list>
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
print(type(items))
docs = doc.find("li")
print(type(docs))
print(docs)
<class 'pyquery.pyquery.PyQuery'>
<class 'pyquery.pyquery.PyQuery'>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
docs = items.children() # 查找所有的直接子元素
docs1 = items.children(".active")
print(docs)
print(docs1)
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
打印后的结果为：
父元素
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
docs = items.parent() # 查找所有的直接父元素
print(docs)
print(type(docs))
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div> <class 'pyquery.pyquery.PyQuery'>
打印后的结果：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".list")
docs = items.parents() # 查找所有的直接子元素
print(type(docs))
print(docs)
<class 'pyquery.pyquery.PyQuery'>
<html><body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div></body></html><body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div></body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div><div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
打印后的结果为：
兄弟元素
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0")
print(type(items.siblings()))
print(items.siblings())
<class 'pyquery.pyquery.PyQuery'>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0active") # 同时匹配.item-0和active，若都包含item-0 + active则打印，否则不打印
item = doc(".item-0")
print(item.siblings())
print(type(items.siblings()))
print(items.siblings())
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li> <class 'pyquery.pyquery.PyQuery'>
打印后的结果为：
遍历
单个元素
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0.active") # 同时匹配.item-0和active，若都包含item-0 + active则打印，否则不打印
print(type(items))
print(items)
<class 'pyquery.pyquery.PyQuery'>
<li class="item-0 active">first item</li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc("li").items() # 遍历items ,此时的items是一个迭代器
print(type(items))
for li in items:
print(li)
<class 'generator'>
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
获取信息
获取属性
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1 a")
print(items.attr("href"))
print(items.attr.href)
link2.html
link2.html
打印后的结果为：
获取文本
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.text())
second item
打印后的结果为：
获取html
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items)
print(items.html())
<li class="item-1"><a href="link2.html">second item</a></li> <a href="link2.html">second item</a>
打印后的结果为：
DOM操作
addClass , removeClass
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.remove_class("active"))
print(items.add_class("actives"))
<li class="item-1"><a href="link2.html">second item</a></li> <li class="item-1 actives"><a href="link2.html">second item</a></li>
打印后的结果为：
attr , css
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.attr("name","names"))
print(items.css("font-size","14px"))
<li class="item-1 active" name="names"><a href="link2.html">second item</a></li> <li class="item-1 active" name="names" style="font: 14px"><a href="link2.html">second item</a></li>
打印后的结果为：
remove
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.text())
print("---------------")
items.find('p').remove()
print(items.text())
second item
Third times
---------------
second item
打印后的结果为：
其他DOM方法
https://pythonhosted.org/pyquery/api.html
伪类选择器
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
li = doc("li:first-child")
print(li)
print("------------------------------------------------------------")
li = doc("li:last-child")
print(li)
print("------------------------------------------------------------")
li = doc("li:gt(2)")
print(li)
print("------------------------------------------------------------")
li = doc("li:nth-child(2)") # 指定一个索引顺序，获取第二个li标签
print(li)
print("------------------------------------------------------------")
li = doc("li:nth-child(2n)") # 指定一个索引顺序，获取偶数的li标签
print(li)
print("------------------------------------------------------------")
li = doc("li:contains(second)") # 查找包含second的文本标签
print(li)
<li class="item-0 active">first item</li>
------------------------------------------------------------
<li class="item-4"><a href="link5.html">fifth item</a></li>
------------------------------------------------------------
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
打印后的结果为：
更多的CSS选择器可以查看：http://www.w3school.com.cn/css/index.asp
Pyquery的官方文档可以查看：https://pythonhosted.org/pyquery/api.html
爬虫--PyQuery的更多相关文章
- 一起学爬虫——PyQuery常用用法总结
什么是PyQuery PyQuery是一个类似于jQuery的解析网页工具,使用lxml操作xml和html文档,它的语法和jQuery很像.和XPATH,Beautiful Soup比起来,PyQu ...
- Python爬虫-- PyQuery库
PyQuery库 PyQuery库也是一个非常强大又灵活的网页解析库,PyQuery 是 Python 仿照 jQuery 的严格实现.语法与 jQuery 几乎完全相同,所以不用再去费心去记一些奇怪 ...
- 爬虫--pyquery使用
强大又灵活的网页解析库. 初始化 字符串初始化 html = ''' <div> <ul> <li class="item-0">first ...
- Python新手需要掌握的知识点
一.基础语法 1 变量 2 逻辑判断 3 循环 4 函数 二.数据结构 1 数字(加减乘除) 2 字符串(一串字符) 3 布尔 (真假) 4 元组 (不能修改的列表) 5 列表(Python的苦力,最 ...
- 一起学爬虫——使用selenium和pyquery爬取京东商品列表
layout: article title: 一起学爬虫--使用selenium和pyquery爬取京东商品列表 mathjax: true --- 今天一起学起使用selenium和pyquery爬 ...
- Python爬虫之PyQuery使用（六）
Python爬虫之PyQuery使用 PyQuery简介 pyquery能够通过选择器精确定位 DOM 树中的目标并进行操作.pyquery相当于jQuery的python实现,可以用于解析HTML网 ...
- Python爬虫利器六之PyQuery的用法
前言 你是否觉得 XPath 的用法多少有点晦涩难记呢? 你是否觉得 BeautifulSoup 的语法多少有些悭吝难懂呢? 你是否甚至还在苦苦研究正则表达式却因为少些了一个点而抓狂呢? 你是否已经有 ...
- PYTHON 爬虫笔记十:利用selenium+PyQuery实现淘宝美食数据搜集并保存至MongeDB（实战项目三）
利用selenium+PyQuery实现淘宝美食数据搜集并保存至MongeDB 目标站点分析 淘宝页面信息很复杂的,含有各种请求参数和加密参数,如果直接请求或者分析Ajax请求的话会很繁琐.所以我们可 ...
- 小白学 Python 爬虫（23）：解析库 pyquery 入门
人生苦短,我用 Python 前文传送门: 小白学 Python 爬虫(1):开篇 小白学 Python 爬虫(2):前置准备(一)基本类库的安装 小白学 Python 爬虫(3):前置准备(二)Li ...
随机推荐
- 在linux下编译线程程序undefined reference to `pthread_create'
由于是Linux新手,所以现在才开始接触线程编程,照着GUN/Linux编程指南中的一个例子输入编译,结果出现如下错误:undefined reference to 'pthread_create'u ...
- 开源人脸识别face_recognition
环境:python36 1.安装dlib.face_recognition windows版 下载dlib,cp后面是py版本 下载地址:https://pypi.org/simple/dlib/ 提 ...
- UVA11735_Corner the Queens
题目是这样的,游戏规则,每个人轮流将二维空间上的皇后往下,往左或者往斜下45度的方向移动. 谁第一个移动到0,0的位置就获胜. 题目给定你若干个矩形,求矩形中任取一点且该点必胜的概率有概率. 其实是这 ...
- BZOJ3714 PA2014Kuglarz（最小生成树）
每次询问所获得的可以看做是两个前缀和的异或.我们只要知道任意前缀和的异或就可以得到答案了.并且显然地,如果知道了a和b的异或及a和c的异或,也就知道了b和c的异或.所以一次询问可以看做是在两点间连边, ...
- 【EF】EntityFramework DBFirst的使用
一.前言 久闻EF大名,之前做C/S产品用的是Dapper对SqlLite进行ORM.然后接触公司授权系统后发现用的是EntityFramework对SQLSever进行ORM.授权系统 ...
- BZOJ1176：[Balkan2007]Mokia——题解
http://www.lydsy.com/JudgeOnline/problem.php?id=1176 Description(题面本人自行修改了一下) 维护一个W*W的矩阵,初始值均为0.每次操作 ...
- matlab图像
1.在网络上发现matlab能画出一些很有意思的图形(立体爱心) clc; const=0; x=-5:0.05:5;y=-5:0.05:5;z=-5:0.05:5; [x,y,z]=meshgrid ...
- 《剑指offer》— JavaScript（14）链表中倒数第k个结点
链表中倒数第k个结点 题目描述 输入一个链表,输出该链表中倒数第k个结点. 思路 两个指针,先让第一个指针和第二个指针都指向头结点,然后再让第一个指正走(k-1)步,到达第k个节点: 然后两个指针同时 ...
- jsp 基本原理
jsp 的本质是 servlet,当用户请求 servlet 的时候,servlet 利用输出流动态输出 HTML 内容. 由于包括了大量的 HTML 标签.大量的静态文本等,导致 servlet 开 ...
- 怎么用spring cloud service-id 进行调用接口
这里最关键的就是加上@LoadBalanced @SpringBootApplication public class ConsumerMovieApplication { @Bean @LoadBa ...