爬虫--PyQuery
什么是PyQuery？
PyQuery
初始化
字符串初始化
from pyquery import PyQuery as pq html="""
<div>
<ul>
<li class="item-0">first item</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-0 active">
<a href="link3.html>
<span class="bold">third item</span>
</a>
</li>
<li class="item-1 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-0">
<a href="link5.html">
fifth item
</a>
</li>
</ul>
</div>
""" doc=pq(html)
print(doc("li"))
<li class="item-0">first item</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-0 active">
<a href="link3.html> <span class=" bold="">third item
</a>
</li>
<li class="item-1 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-0">
<a href="link5.html">
fifth item
</a>
</li>
打印后的结果为：
URL初始化
from pyquery import PyQuery as pq doc = pq(url="http://www.baidu.com")
print(doc("head"))
<head><meta http-equiv="content-type" content="text/html;charset=utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=Edge"/><meta content="always" name="referrer"/><link rel="stylesheet" type="text/css" href="http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css"/><title>ç¾åº¦ä¸ä¸ï¼ä½ å°±ç¥é</title></head>
打印后的结果为：
文件初始化
from pyquery import PyQuery as pq doc = pq(filename="demo.html")
print(doc("li"))
基本CSS选择器
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul>
<li class="item-0">
first item
</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-2 active">
<a href="link3.html>
<span class="bold">third item</span>
</a>
</li>
<li class="item-3 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-4">
<a href="link5.html">
fifth item
</a>
</li>
</ul>
</div>
"""
doc = pq(html)
print(doc("#container .item-0"))
<li class="item-0">
first item
</li>
打印的结果为：
查找元素
子元素
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list>
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
print(type(items))
docs = doc.find("li")
print(type(docs))
print(docs)
<class 'pyquery.pyquery.PyQuery'>
<class 'pyquery.pyquery.PyQuery'>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
docs = items.children() # 查找所有的直接子元素
docs1 = items.children(".active")
print(docs)
print(docs1)
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
打印后的结果为：
父元素
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
docs = items.parent() # 查找所有的直接父元素
print(docs)
print(type(docs))
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div> <class 'pyquery.pyquery.PyQuery'>
打印后的结果：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".list")
docs = items.parents() # 查找所有的直接子元素
print(type(docs))
print(docs)
<class 'pyquery.pyquery.PyQuery'>
<html><body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div></body></html><body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div></body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div><div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
打印后的结果为：
兄弟元素
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0")
print(type(items.siblings()))
print(items.siblings())
<class 'pyquery.pyquery.PyQuery'>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0active") # 同时匹配.item-0和active，若都包含item-0 + active则打印，否则不打印
item = doc(".item-0")
print(item.siblings())
print(type(items.siblings()))
print(items.siblings())
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li> <class 'pyquery.pyquery.PyQuery'>
打印后的结果为：
遍历
单个元素
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0.active") # 同时匹配.item-0和active，若都包含item-0 + active则打印，否则不打印
print(type(items))
print(items)
<class 'pyquery.pyquery.PyQuery'>
<li class="item-0 active">first item</li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc("li").items() # 遍历items ,此时的items是一个迭代器
print(type(items))
for li in items:
print(li)
<class 'generator'>
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
获取信息
获取属性
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1 a")
print(items.attr("href"))
print(items.attr.href)
link2.html
link2.html
打印后的结果为：
获取文本
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.text())
second item
打印后的结果为：
获取html
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items)
print(items.html())
<li class="item-1"><a href="link2.html">second item</a></li> <a href="link2.html">second item</a>
打印后的结果为：
DOM操作
addClass , removeClass
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.remove_class("active"))
print(items.add_class("actives"))
<li class="item-1"><a href="link2.html">second item</a></li> <li class="item-1 actives"><a href="link2.html">second item</a></li>
打印后的结果为：
attr , css
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.attr("name","names"))
print(items.css("font-size","14px"))
<li class="item-1 active" name="names"><a href="link2.html">second item</a></li> <li class="item-1 active" name="names" style="font: 14px"><a href="link2.html">second item</a></li>
打印后的结果为：
remove
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.text())
print("---------------")
items.find('p').remove()
print(items.text())
second item
Third times
---------------
second item
打印后的结果为：
其他DOM方法
https://pythonhosted.org/pyquery/api.html
伪类选择器
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
li = doc("li:first-child")
print(li)
print("------------------------------------------------------------")
li = doc("li:last-child")
print(li)
print("------------------------------------------------------------")
li = doc("li:gt(2)")
print(li)
print("------------------------------------------------------------")
li = doc("li:nth-child(2)") # 指定一个索引顺序，获取第二个li标签
print(li)
print("------------------------------------------------------------")
li = doc("li:nth-child(2n)") # 指定一个索引顺序，获取偶数的li标签
print(li)
print("------------------------------------------------------------")
li = doc("li:contains(second)") # 查找包含second的文本标签
print(li)
<li class="item-0 active">first item</li>
------------------------------------------------------------
<li class="item-4"><a href="link5.html">fifth item</a></li>
------------------------------------------------------------
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
打印后的结果为：
更多的CSS选择器可以查看：http://www.w3school.com.cn/css/index.asp
Pyquery的官方文档可以查看：https://pythonhosted.org/pyquery/api.html
爬虫--PyQuery的更多相关文章
- 一起学爬虫——PyQuery常用用法总结
什么是PyQuery PyQuery是一个类似于jQuery的解析网页工具,使用lxml操作xml和html文档,它的语法和jQuery很像.和XPATH,Beautiful Soup比起来,PyQu ...
- Python爬虫-- PyQuery库
PyQuery库 PyQuery库也是一个非常强大又灵活的网页解析库,PyQuery 是 Python 仿照 jQuery 的严格实现.语法与 jQuery 几乎完全相同,所以不用再去费心去记一些奇怪 ...
- 爬虫--pyquery使用
强大又灵活的网页解析库. 初始化 字符串初始化 html = ''' <div> <ul> <li class="item-0">first ...
- Python新手需要掌握的知识点
一.基础语法 1 变量 2 逻辑判断 3 循环 4 函数 二.数据结构 1 数字(加减乘除) 2 字符串(一串字符) 3 布尔 (真假) 4 元组 (不能修改的列表) 5 列表(Python的苦力,最 ...
- 一起学爬虫——使用selenium和pyquery爬取京东商品列表
layout: article title: 一起学爬虫--使用selenium和pyquery爬取京东商品列表 mathjax: true --- 今天一起学起使用selenium和pyquery爬 ...
- Python爬虫之PyQuery使用（六）
Python爬虫之PyQuery使用 PyQuery简介 pyquery能够通过选择器精确定位 DOM 树中的目标并进行操作.pyquery相当于jQuery的python实现,可以用于解析HTML网 ...
- Python爬虫利器六之PyQuery的用法
前言 你是否觉得 XPath 的用法多少有点晦涩难记呢? 你是否觉得 BeautifulSoup 的语法多少有些悭吝难懂呢? 你是否甚至还在苦苦研究正则表达式却因为少些了一个点而抓狂呢? 你是否已经有 ...
- PYTHON 爬虫笔记十:利用selenium+PyQuery实现淘宝美食数据搜集并保存至MongeDB（实战项目三）
利用selenium+PyQuery实现淘宝美食数据搜集并保存至MongeDB 目标站点分析 淘宝页面信息很复杂的,含有各种请求参数和加密参数,如果直接请求或者分析Ajax请求的话会很繁琐.所以我们可 ...
- 小白学 Python 爬虫（23）：解析库 pyquery 入门
人生苦短,我用 Python 前文传送门: 小白学 Python 爬虫(1):开篇 小白学 Python 爬虫(2):前置准备(一)基本类库的安装 小白学 Python 爬虫(3):前置准备(二)Li ...
随机推荐
- DNS原理及解析过程
本文主要参考自:http://369369.blog.51cto.com/319630/812889 并做了小幅修改 什么是DNS? 因特网上的主机和人类一样,也可以使用多种方式进行识别.主机的一种识 ...
- iOS----MRC（手动内存管理）
1.MRC是什么,有什么用? 在苹果开发中,我们是没有垃圾回收机制的.所以在ARC推出之前,我们苹果开发程序员需要通过手动代码的形式尽量严密的管理我们的App的内存: ---------------- ...
- 【bzoj4542】[Hnoi2016]大数 莫队算法
题目描述 给出一个数字串,多次询问一段区间有多少个子区间对应的数为P的倍数.其中P为质数. 输入 第一行一个整数:P.第二行一个串:S.第三行一个整数:M.接下来M行,每行两个整数 fr,to,表示对 ...
- 病毒侵袭 HDU - 2896（ac自动机 板题）
当太阳的光辉逐渐被月亮遮蔽,世界失去了光明,大地迎来最黑暗的时刻....在这样的时刻,人们却异常兴奋——我们能在有生之年看到500年一遇的世界奇观,那是多么幸福的事儿啊~~ 但网路上总有那么些网站,开 ...
- 小Z的袜子 题解报告【莫队】
Description 作为一个生活散漫的人,小Z每天早上都要耗费很久从一堆五颜六色的袜子中找出一双来穿.终于有一天,小Z再也无法忍受这恼人的找袜子过程,于是他决定听天由命-- 具体来说,小Z把这N只 ...
- selenium - switch_to_alert() - 警告框处理
在WebDriver中处理JavaScript所生成的alert.confirm以及prompt十分简单,具体做法是使用 switch_to.alert 方法定位到 alert/confirm/pro ...
- bzoj 2929 [Poi1999]洞穴攀行 网络流
2929: [Poi1999]洞穴攀行 Time Limit: 1 Sec Memory Limit: 128 MBSubmit: 499 Solved: 295[Submit][Status][ ...
- UVA-11582 数学
UVA-11582 题意: 求f[a^b]%n ,其中f是斐波那契数列,1<=n<=1000,0<=a,b<=2^64; 代码: //这题重点是要发现 f[i]%n会出现循环, ...
- js 时间戳 转化
new Date((1524142795*1000)).toJSON().slice(11,16)
- need
php面试题: http://lib.csdn.net/article/php/43624 小程序: 官方手册: https://mp.weixin.qq.com/debug/wxadoc/dev/d ...