爬虫之pyquery库

官方文档：https://pyquery.readthedocs.io/en/latest/

PyQuery是一个强大又灵活的网页解析库。如果你觉得正则写起来太麻烦、BeautifulSoup语法太难记，而你熟悉jQury的语法，那么PyQuery就是你的绝佳选择。

一、开始

字符串初始化：

from pyquery import PyQuery as pq

d = pq("<html>哈哈哈</html>")  # 现在d就相当于jQuery的$

print(d("html"))

URL初始化：

from pyquery import PyQuery as pq

d = pq(url="https://www.baidu.com")

print(d("head"))

文件初始化：

from pyquery import PyQuery as pq

d = pq(filename='demo.html')  # filename指定文件路径

print(d("head"))

二、基本CSS选择器

html = """

<div id="container">

    <ul class="list">

         <li class="item-0">first item</li>

         <li class="item-1"><a href="link2.html">second item</a></li>

         <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

         <li class="item-1 active"><a href="link4.html">fourth item</a></li>

         <li class="item-0"><a href="link5.html">fifth item</a></li>

     </ul>

 </div>

"""

from pyquery import PyQuery as pq

d = pq(html)

print(d("#container .list li"))

三、查找元素

子元素

d("css选择器").find("li")

html = """

<div id="container">

    <ul class="list">

         <li class="item-0">first item</li>

         <li class="item-1"><a href="link2.html">second item</a></li>

         <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

         <li class="item-1 active"><a href="link4.html">fourth item</a></li>

         <li class="item-0"><a href="link5.html">fifth item</a></li>

     </ul>

 </div>

"""

from pyquery import PyQuery as pq

d = pq(html)

items = d(".list")

print(type(items))  # <class 'pyquery.pyquery.PyQuery'>

li = items.find("li")

print(type(li))    # <class 'pyquery.pyquery.PyQuery'>

print(li)

"""

 <li class="item-0">first item</li>

 <li class="item-1"><a href="link2.html">second item</a></li>

 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

 <li class="item-1 active"><a href="link4.html">fourth item</a></li>

 <li class="item-0"><a href="link5.html">fifth item</a></li>

"""

父元素

d("css选择器").parent(<css选择器(可无)>)

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

items = d(".list")

parents = items.parents()

print(parents)

"""

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

        </ul>

    </div>

</div>

<div id="container">

    <ul class="list">

         <li class="item-0">first item</li>

         <li class="item-1"><a href="link2.html">second item</a></li>

         <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

         <li class="item-1 active"><a href="link4.html">fourth item</a></li>

         <li class="item-0"><a href="link5.html">fifth item</a></li>

    </ul>

</div>

"""

d(".list").parents()

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

items = d(".list")

parents = items.parents(".wrap")

print(parents)

"""

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

        </ul>

    </div>

</div>

"""

d(".list").parents(".wrap")

兄弟元素

d("css选择器").siblings(<css选择器(可无)>)

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d(".list .item-0.active")

print(li.siblings())

"""

<li class="item-1"><a href="link2.html">second item</a></li>

<li class="item-0">first item</li>

<li class="item-1 active"><a href="link4.html">fourth item</a></li>

<li class="item-0"><a href="link5.html">fifth item</a></li>

"""

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d(".list .item-0.active")

print(li.siblings(".active"))

"""

<li class="item-1 active"><a href="link4.html">fourth item</a></li>

"""

四、遍历

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d("li").items()

print(type(li))  # <class 'generator'>

for i in li:

    print(i)

"""

<li class="item-0">first item</li>

<li class="item-1"><a href="link2.html">second item</a></li>

<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

<li class="item-1 active"><a href="link4.html">fourth item</a></li>

<li class="item-0"><a href="link5.html">fifth item</a></li>

"""

五、获取信息

获取属性

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

a = d(".item-0.active a")

print(a.attr("href"))

print(a.attr.href)

获取文本

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

a = d(".item-0.active a")

print(a.text())

"""

third item

"""

获取html

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d(".item-0.active")

print(li)

print(li.html())

"""

<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

<a href="link3.html"><span class="bold">third item</span></a>

"""

六、DOM操作

addClass()、removeClass()

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d(".item-0.active")

print(li)

li.removeClass("active")

print(li)

li.addClass("active")

print(li)

"""

<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

<li class="item-0"><a href="link3.html"><span class="bold">third item</span></a></li>

<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

"""

attr()、css()

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d(".item-0.active")

print(li)

li.attr("name", "link")

print(li)

li.css("font-size", "14px")

print(li)

"""

<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

<li class="item-0 active" name="link"><a href="link3.html"><span class="bold">third item</span></a></li>

<li class="item-0 active" name="link" style="font-size: 14px"><a href="link3.html"><span class="bold">third item</span></a></li>

"""

remove()

html = """

<div class="wrap">

    Hello, World.

    <p>This is a paragraph.</p>

 </div>

"""

from pyquery import PyQuery as pq

d = pq(html)

wrap = d(".wrap")

print(wrap.text())

"""

Hello, World.

This is a paragraph.

"""

wrap.find("p").remove()

print(wrap.text())  # Hello, World.

其他DOM方法

https://pyquery.readthedocs.io/en/latest/api.html

七、伪类选择器

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

        </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d("li:first-child")

print(li)  # <li class="item-0">first item</li>

li = d("li:last-child")

print(li)  # <li class="item-0"><a href="link5.html">fifth item</a></li>

li = d("li:nth-child(2)")

print(li)  # <li class="item-1"><a href="link2.html">second item</a></li>

li = d("li:gt(2)")  # 从0开始计数，索引大于2

print(li)

"""

<li class="item-1 active"><a href="link4.html">fourth item</a></li>

 <li class="item-0"><a href="link5.html">fifth item</a></li>

"""

li = d("li:nth-child(2n)")  # 获取偶数顺序的元素（从1开始）

print(li)

"""

<li class="item-1"><a href="link2.html">second item</a></li>

<li class="item-1 active"><a href="link4.html">fourth item</a></li>

"""

li = d("li:contains(second)")  # 根据文本匹配，匹配文本包含second的标签

print(li)  # <li class="item-1"><a href="link2.html">second item</a></li>

爬虫之pyquery库的更多相关文章

Python爬虫之pyquery库的基本使用
# 字符串初始化 html = ''' <div> <ul> <li class = "item-0">first item</li> ...
python爬虫从入门到放弃（七）之 PyQuery库的使用
PyQuery库也是一个非常强大又灵活的网页解析库,如果你有前端开发经验的,都应该接触过jQuery,那么PyQuery就是你非常绝佳的选择,PyQuery 是 Python 仿照 jQuery 的严 ...
爬虫常用库之pyquery 库
pyquery库是jQuery的Python实现,可以用于解析HTML网页内容,我个人写过的一些抓取网页数据的脚本就是用它来解析html获取数据的.他的官方文档地址是:http://packages. ...
Python爬虫-- PyQuery库
PyQuery库 PyQuery库也是一个非常强大又灵活的网页解析库,PyQuery 是 Python 仿照 jQuery 的严格实现.语法与 jQuery 几乎完全相同,所以不用再去费心去记一些奇怪 ...
PYTHON 爬虫笔记六:PyQuery库基础用法
知识点一:PyQuery库详解及其基本使用初始化字符串初始化 html = ''' <div> <ul> <li class="item-0"&g ...
第四节：Web爬虫之pyquery解析库
PyQuery库也是一个非常强大又灵活的网页解析库,如果你有前端开发经验的,都应该接触过jQuery,那么PyQuery就是你非常绝佳的选择,PyQuery 是 Python 仿照 jQuery 的严 ...
python之爬虫（九）PyQuery库的使用
PyQuery库也是一个非常强大又灵活的网页解析库,如果你有前端开发经验的,都应该接触过jQuery,那么PyQuery就是你非常绝佳的选择,PyQuery 是 Python 仿照 jQuery 的严 ...
Python3 网络爬虫（请求库的安装）
Python3 网络爬虫(请求库的安装) 爬虫可以简单分为几步:抓取页面,分析页面和存储数据在页面爬取的过程中我们需要模拟浏览器向服务器发送请求,所以需要用到一些python库来实现HTTP的请求操 ...
爬虫之PyQuery的base了解
爬虫之PyQuery的base了解 pyquery库是jQuery的Python实现,能够以jQuery的语法来操作解析 HTML 文档,易用性和解析速度都很好,和它差不多的还有BeautifulSo ...

随机推荐

oracle-扫盲贴：存储过程实现增删改查
原文引入:http://blog.csdn.net/yangzhawen/article/details/8617179 oracle-扫盲贴:存储过程实现增删改查分类: oracle2013-02 ...
bzoj3566
3566: [SHOI2014]概率充电器 Time Limit: 40 Sec Memory Limit: 256 MBSubmit: 982 Solved: 422[Submit][Statu ...
C# 判断两个矩形是否相交
源代码 public bool JudgeRectangleIntersect(double RecAleftX, double RecAleftY, double RecArightX, doubl ...
用户能够在下次登录系统时被重新配置---或win10早期更新不成功的bug就需要删除多余的登陆用户
有时候我们希望删除本地计算机上的一位用户,或者期待一位用户能够在下次登录系统时被重新配置.但是仅删除%userprofile%目录是不够的,因为一个登录名会对应一个SSID,这个SSID还保留在注册表 ...
MySQL权限及登陆、退出方法
用户权限列表 SELECT 查询权限 INSERT 插入权限 UPDATE 更新权限 DELETE 删除权限(用于删除数据) CREATE 创建权限 DROP 删除权限(用户删除文件) RELOAD ...
2017西安网络赛B_Coin
样例输入 2 2 1 1 3 1 2 样例输出 500000004 555555560 思路: n重伯努利实验概率分布题. 设q=1-p,p为事件概率. Y为出现偶数次的概率. 所以 Y=1/2*( ...
ACM_数数有多少（第二类Stirling数-递推dp）
数数有多少 Time Limit: 2000/1000ms (Java/Others) Problem Description: 小财最近新开了一家公司,招了n个员工,但是因为资金问题,办公楼只有m间 ...
C#---数据库访问通用类、Access数据库操作类、mysql类 .
//C# 数据库访问通用类 (ADO.NET)using System;using System.Collections.Generic;using System.Text;using System. ...
17 C#中的循环执行 while循环
在编程中有代码的执行主要有三种方式.(1)顺序执行,也就是一条语句一条语句按顺序执行:(2)条件执行,也就是if...else.当某种条件满足时执行一些代码:(3)循环执行,就是当某种条件满足的时候, ...
Git学习笔记(2)-创建仓库
一.Git简介 1.Git是什么 Git是分布式版本控制系统 2.Git有什么特点 (1)Git是分布式的SCM,SVN是集中式的 (2)Git每个历史版本存储完整的文件,SVN存储文件差异 (3)G ...

爬虫之pyquery库

一、开始

二、基本CSS选择器

三、查找元素

子元素

父元素

兄弟元素

四、遍历

五、获取信息

获取属性

获取文本

获取html

六、DOM操作

addClass()、removeClass()

attr()、css()

remove()

其他DOM方法

七、伪类选择器

爬虫之pyquery库的更多相关文章

随机推荐

热门专题