Python爬虫之PyQuery使用

PyQuery简介

pyquery能够通过选择器精确定位 DOM 树中的目标并进行操作。pyquery相当于jQuery的python实现，可以用于解析HTML网页等。它的语法与jQuery几乎完全相同，对于使用过jQuery的人来说很熟悉，也很好上手。

初始化

有 4 种方法可以进行初始化：
可以通过传入字符串、lxml、文件或者 url 来使用PyQuery

from pyquery import PyQuery as pq

from lxml import etree

d = pq("<html></html>")#传入字符串

d = pq(etree.fromstring("<html></html>"))#传入lxml

d = pq(url='http://baidu.com/') #传入url

d = pq(filename=path_to_html_file) #传入文件

基本CSS选择器

html='''

<html>

 <body>

  <ul class="mh-col">

   <li class="g-ellipsis">

    <a class="g-a-noline" data-md='{"b":"list","p":"1-1"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E6%AD%8C%E6%89%8B%E9%AB%98%E7%A9%BA%E6%8B%8DMV%E5%9D%A0%E4%BA%A1&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     歌手高空拍MV坠亡

    </a>

    <span class="mh-ico-up">

    </span>

   </li>

   <li class="g-ellipsis">

    <a class="g-a-noline" data-md='{"b":"list","p":"1-2"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BD%9B%E7%A5%96%E6%9C%B1%E9%BE%99%E5%B9%BF%E9%87%91%E5%A9%9A&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D&fr=hao_360so_history_b" target="_blank">

     佛祖朱龙广金婚

    </a>

    <span class="mh-ico-down">

    </span>

   </li>

   <li class="g-ellipsis">

    <a class="g-a-noline" data-md='{"b":"list","p":"1-3"}' href="http://www.so.com/link?url=http%3A%2F%2Fbaike.so.com%2Fzt%2Fzufangfangpian.html%3Fsrc%3Dreci&q=%E5%91%A8%E6%9D%B0%E4%BC%A6%E7%9A%84%E6%AD%8C&ts=1540365398&t=c6890eee0e669832ca96d5223582d6e" target="_blank">

     常见租房陷阱

    </a>

    <span class="mh-ico-up">

    </span>

   </li>

   <li class="g-ellipsis">

    <a class="g-a-noline" data-md='{"b":"list","p":"1-4"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E9%9D%B3%E4%B8%9C%E5%9B%9E%E5%BA%94%E5%8F%91%E9%94%99%E8%AF%97%E8%AF%8D&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     靳东回应发错诗词

    </a>

   </li>

   <li class="g-ellipsis">

    <a class="g-a-noline" data-md='{"b":"list","p":"1-5"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%85%A4%E8%80%81%E6%9D%BF%E4%BB%AC%E7%9A%84%E5%BD%B1%E8%A7%86%E6%B1%9F%E6%B9%96&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     煤老板们的影视江湖

    </a>

   </li>

   <li class="g-ellipsis">

    <a class="g-a-noline" data-md='{"b":"list","p":"1-6"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=1024%E7%A8%8B%E5%BA%8F%E5%91%98%E8%8A%82&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     1024程序员节

    </a>

   </li>

   <li class="g-ellipsis">

    <a class="g-a-noline" data-md='{"b":"list","p":"1-7"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%BE%8E%E7%9A%84%E5%90%88%E5%B9%B6%E5%B0%8F%E5%A4%A9%E9%B9%85&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     美的合并小天鹅

    </a>

   </li>

   <li class="g-ellipsis">

    <a class="g-a-noline" data-md='{"b":"list","p":"1-8"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BA%AC%E6%98%86%E9%AB%98%E9%80%9F4%E8%BD%A6%E7%9B%B8%E6%92%9E&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     京昆高速4车相撞

    </a>

   </li>

  </ul>

 </body>

</html>

'''

from pyquery import PyQuery as pq
doc = pq(html)

# 获取所有a标签
print(doc('body .mh-col li a'))

注意：
类名用.
id用#
标签用标签名

另外选择的是具有层级关系，从左到右，不是直接的父子的关系。

运行结果如下：

<a class="g-a-noline" data-md="{"b":"list","p":"1-1"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E6%AD%8C%E6%89%8B%E9%AB%98%E7%A9%BA%E6%8B%8DMV%E5%9D%A0%E4%BA%A1&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     歌手高空拍MV坠亡

    </a>

    <a class="g-a-noline" data-md="{"b":"list","p":"1-2"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BD%9B%E7%A5%96%E6%9C%B1%E9%BE%99%E5%B9%BF%E9%87%91%E5%A9%9A&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D&fr=hao_360so_history_b" target="_blank">

     佛祖朱龙广金婚

    </a>

    <a class="g-a-noline" data-md="{"b":"list","p":"1-3"}" href="http://www.so.com/link?url=http%3A%2F%2Fbaike.so.com%2Fzt%2Fzufangfangpian.html%3Fsrc%3Dreci&q=%E5%91%A8%E6%9D%B0%E4%BC%A6%E7%9A%84%E6%AD%8C&ts=1540365398&t=c6890eee0e669832ca96d5223582d6e" target="_blank">

     常见租房陷阱

    </a>

    <a class="g-a-noline" data-md="{"b":"list","p":"1-4"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E9%9D%B3%E4%B8%9C%E5%9B%9E%E5%BA%94%E5%8F%91%E9%94%99%E8%AF%97%E8%AF%8D&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     靳东回应发错诗词

    </a>

   <a class="g-a-noline" data-md="{"b":"list","p":"1-5"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%85%A4%E8%80%81%E6%9D%BF%E4%BB%AC%E7%9A%84%E5%BD%B1%E8%A7%86%E6%B1%9F%E6%B9%96&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     煤老板们的影视江湖

    </a>

   <a class="g-a-noline" data-md="{"b":"list","p":"1-6"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=1024%E7%A8%8B%E5%BA%8F%E5%91%98%E8%8A%82&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     1024程序员节

    </a>

   <a class="g-a-noline" data-md="{"b":"list","p":"1-7"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%BE%8E%E7%9A%84%E5%90%88%E5%B9%B6%E5%B0%8F%E5%A4%A9%E9%B9%85&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     美的合并小天鹅

    </a>

   <a class="g-a-noline" data-md="{"b":"list","p":"1-8"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BA%AC%E6%98%86%E9%AB%98%E9%80%9F4%E8%BD%A6%E7%9B%B8%E6%92%9E&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     京昆高速4车相撞

    </a>

操作

html='''

<html>

 <body>

  <ul class="mh-col">

   <li class="g-ellipsis1">

    <a class="g-a-noline1" data-md='{"b":"list","p":"1-1"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E6%AD%8C%E6%89%8B%E9%AB%98%E7%A9%BA%E6%8B%8DMV%E5%9D%A0%E4%BA%A1&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     歌手高空拍MV坠亡

    </a>

    <span class="mh-ico-up">

    </span>

   </li>

   <li class="g-ellipsis2">

    <a class="g-a-noline2" data-md='{"b":"list","p":"1-2"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BD%9B%E7%A5%96%E6%9C%B1%E9%BE%99%E5%B9%BF%E9%87%91%E5%A9%9A&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D&fr=hao_360so_history_b" target="_blank">

     佛祖朱龙广金婚

    </a>

    <span class="mh-ico-down">

    </span>

   </li>

   <li class="g-ellipsis3">

    <a class="g-a-noline3" data-md='{"b":"list","p":"1-3"}' href="http://www.so.com/link?url=http%3A%2F%2Fbaike.so.com%2Fzt%2Fzufangfangpian.html%3Fsrc%3Dreci&q=%E5%91%A8%E6%9D%B0%E4%BC%A6%E7%9A%84%E6%AD%8C&ts=1540365398&t=c6890eee0e669832ca96d5223582d6e" target="_blank">

     常见租房陷阱

    </a>

    <span class="mh-ico-up">

    </span>

   </li>

   <li class="g-ellipsis4">

    <a class="g-a-noline4" data-md='{"b":"list","p":"1-4"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E9%9D%B3%E4%B8%9C%E5%9B%9E%E5%BA%94%E5%8F%91%E9%94%99%E8%AF%97%E8%AF%8D&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     靳东回应发错诗词

    </a>

   </li>

   <li class="g-ellipsis5">

    <a class="g-a-noline5" data-md='{"b":"list","p":"1-5"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%85%A4%E8%80%81%E6%9D%BF%E4%BB%AC%E7%9A%84%E5%BD%B1%E8%A7%86%E6%B1%9F%E6%B9%96&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     煤老板们的影视江湖

    </a>

   </li>

   <li class="g-ellipsis6">

    <a class="g-a-noline6" data-md='{"b":"list","p":"1-6"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=1024%E7%A8%8B%E5%BA%8F%E5%91%98%E8%8A%82&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     1024程序员节

    </a>

   </li>

   <li class="g-ellipsis7">

    <a class="g-a-noline7" data-md='{"b":"list","p":"1-7"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%BE%8E%E7%9A%84%E5%90%88%E5%B9%B6%E5%B0%8F%E5%A4%A9%E9%B9%85&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     美的合并小天鹅

    </a>

   </li>

   <li class="g-ellipsis8">

    <a class="g-a-noline8" data-md='{"b":"list","p":"1-8"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BA%AC%E6%98%86%E9%AB%98%E9%80%9F4%E8%BD%A6%E7%9B%B8%E6%92%9E&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">

     <p>新闻</p>

     京昆高速4车相撞

    </a>

   </li>

  </ul>

 </body>

</html>

'''

from pyquery import PyQuery as pq

doc = pq(html)

items = doc('.mh-col')

#.find()：查找嵌套元素

alist = items.find('li a')

print(alist)


#查找所有子元素

alist2 = items.children()

print(alist2)

#查找指定的子元素

alist3 = items.children('.g-ellipsis1')
print(alist2)

#查找父元素 
#注意：一个元素只有一个父元素 
body = items.parent() 
print(body) 

#查找祖先元素 
content = items.parents() 
print(content) 

#查找兄弟元素 
li = doc('.mh-col .g-ellipsis1') 
print(li.siblings()) 

#遍历 单个元素 
#遍历所有的a标签
alist =doc('.mh-col li a').items() 
　　for a in alist: print(a)

获取信息

获取属性

a =doc('.mh-col li .g-a-noline8')

print(a.attr['href'])

print(a.attr.href)

获取文本

a =doc('.mh-col li .g-a-noline8')

print(a.text())

获取HTML

a =doc('.mh-col li .g-a-noline8')

print(a.html())

简单的DOM操作

#addClass、removerClass
#修改类名

a =doc('.mh-col li .g-a-noline8')

print(a)

a.removeClass('g-a-noline8')

print(a)

a.addClass('g-a-noline8')

print(a)

#attr、css
#修改属性和样式

a =doc('.mh-col li .g-a-noline8')

print(a)

a.attr('name','link')

print(a)

a.css('font-size','14px')

print(a)

#remove
#删除标签
li = doc('.mh-col .g-ellipsis8')
print(li)
li.find('a').remove()
print(li)

更多的DOM操作：https://pyquery.readthedocs.io/en/latest/api.html

Python爬虫之PyQuery使用（六）的更多相关文章

[Python爬虫] 之二十六：Selenium +phantomjs 利用 pyquery抓取智能电视网站图片信息
一.介绍本例子用Selenium +phantomjs爬取智能电视网站(http://www.tvhome.com/news/)的资讯信息,输入给定关键字抓取图片信息. 给定关键字:数字:融合:电视 ...
python爬虫神器PyQuery的使用方法
你是否觉得 XPath 的用法多少有点晦涩难记呢? 你是否觉得 BeautifulSoup 的语法多少有些悭吝难懂呢? 你是否甚至还在苦苦研究正则表达式却因为少些了一个点而抓狂呢? 你是否已经有了一些 ...
Python爬虫小白入门（六）爬取披头士乐队历年专辑封面-网易云音乐
一.前言前文说过我的设计师小伙伴的设计需求,他想做一个披头士乐队历年专辑的瀑布图. 通过搜索,发现网易云音乐上有比较全的历年专辑信息加配图,图片质量还可以,虽然有大有小. 我的例子怎么都是爬取图片? ...
python爬虫之PyQuery的基本使用
PyQuery库也是一个非常强大又灵活的网页解析库,如果你有前端开发经验的,都应该接触过jQuery,那么PyQuery就是你非常绝佳的选择,PyQuery 是 Python 仿照 jQuery 的严 ...
python爬虫之pyquery学习
相关内容: pyquery的介绍 pyquery的使用安装模块导入模块解析对象初始化 css选择器在选定元素之后的元素再选取元素的文本.属性等内容的获取 pyquery执行DOM操作.css ...
【Python爬虫】PyQuery解析库
PyQuery解析库阅读目录初始化基本CSS选择器查找元素遍历获取信息 DOM操作伪类选择器 PyQuery 是 Python 仿照 jQuery 的严格实现.语法与 jQuery 几乎 ...
爬取网易云音乐评论！python 爬虫入门实战（六）selenium 入门！
说到爬虫,第一时间可能就会想到网易云音乐的评论.网易云音乐评论里藏了许多宝藏,那么让我们一起学习如何用 python 挖宝藏吧! 既然是宝藏,肯定是用要用钥匙加密的.打开 Chrome 分析 Head ...
Python爬虫之pyquery库的基本使用
# 字符串初始化 html = ''' <div> <ul> <li class = "item-0">first item</li> ...
python爬虫知识点总结（六）BeautifulSoup库详解
官方学习文档:https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh/ 一.什么时BeautifulSoup? 答:灵活又方便的网页解析库,处 ...

随机推荐

MySQL中查询行数最多的表并且排序
#切换到schema use information_schema; #查询数据量最大的30张表并排序 select table_name,table_rows from tables order ...
C# 反编译项目修复
1.反编译测试程序 1>.将测试程序添加到.NET Reflector 2>.选中测试程序后右键选择导出 2.反编译项目修复 1>.问题一问题现象: base.AutoScaleM ...
img格式镜像转ISO格式
在做汇编学习时,需要用比较老的Windows XP来进行调试学习,因此找了最老的Windows XP(CN_WINXP_PRO_ISO,无SP版本 ),下载后发现镜像文件格式是img的,而virtua ...
Java泛型方法与泛型类的使用------------（五）
泛型的本质就是将数据类型也参数化, 普通方法的输入参数的值是可以变的,但是类型(比如: String)是不能变的,它使得了在面对不同类型的输入参数的时候我们要重载方法才行. 泛型就是将这个数据类型也搞 ...
电脑kail linux 连接手机Nethunter，手机和电脑互传文件
1.开启nethunter的ssh 修改/etc/ssh/sshd_config 参考:解决kali linux 开启ssh服务后连接不上的问题 2.如果在手机终端修改不了(我的就是怎么也改不了),可 ...
python高级编程读书笔记（一）
python高级编程读书笔记(一) python 高级编程读书笔记,记录一下基础和高级用法 python2和python3兼容处理使用sys模块使程序python2和python3兼容 import ...
c# LINQ 使用
linq是个好东西,让开发人员省时省力.很多人可能只知道怎么使用, 对它没有全面深入的了解.所谓磨刀不误砍柴工,今天就来学习下. 一.与LINQ有关的语言特性 1.扩展方法在System.Linq命 ...
WordCloud词云包的安装
1,下载 https://www.lfd.uci.edu/~gohlke/pythonlibs/#wordcloud 2,安装 (window环境安装) 找的下载文件的路径安装 pip instal ...
Servlet注释与部署描述符
值得注意的是,部署描述符优先于注释.换句话说,部署描述符覆盖通过注释机制所规定的配置信息.Web 部署描述符的 3.0 版本在 web-app 元素上包含一种名为 metadata-complete ...
Go语言规格说明书之变量声明（Variable/Short variable declarations）
go version go1.11 windows/amd64 本文为阅读Go语言中文官网的规则说明书(https://golang.google.cn/ref/spec)而做的笔记,完整的介绍Go语 ...

Python爬虫之PyQuery使用（六）