简介

官方文档：https://www.crummy.com/software/BeautifulSoup/bs4/doc/#

中文文档：https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.zh.html

Beautiful Soup 是一个可以从HTML或XML文本中提取数据的Python库，它能对HTML、XML格式进行解析成树形结构并提取相关信息。

Beautiful Soup库是一个灵活又方便的网页解析库，处理高效，支持多种解析库（后面会介绍），利用它不用编写正则表达式即可方便地实现网页信息的提取。

安装

Beautiful Soup 3 目前已经停止开发，推荐在现在的项目中使用Beautiful Soup 4，安装方法：

pip install beautifulsoup4

 

Beautiful Soup库的4种解析器

解析器 使用方法 优势 劣势
Python标准库 BeautifulSoup(markup, "html.parser") Python的内置标准库、执行速度适中 、文档容错能力强 Python 2.7.3 or 3.2.2)前的版本中文容错能力差
lxml HTML 解析器 BeautifulSoup(markup, "lxml") 速度快、文档容错能力强 需要安装C语言库
lxml XML 解析器 BeautifulSoup(markup, "xml") 速度快、唯一支持XML的解析器 需要安装C语言库
html5lib BeautifulSoup(markup, "html5lib") 最好的容错性、以浏览器的方式解析文档、生成HTML5格式的文档 速度慢、不依赖外部扩展

如果仅是想要解析HTML文档，只要用文档创建 BeautifulSoup 对象就可以了。Beautiful Soup会自动选择一个解析器来解析文档.但是还可以通过参数指定使用那种解析器来解析当前文档。BeautifulSoup 第一个参数应该是要被解析的文档字符串或是文件句柄,第二个参数用来标识怎样解析文档.如果第二个参数为空,那么Beautiful Soup根据当前系统安装的库自动选择解析器,解析器的优先数序: lxml, html5lib, Python标准库(python自带的解析库).

安装解析器库：

pip install html5lib

pip  install lxml

Beautiful Soup类的基本元素

基本使用

容错处理,文档的容错能力指的是在html代码不完整的情况下,使用该模块可以识别该错误。使用BeautifulSoup解析上述代码,能够得到一个 BeautifulSoup 的对象,并能按照标准的缩进格式的结构输出

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.prettify())  #处理好缩进，结构化显示

print(soup.title.string)

<html>

 <head>

  <title>

   The Dormouse's story

  </title>

 </head>

 <body>

  <p class="title" name="dromouse">

   <b>

    The Dormouse's story

   </b>

  </p>

  <p class="story">

   Once upon a time there were three little sisters; and their names were

   <a class="sister" href="http://example.com/elsie" id="link1">

    <!-- Elsie -->

   </a>

   ,

   <a class="sister" href="http://example.com/lacie" id="link2">

    Lacie

   </a>

   and

   <a class="sister" href="http://example.com/tillie" id="link3">

    Tillie

   </a>

   ;

and they lived at the bottom of a well.

  </p>

  <p class="story">

   ...

  </p>

 </body>

</html>

The Dormouse's story

output

标签选择器

选择标签元素（存在多个取第一个）

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.title)

print(type(soup.title))

print(soup.head)

print(soup.p)

<title>The Dormouse's story</title>

<class 'bs4.element.Tag'>

<head><title>The Dormouse's story</title></head>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

output

获取名称

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.title.name)

title

获取属性

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.p.attrs['name'])

print(soup.p['name'])  # 也可以这样写

dromouse

dromouse

获取内容

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p clss="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.p.string)

The Dormouse's story

这里有一个坑：

解决方法就是做判断(需要导入import bs4)：

if type(soup.b.string)==bs4.element.Comment:

    pass

还有一点需要注意：NavigableString可以跨越多个层次

嵌套选择

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<div class="title" name="dromouse"><b class='bb'>The Dormouse's story</b></div>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.div.b['class'])

['bb']

多层嵌套时

html = """<html>

<head>

    <title>Fucking Title</title>

</head>

<body>

    <p class="title"><b><!--This is a fucking comment!-->chantress pheny carbamic picamar.</b></p>

    <p class="course">Lorem ipsum dolor sit amet laborum. <a href="http://www.baidu.com" class="py1" id="link1">Basic Python

        </a> and <a href="http://www.google.com" class="py2" id="link2">Python</a>.

    </p>

</body>

</html>"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(list(soup.p.stripped_strings))

['chantress pheny carbamic picamar.']

节点操作

子节点和子孙节点

对于一个标签的儿子节点不仅包括标签节点，也包括字符串节点，空格表示为'\n'

html = """

<html>

    <head>

        <title>The Dormouse's story</title>

    </head>

    <body>

        <p class="story">

            Once upon a time there were three little sisters; and their names were

            <a href="http://example.com/elsie" class="sister" id="link1">

                <span>Elsie</span>

            </a>

            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>

            and

            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>

        <p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.p.contents)

['\n            Once upon a time there were three little sisters; and their names were\n            ', <a class="sister" href="http://example.com/elsie" id="link1">

<span>Elsie</span>

</a>, '\n', <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, ' \n            and\n            ', <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>, '\n            and they lived at the bottom of a well.\n        ']

output

html = """

<html>

    <head>

        <title>The Dormouse's story</title>

    </head>

    <body>

        <p class="story">

            Once upon a time there were three little sisters; and their names were

            <a href="http://example.com/elsie" class="sister" id="link1">

                <span>Elsie</span>

            </a>

            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>

            and

            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>

        <p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.p.children)

for i, child in enumerate(soup.p.children):

    print(i, child)

<list_iterator object at 0x1064f7dd8>

            Once upon a time there were three little sisters; and their names were

 <a class="sister" href="http://example.com/elsie" id="link1">

<span>Elsie</span>

</a>

 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>

            and

 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

            and they lived at the bottom of a well.

output

html = """

<html>

    <head>

        <title>The Dormouse's story</title>

    </head>

    <body>

        <p class="story">

            Once upon a time there were three little sisters; and their names were

            <a href="http://example.com/elsie" class="sister" id="link1">

                <span>Elsie</span>

            </a>

            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>

            and

            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>

        <p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.p.descendants)

for i, child in enumerate(soup.p.descendants):

    print(i, child)

<generator object descendants at 0x10650e678>

            Once upon a time there were three little sisters; and their names were

 <a class="sister" href="http://example.com/elsie" id="link1">

<span>Elsie</span>

</a>

 <span>Elsie</span>

 Elsie

 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>

 Lacie

            and

 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

 Tillie

            and they lived at the bottom of a well.

output

父节点和祖先节点

html = """

<html>

    <head>

        <title>The Dormouse's story</title>

    </head>

    <body>

        <p class="story">

            Once upon a time there were three little sisters; and their names were

            <a href="http://example.com/elsie" class="sister" id="link1">

                <span>Elsie</span>

            </a>

            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>

            and

            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>

        <p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.a.parent)

<p class="story">

            Once upon a time there were three little sisters; and their names were

            <a class="sister" href="http://example.com/elsie" id="link1">

<span>Elsie</span>

</a>

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>

            and

            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>

output

html = """

<html>

    <head>

        <title>The Dormouse's story</title>

    </head>

    <body>

        <p class="story">

            Once upon a time there were three little sisters; and their names were

            <a href="http://example.com/elsie" class="sister" id="link1">

                <span>Elsie</span>

            </a>

            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>

            and

            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>

        <p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(list(enumerate(soup.a.parents)))

[(, <p class="story">

            Once upon a time there were three little sisters; and their names were

            <a class="sister" href="http://example.com/elsie" id="link1">

<span>Elsie</span>

</a>

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>

            and

            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>), (, <body>

<p class="story">

            Once upon a time there were three little sisters; and their names were

            <a class="sister" href="http://example.com/elsie" id="link1">

<span>Elsie</span>

</a>

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>

            and

            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>

<p class="story">...</p>

</body>), (, <html>

<head>

<title>The Dormouse's story</title>

</head>

<body>

<p class="story">

            Once upon a time there were three little sisters; and their names were

            <a class="sister" href="http://example.com/elsie" id="link1">

<span>Elsie</span>

</a>

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>

            and

            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>

<p class="story">...</p>

</body></html>), (, <html>

<head>

<title>The Dormouse's story</title>

</head>

<body>

<p class="story">

            Once upon a time there were three little sisters; and their names were

            <a class="sister" href="http://example.com/elsie" id="link1">

<span>Elsie</span>

</a>

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>

            and

            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>

<p class="story">...</p>

</body></html>)]

output

兄弟节点

html = """

<html>

    <head>

        <title>The Dormouse's story</title>

    </head>

    <body>

        <p class="story">

            Once upon a time there were three little sisters; and their names were

            <a href="http://example.com/elsie" class="sister" id="link1">

                <span>Elsie</span>

            </a>

            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>

            and

            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>

            and they lived at the bottom of a well.

        </p>

        <p class="story">...</p>

"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(list(enumerate(soup.a.next_siblings)))

print(list(enumerate(soup.a.previous_siblings)))

[(, '\n'), (, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>), (, ' \n            and\n            '), (, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>), (, '\n            and they lived at the bottom of a well.\n        ')]

[(, '\n            Once upon a time there were three little sisters; and their names were\n            ')]

output

标准选择器

基于bs4库的HTML内容查找方法

<>.find_all(name,attrs,recursive,text,**kwargs)  # 返回一个列表类型,存储查找的结果

name      对标签名称的检索字符串

attrs     对标签属性值的检索字符串,可标注属性检索

recursive 是否对子孙全部搜索,默认True

text      对文本内容进行检索

由find_all()扩展的七个方法:

find_all( name , attrs , recursive , text , **kwargs )

可根据标签名、属性、内容查找文档

name

html='''

<div class="panel">

    <div class="panel-heading">

        <h4>Hello</h4>

    </div>

    <div class="panel-body">

        <ul class="list" id="list-1">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

            <li class="element">Jay</li>

        </ul>

        <ul class="list list-small" id="list-2">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

        </ul>

    </div>

</div>

'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.find_all('ul'))

print(type(soup.find_all('ul')[0]))

[<ul class="list" id="list-1">

<li class="element">Foo</li>

<li class="element">Bar</li>

<li class="element">Jay</li>

</ul>, <ul class="list list-small" id="list-2">

<li class="element">Foo</li>

<li class="element">Bar</li>

</ul>]

<class 'bs4.element.Tag'>
html='''

<div class="panel">

    <div class="panel-heading">

        <h4>Hello</h4>

    </div>

    <div class="panel-body">

        <ul class="list" id="list-1">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

            <li class="element">Jay</li>

        </ul>

        <ul class="list list-small" id="list-2">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

        </ul>

    </div>

</div>

'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

for ul in soup.find_all('ul'):

    print(ul.find_all('li'))

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]

[<li class="element">Foo</li>, <li class="element">Bar</li>]

output

attrs

html='''

<div class="panel">

    <div class="panel-heading">

        <h4>Hello</h4>

    </div>

    <div class="panel-body">

        <ul class="list" id="list-1" name="elements">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

            <li class="element">Jay</li>

        </ul>

        <ul class="list list-small" id="list-2">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

        </ul>

    </div>

</div>

'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.find_all(attrs={'id': 'list-1'}))

print(soup.find_all(attrs={'name': 'elements'}))

[<ul class="list" id="list-1" name="elements">

<li class="element">Foo</li>

<li class="element">Bar</li>

<li class="element">Jay</li>

</ul>]

[<ul class="list" id="list-1" name="elements">

<li class="element">Foo</li>

<li class="element">Bar</li>

<li class="element">Jay</li>

</ul>]

output

也可以这样

html='''

<div class="panel">

    <div class="panel-heading">

        <h4>Hello</h4>

    </div>

    <div class="panel-body">

        <ul class="list" id="list-1">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

            <li class="element">Jay</li>

        </ul>

        <ul class="list list-small" id="list-2">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

        </ul>

    </div>

</div>

'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.find_all(id='list-1'))

print(soup.find_all(class_='element'))

[<ul class="list" id="list-1">

<li class="element">Foo</li>

<li class="element">Bar</li>

<li class="element">Jay</li>

</ul>]

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]

output

text

html='''

<div class="panel">

    <div class="panel-heading">

        <h4>Hello</h4>

    </div>

    <div class="panel-body">

        <ul class="list" id="list-1">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

            <li class="element">Jay</li>

        </ul>

        <ul class="list list-small" id="list-2">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

        </ul>

    </div>

</div>

'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.find_all(text='Foo'))

['Foo', 'Foo']

find( name , attrs , recursive , text , **kwargs )

find返回单个元素，find_all返回所有元素

html='''

<div class="panel">

    <div class="panel-heading">

        <h4>Hello</h4>

    </div>

    <div class="panel-body">

        <ul class="list" id="list-1">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

            <li class="element">Jay</li>

        </ul>

        <ul class="list list-small" id="list-2">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

        </ul>

    </div>

</div>

'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.find('ul'))

print(type(soup.find('ul')))

print(soup.find('page'))

<ul class="list" id="list-1">

<li class="element">Foo</li>

<li class="element">Bar</li>

<li class="element">Jay</li>

</ul>

<class 'bs4.element.Tag'>

None

find_parents() find_parent()

find_parents()返回所有祖先节点，find_parent()返回直接父节点。

find_next_siblings() find_next_sibling()

find_next_siblings()返回后面所有兄弟节点，find_next_sibling()返回后面第一个兄弟节点。

find_previous_siblings() find_previous_sibling()

find_previous_siblings()返回前面所有兄弟节点，find_previous_sibling()返回前面第一个兄弟节点。

find_all_next() find_next()

find_all_next()返回节点后所有符合条件的节点, find_next()返回第一个符合条件的节点

find_all_previous() 和 find_previous()

find_all_previous()返回节点后所有符合条件的节点, find_previous()返回第一个符合条件的节点

CSS选择器

通过select()直接传入CSS选择器即可完成选择

html='''

<div class="panel">

    <div class="panel-heading">

        <h4>Hello</h4>

    </div>

    <div class="panel-body">

        <ul class="list" id="list-1">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

            <li class="element">Jay</li>

        </ul>

        <ul class="list list-small" id="list-2">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

        </ul>

    </div>

</div>

'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

print(soup.select('.panel .panel-heading'))

print(soup.select('ul li'))

print(soup.select('#list-2 .element'))

print(type(soup.select('ul')[0]))

[<div class="panel-heading">

<h4>Hello</h4>

</div>]

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]

[<li class="element">Foo</li>, <li class="element">Bar</li>]

<class 'bs4.element.Tag'>

output

html='''

<div class="panel">

    <div class="panel-heading">

        <h4>Hello</h4>

    </div>

    <div class="panel-body">

        <ul class="list" id="list-1">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

            <li class="element">Jay</li>

        </ul>

        <ul class="list list-small" id="list-2">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

        </ul>

    </div>

</div>

'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

for ul in soup.select('ul'):

    print(ul.select('li'))

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]

[<li class="element">Foo</li>, <li class="element">Bar</li>]

output

获取属性

html='''

<div class="panel">

    <div class="panel-heading">

        <h4>Hello</h4>

    </div>

    <div class="panel-body">

        <ul class="list" id="list-1">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

            <li class="element">Jay</li>

        </ul>

        <ul class="list list-small" id="list-2">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

        </ul>

    </div>

</div>

'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

for ul in soup.select('ul'):

    print(ul['id'])

    print(ul.attrs['id'])

list-

list-

list-

list-

获取内容

html='''

<div class="panel">

    <div class="panel-heading">

        <h4>Hello</h4>

    </div>

    <div class="panel-body">

        <ul class="list" id="list-1">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

            <li class="element">Jay</li>

        </ul>

        <ul class="list list-small" id="list-2">

            <li class="element">Foo</li>

            <li class="element">Bar</li>

        </ul>

    </div>

</div>

'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'lxml')

for li in soup.select('li'):

    print(li.get_text())

Foo

Bar

Jay

Foo

Bar

总结

  • 推荐使用lxml解析库，必要时使用html.parser
  • 标签选择筛选功能弱但是速度快
  • 建议使用find()、find_all() 查询匹配单个结果或者多个结果
  • 如果对CSS选择器熟悉建议使用select()

实例：中国大学排名爬虫

步骤1:从网络上获取大学排名网页内容getHTMLText()
步骤2:提取网页内容中信息到合适的数据结构fillUnivList()
步骤3:利用数据结构展示并输出结果printUnivLise()

import requests

from bs4 import BeautifulSoup

import bs4

def getHTMLText(url):

    try:

        r = requests.get(url, timeout=30)

        r.raise_for_status()

        r.encoding = r.apparent_encoding

        return r.text

    except:

        return "error"

def fillUnivList(ulist, html):

    soup = BeautifulSoup(html, "html.parser")

    for tr in soup.find('tbody').children:

        if isinstance(tr, bs4.element.Tag):  # 过滤掉非标签类型

            tds = tr('td')

            ulist.append([tds[0].string, tds[1].string, tds[3].string])

# 中文对齐问题的解决：

# 采用中文字符的空格填充 chr(12288)

def printUnivList(ulist, num):

    tplt = "{0:^10}\t{1:{3}^10}\t{2:^10}"

    print(tplt.format("排名", "学校名称", "总分", chr(12288)))

    for i in range(num):

        u = ulist[i]

        print(tplt.format(u[0], u[1], u[2], chr(12288)))

def main():

    uinfo = []

    url = 'http://www.zuihaodaxue.cn/zuihaodaxuepaiming2016.html'

    html = getHTMLText(url)

    fillUnivList(uinfo, html)

    printUnivList(uinfo, 20)

if __name__ == '__main__':

    main()

　　

　　

Beautiful Soup库基础用法（爬虫）的更多相关文章

  1. Python Beautiful Soup库

    Beautiful Soup库 Beautiful Soup库:https://www.crummy.com/software/BeautifulSoup/ 安装Beautiful Soup: 使用B ...

  2. 【转载】Beautiful Soup库（bs4）入门

    转载自:Beautiful Soup库(bs4)入门 该库能够解析HTML和XML 使用Beautiful Soup库:      from bs4 import BeautifulSoup impo ...

  3. Beautiful Soup库入门

    1.安装:pip install beautifulsoup4 Beautiful Soup库是解析.遍历.维护“标签树”的功能库 2.引用:(1)from bs4 import BeautifulS ...

  4. python beautiful soup库的超详细用法

    原文地址https://blog.csdn.net/love666666shen/article/details/77512353 参考文章https://cuiqingcai.com/1319.ht ...

  5. 【Python爬虫学习笔记（3）】Beautiful Soup库相关知识点总结

    1. Beautiful Soup简介     Beautiful Soup是将数据从HTML和XML文件中解析出来的一个python库,它能够提供一种符合习惯的方法去遍历搜索和修改解析树,这将大大减 ...

  6. Beautiful Soup库

    原文传送门:静觅 » Python爬虫利器二之Beautiful Soup的用法

  7. python之Beautiful Soup库

    1.简介 简单来说,Beautiful Soup是python的一个库,最主要的功能是从网页抓取数据.官方解释如下: Beautiful Soup提供一些简单的.python式的函数用来处理导航.搜索 ...

  8. Beautiful Soup库介绍

    开始前需安装Beautiful Soup 和lxml. Beautiful Soup在解析时依赖解析器,下表列出bs4支持的解析器. 解析器 使用方法 Python标准库 BeautifulSoup( ...

  9. crawler碎碎念4 关于python requests、Beautiful Soup库、SQLlite的基本操作

    Requests import requests from PIL import Image from io improt BytesTO import jason url = "..... ...

随机推荐

  1. 黏包-黏包的成因、解决方式及struct模块初识、文件的上传和下载

    黏包: 同时执行多条命令之后,得到的结果很可能只有一部分,在执行其他命令的时候又接收到之前执行的另外一部分结果,这种显现就是黏包. 只有TCP协议中才会产生黏包,UDP协议中不会有黏包(udp协议中数 ...

  2. Mac系统在Pycharm中切换解释器

    1. 2. 3. 4. 5.

  3. python笔记17-全局变量、局部变量

    在函数里面定义变量叫局部变量,它只能在函数里面用出了该函数外,就不能使用了在函数外面定义的变量,是全局变量,在函数内也可以使用 如果想在函数里面修改全局变量的值,那么要先用global关键字声明 要修 ...

  4. NSIS脚本 打包安装程序

    相关工具 nsis http://nsis.sourceforge.net/Special_Builds HM NIS Edit http://hmne.sourceforge.net/ 例子: ; ...

  5. tp5阿里云短信发送

    到阿里云下载php版demo,下完整版的,不是轻量级的; 框架  :TP5 把下载下来的文件放到extend里面 文件名:alimsg 里面的文件 import('alimsg.api_demo.Sm ...

  6. shell中括号的特殊用法 linux if多条件判断

    一．   bash [  ] 单双括号 基本要素: Ø  [ ] 两个符号左右都要有空格分隔 Ø  内部操作符与操作变量之间要有空格:如  [  “a”  =  “b”  ] Ø  字符串比较中,&g ...

  7. 使用Jackson解析首字母大写的json字符串

    Jackson在解析返回的json字符串时始终报错,纠结很久之后才找到原因,原来是是由于json字符串中的字母都是首字母大写,导致jackson找不到相应的KEY. 在项目中经常使用从服务器获取的数据 ...

  8. IDEA控制台乱码解决

    打开Intellij的安装的bin目录(D:\Program Files\JetBrains\IntelliJ IDEA 14.0\bin ),找到上图的两个文件(根据你的系统是32位或64位选择其中 ...

  9. php之强制回调类型callable

    <?php function demo(callable $fn) { $fn(); } function callback() { echo __FUNCTION__,'<br/> ...

  10. 关于WebAPI跨域踩到的一点坑

    最近在尝试前后端分离的WebAPI+AngularJS方案,在率先处理授权的时候,踩到了一点WebAPI跨域的坑,其实严格意义上来说也不算是坑吧,只是我自己对WebAPI不熟悉而已,这里我与大家分享一 ...