python利用beautifulSoup写爬虫

python BeautifulSoup模块的安装

安装包下载地址：http://www.crummy.com/software/BeautifulSoup/#Download

文档：http://www.crummy.com/software/BeautifulSoup/documentation.html

下载后解压， 然后进入目录执行 ：

python setup.py build
python setup.py install

引入包要用

 import bs4
 from bs4 import BeautifulSoup

利用BeautifulSoup抓取网页内容

 # coding=utf-8
 from bs4 import BeautifulSoup
 import urllib
 import re

 url ='http://www.baidu.com/s'
 values ={'wd':u'渗透'}
 encoded_param = urllib.urlencode(values)
 full_url = url +'?'+ encoded_param
 response = urllib.urlopen(full_url)
 soup =BeautifulSoup(response)
 alinks = soup.find_all('a', href=re.compile('^http|^/'))