python scrapy简单爬虫记录(实现简单爬取知乎)

之前写了个scrapy的学习记录，只是简单的介绍了下scrapy的一些内容，并没有实际的例子，现在开始记录例子

使用的环境是python2.7， scrapy1.2.0

首先创建项目

在要建立项目的目录下执行命令scrapy startproject tutorial

scrapy会帮你建立好项目，接着创建蜘蛛scrapy genspider　　zhuhuSpider zhihu.com

当前的文件结构是

--tutorial

　　--spiders

　　　　--__init__.py

　　　　--zhihuSpider.py

　　--__init__.py

　　--items.py

　　--pipelines.py

　　--settings.py

主要用到的文件有三个zhihuspider.py,items.py,settings.py

zhihuspider是用来写爬行的蜘蛛的，items用来保存爬取的数据，settings修改配置

# -*- coding: utf-8 -*-
from scrapy.spiders import Spider
from scrapy.selector import Selector
from tutorial.items import ZhihuItem

#蜘蛛的主体
class ZhihuSpider(Spider):
    name = "zhihuSpider"
    allowed_domains = ['zhihu.com']
    start_urls = []

　　#获取要爬取的url，我把关键词放在zhihu.txt中
    def start_requests(self):
        url_head = 'https://www.zhihu.com/search?type=content&q='
        with open('zhihu.txt', 'r') as f:
            datas = f.readlines()
        for data in datas:
            url = url_head+data
            print url
            self.start_urls.append(url)

        for url in self.start_urls:
            yield self.make_requests_from_url(url)

　　#对返回的response进行解析，该步骤可以配合浏览器进行xpath的查找
    def parse(self, response):
　　　　#删除所有的<em>
        response = response.replace(body=response.body.replace('<em>', ''))
        hxs = Selector(response)

        contents = hxs.xpath('//*[@class="zu-main-content"]//*[contains(@class, "list")]')
        item = ZhihuItem()
        for content in contents:
            item['search_title'] = content.xpath('//*[@class="title"]/a/text()').extract()
            item['search_title_link'] = content.xpath('//*[@class="title"]/a/@href').extract()
            item['search_answer'] = content.xpath('//*[@class="content"]//*[contains(@class, "entry-content")]//*[contains(@class, "summary")]/text()').extract()
            item['search_answer_link'] = content.xpath('//*[@class="content"]//*[contains(@class, "entry-content")]//*[contains(@class, "summary")]/a/@href').extract()
            item['search_answer_writer'] = content.xpath('//*[@class="content"]//*[contains(@class, "entry-meta")]//a[contains(@class, "author")]/text()').extract()
            print item
            yield item

from scrapy.item import Item, Field

class ZhihuItem(Item):
    search_title = Field()
    search_title_link = Field()
    search_answer = Field()
    search_answer_link = Field()
    search_answer_writer = Field()

该items文件存放数据

因为知乎对爬虫有限制，所以需要加入反爬虫机制

基本上有4种，添加useragent，添加代理，禁用cookie以及爬取时间限制

在当前项目下添加python包middlewares，和setttings在同一个目录下，下面只有一个__init__.py文件，新建一个RandomUserAgent.py文件

#coding:utf-8
import random

class RandomUserAgent(object):

    def __init__(self, agents):
        self.agents = agents

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist('USER_AGENTS'))

    def process_request(self, request, spider):
        request.headers.setdefault('User-Agent', random.choice(self.agents))

简单的加入user-agent

最后在settings中设置

DOWNLOADER_MIDDLEWARES = {
   # 'tutorial.middlewares.MyCustomDownloaderMiddleware': 543,
    'tutorial.middlewares.RandomUserAgent.RandomUserAgent': 1,
}
USER_AGENTS = [

    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",

    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",

    "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",

    "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",

    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",

    "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",

    "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",

    "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",

    "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",

    "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",

    "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",

    "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",

    "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",

    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",

    "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER",

    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)",

    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 LBBROWSER",

    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",

    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)",

    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",

    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)",

    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",

    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",

    "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",

    "Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5",

    "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110307 Firefox/4.0b13pre",

    "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11",

    "Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10"

]
COOKIES_ENABLED=False
DOWNLOAD_DELAY=3

整个项目完成了，scrapy crawl zhihuSpider -o zhihu.json导出数据。就是这么简单

如果想要不喜欢用-o来导出数据，可以让scrapy自己保存数据，编辑pipelines

import json
import codecs

class xxxPipeline(object):
    def __init__(self):
        self.file = codecs.open('xxx.csv', 'w', encoding='utf-8')

    def process_item(self, item, spider):
        datas = json.dumps(dict(item), ensure_ascii=False) + '\n'
        self.file.write(datas)
        return item

    def spider_closed(self, spider):
        self.file.close()

然后在setting中启用

ITEM_PIPELINES = {
   'tutorial.pipelines.xxxPipeline': 300, 
}

把对应的pipeline修改成自己的，最后那个数字表示启动顺序，越低顺序越快。

最后说一句，爬取数据时要注意对爬取时间的设置，别给服务器造成负担

项目已经提交到github上地址是https://github.com/lin344902118/doubanSpider.git

里面还有个对豆瓣的爬虫