8.scrapy的第一个实例

【目标】要完成的任务如下：

※ 创建一个 Scrap项目。
※ 创建一个 Spider来抓取站点和处理数据。
※ 通过命令行将抓取的内容导出。
※ 将抓取的内容保存的到 MongoDB数据库。
==============================================

【准备工作】需要安装好 Scrapy框架、 MongoDB和 PyMongo库

1.创建项目：

【操作】在想创建项目的目录按：shift+右键——在此处打开命令窗口（或 在cmd里cd进入想要的目录）输入CMD命令（此处不用test，因为系统已经有一个此名字，将报错提示已存在）：

注意：在linux 系统内如提示权限问题，则在命令前加 sudo ~

scrapy startproject tutorial   # 生成项目命令：最后跟的参数tutorial是项目名，也将是文件夹名

【说明】生成的目录结构如下：

tutorial        # 项目总文件夹
　　tutorial    # tutorial文件夹：保存的是要引用的模块；
　　crapy.cfg   # cfg文件：是scrapy部署时的配置文件；

【说明】../tutorial/tutorial 目录下文件及作用：

tutorial
 __init__.py
  items.py          # Items的定义,定义爬取的数据结构
  middlewares.py    # Middlewares的定义,定义爬取时的中间件
  pipelines. py     # Pipelines的定义,定义数据管道
  settings.py       # 配置文件
spiders            # 放置 Spiders的文件夹
  __init__.py

2.创建 Spider

【操作】Spider是自己定义的类, Scrapy用它来从网页里抓取内容,并解析抓取的结果。不过这个类必须继承 Scrapy提供的 Spider类 scrapy. Spider,还要定义Spider的名称和起始请求,以及怎样处理爬取后的结果的方法。也可以使用命令行创建一个 Spider。比如要生成 quotes这个 Spider,可以执行如下cmd命令

【命令解释】进入刚才创建的tutorial文件夹，然后执行 genspider命令：第一个参数是 Spider的名称，第二个参数是网站域名：

cd tutorial
scrapy genspider quotes quotes.toscrape.com

【说明】执行完毕之后, spiders文件夹中多了一个 quotes. py（目录：E:\a\scrapy_test\tutorial\tutorial\spiders）,它就是刚刚创建的 Spider,内容如下(其网址，名仅作表示，真代码见4步)：

# -*- coding: utf-8 -*-
import scrapy
 
class QuotesSpider(scrapy.Spider):
    name = 'baidu'
    allowed_domains = ['fanyi.baidu.com\']
    start_urls = ['http://fanyi.baidu.com\']
 
    def parse(self, response):
        pass

【说明】这里有三个属性—name、 allowed domains和 start urls,还有一个方法 parse：

★name：它是每个项目唯一的名字,用来区分不同的 Spider

★allowed domains：它是允许爬取的域名,如果初始或后续的请求链接不是这个域名下的,则请求链接会被过滤掉。

★start urls：它包含了 Spider在启动时爬取的url列表,初始请求是由它来定义的。

★parse：它是 Spider的一个方法。默认情况下,被调用时 start urls里面的链接构成的请求完成下载执行后,返回的响应就会作为唯一的参数传递给这个函数。该方法负责解析返回的响应、提取数据或者进一步生成要处理的请求。

3.创建ltem

【说明】Item是保存爬取数据的容器,它的使用方法和字典类似。不过,相比字典,ltem多了额外的保护机制,可以避免拼写错误或者定义字段错误。
创建Iem需要继承 scrapy.Item类,并且定义类型为 scrapy. Field的字段。

【操作】定义Item,观察目标网站,我们可以获取到到内容有text、author、tags。此时将 Items.py修改如下:

import scrapy
class QuoteItem(scrapy.Item)
text= scrapy. Field()
author= scrapy. Field()
tags= scrapy. Field()
#这里定义了三个字段,接下来爬取时我们会使用到这个Item。

4.解析response

【说明】quotes.py中我们看到, parse()方法的参数 respose是 start urls里面的链接爬取后的结果。所以在parse()方法中,我们可以直接对 response变量包含的内容进行解析,比如浏览请求结果的网页源代码,或者进一步分析源代码内容,或者找出结果中的链接而得到下一个请求。

【网页分析】我们可以看到网页中既有我们想要的结果,又有下一页的链接,这两部分内容我们都要进行处理。首先看看网页结构,如图13-2所示。每一页都有多个class为 quote的区块,每个区块内都包含text、 author、tags。那么我们先找出所有的 quote,然后提取每一个 quote中的内容。

quotes.py写成如下：

# -*- coding: utf-8 -*-
import scrapy
 
class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/']
 
    def parse(self, response):
        quotes=response.css('.quote')
        for quote in quotes:
            text=quote.css('.text::text').extract_first()
            author=quote.css('.author::text').extract_first()
            tags=quote.css('.tags .tag::text').extract()

　　　　　　　　#或<meta class="keywords" itemprop="keywords" content="change,deep-thoughts,thinking,world">
　　　　　　　　tags=quote.css('.keywords::attr(content)').extract_first()

5.使用Item

【说明】上文定义了Item,接下来就要使用它了。Item可以理解为一个字典,不过在声明的时候需要实例化。然后依次用刚才解析的结果赋值Item的每一个字段,最后将Item返回即可。
Quotes Spider的改写quotes.py如下所示:

# -*- coding: utf-8 -*-
import scrapy
from tutorial.items import QuoteItem
 
class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ['http://quotes.toscrape.com/']
 
    def parse(self, response):
        quotes = response.css('.quote')
        for quote in quotes:
            item = QuoteItem()
            item['text'] = quote.css('.text::text').extract_first()
            item['author'] = quote.css('.author::text').extract_first()
            item['tags'] = quote.css('.tags .tag::text').extract()
            yield item

【代码解释】如此，所有首页内容被解析出来，赋值成一个个QuotesItem

6.后续Request（生成下一页请求）

【说明】抓取下一页内容：从当前页找到一个信息，生成下一页请求，从而往复迭代，抓取整站

【分析】找到下一页面按钮分析源码：

【方法说明】构造请求需用到：scrapy.Request(url=url, callback=self.parse)

参数1：url是请求的链接；

参数2：callback是回调函数。当指定了该回调函数的请求完成之后,获取到响应,引擎会将该响应作为参数传递给这个回调函数。回调函数进行解析或生成下一个请求,回调函数即刚写的 parse()函数。

由于 parse()就是解析text、 author、tags的方法,而下一页的结构和刚才已经解析的页面结构是一样的,所以我们可以再次使用 parse()方法来做页面解析。

【操作】写的下一页代码如下：

        next = response.css('.pager .next a::attr(href)').extract_first()
        url = response.urljoin(next)
        yield scrapy.Request(url=url, callback=self.parse)

【源码说明】：

第1行首先通过CSS选择器获取下一个页面的链接,即要获取a超链接中的href属性。这里用到了:attr(href)操作。然后再调用 extract first()方法获取内容。
第2行调用了 urljoin()方法, urljoin()方法可以将相对URL构造成一个绝对的URL。例如,获取到的下一页地址是page2, unjoint()方法处理后得到的结果就是:htp/ quotes.toscrape. com/page/2/
第3行通过ur1和 callback变量构造了一个新的请求,回调函数 callback依然使用 parse()方法。这个请求完成后,响应会重新经过 parse方法处理,得到第2页的解析结果,然后生成第2页的下一页,即第3页的请求

这样爬虫就进入了一个循环,直到最后一页。通过几行代码,我们就轻松实现了一个抓取循环,将每个页面的结果抓取下来了

【quotes.py 最终代码】如下：

# -*- coding: utf-8 -*-
import scrapy
 
from tutorial.items import QuoteItem
 
class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ['http://quotes.toscrape.com/']
 
    def parse(self, response):
        quotes = response.css('.quote')
        for quote in quotes:
            item = QuoteItem()
            item['text'] = quote.css('.text::text').extract_first()
            item['author'] = quote.css('.author::text').extract_first()
            item['tags'] = quote.css('.tags .tag::text').extract()
            yield item
 
        next = response.css('.pager .next a::attr(href)').extract_first()
        url = response.urljoin(next)
        yield scrapy.Request(url=url, callback=self.parse)

7.运行爬虫

【运行前】记得把 settings.py中的，顺从爬虫规则去年注释，并改成False，否则，默认为Ture，可能会过滤掉许多网址

# Obey robots.txt rules，默认为Ture，即按网站规则来，把有些网址过滤掉
ROBOTSTXT_OBEY = False

【运行爬虫】进入项目目录（E:\a\scrapy_test\tutorial），运行命令即可运行爬虫，进行爬取：

scrapy crawl quotes

【结果】如下：

scrapy crawl quotes
 
'tags': ['friendship', 'love'],
 'text': '“There is nothing I would not do for those who are...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/6/>
{'author': 'Eleanor Roosevelt',
 'tags': ['attributed', 'fear', 'inspiration'],
 'text': '“Do one thing every day that scares you.”'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/6/>
{'author': 'Marilyn Monroe',
 'tags': ['attributed-no-source'],
 'text': '“I am good, but not an angel. I do sin, but I am n...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/6/>
{'author': 'Albert Einstein',
 'tags': ['music'],
 'text': '“If I were not a physicist, I would probably be a...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/6/>
{'author': 'Haruki Murakami',
 'tags': ['books', 'thought'],
 'text': '“If you only read the books that everyone else is...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/6/>
{'author': 'Alexandre Dumas fils',
 'tags': ['misattributed-to-einstein'],
 'text': '“The difference between genius and stupidity is: g...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/6/>
{'author': 'Stephenie Meyer',
 'tags': ['drug', 'romance', 'simile'],
 'text': "“He's like a drug for you, Bella.”"}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/6/>
{'author': 'Ernest Hemingway',
 'tags': ['books', 'friends', 'novelist-quotes'],
 'text': '“There is no friend as loyal as a book.”'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/6/>
{'author': 'Helen Keller',
 'tags': ['inspirational'],
 'text': '“When one door of happiness closes, another opens;...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/6/>
{'author': 'George Bernard Shaw',
 'tags': ['inspirational', 'life', 'yourself'],
 'text': "“Life isn't about finding yourself. Life is about..."}
2019-04-26 13:53:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes
.toscrape.com/page/7/> (referer: http://quotes.toscrape.com/page/6/)
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/7/>
{'author': 'Charles Bukowski',
 'tags': ['alcohol'],
 'text': "“That's the problem with drinking, I thought, as I..."}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/7/>
{'author': 'Suzanne Collins',
 'tags': ['the-hunger-games'],
 'text': '“You don’t forget the face of the person who was y...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/7/>
{'author': 'Suzanne Collins',
 'tags': ['humor'],
 'text': "“Remember, we're madly in love, so it's all right..."}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/7/>
{'author': 'C.S. Lewis',
 'tags': ['love'],
 'text': '“To love at all is to be vulnerable. Love anything...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/7/>
{'author': 'J.R.R. Tolkien',
 'tags': ['bilbo', 'journey', 'lost', 'quest', 'travel', 'wander'],
 'text': '“Not all those who wander are lost.”'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/7/>
{'author': 'J.K. Rowling',
 'tags': ['live-death-love'],
 'text': '“Do not pity the dead, Harry. Pity the living, and...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/7/>
{'author': 'Ernest Hemingway',
 'tags': ['good', 'writing'],
 'text': '“There is nothing to writing. All you do is sit do...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/7/>
{'author': 'Ralph Waldo Emerson',
 'tags': ['life', 'regrets'],
 'text': '“Finish each day and be done with it. You have don...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/7/>
{'author': 'Mark Twain',
 'tags': ['education'],
 'text': '“I have never let my schooling interfere with my e...'}
2019-04-26 13:53:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/7/>
{'author': 'Dr. Seuss',
 'tags': ['troubles'],
 'text': '“I have heard there are troubles of more than one...'}
2019-04-26 13:53:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes
.toscrape.com/page/8/> (referer: http://quotes.toscrape.com/page/7/)
2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/8/>
{'author': 'Alfred Tennyson',
 'tags': ['friendship', 'love'],
 'text': '“If I had a flower for every time I thought of you...'}
2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/8/>
{'author': 'Charles Bukowski',
 'tags': ['humor'],
 'text': '“Some people never go crazy. What truly horrible l...'}
2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/8/>
{'author': 'Terry Pratchett',
 'tags': ['humor', 'open-mind', 'thinking'],
 'text': '“The trouble with having an open mind, of course,...'}
2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/8/>
{'author': 'Dr. Seuss',
 'tags': ['humor', 'philosophy'],
 'text': '“Think left and think right and think low and thin...'}
2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/8/>
{'author': 'J.D. Salinger',
 'tags': ['authors', 'books', 'literature', 'reading', 'writing'],
 'text': '“What really knocks me out is a book that, when yo...'}
2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/8/>
{'author': 'George Carlin',
 'tags': ['humor', 'insanity', 'lies', 'lying', 'self-indulgence', 'truth'],
 'text': '“The reason I talk to myself is because I’m the on...'}
2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/8/>
{'author': 'John Lennon',
 'tags': ['beatles',
          'connection',
          'dreamers',
          'dreaming',
          'dreams',
          'hope',
          'inspirational',
          'peace'],
 'text': "“You may say I'm a dreamer, but I'm not the only o..."}
2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/8/>
{'author': 'W.C. Fields',
 'tags': ['humor', 'sinister'],
 'text': '“I am free of all prejudice. I hate everyone equal...'}
2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/8/>
{'author': 'Ayn Rand',
 'tags': [],
 'text': "“The question isn't who is going to let me; it's w..."}
2019-04-26 13:53:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/8/>
{'author': 'Mark Twain',
 'tags': ['books', 'classic', 'reading'],
 'text': "“′Classic′ - a book which people praise and don't..."}
2019-04-26 13:53:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes
.toscrape.com/page/9/> (referer: http://quotes.toscrape.com/page/8/)
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/9/>
{'author': 'Albert Einstein',
 'tags': ['mistakes'],
 'text': '“Anyone who has never made a mistake has never tri...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/9/>
{'author': 'Jane Austen',
 'tags': ['humor', 'love', 'romantic', 'women'],
 'text': "“A lady's imagination is very rapid; it jumps from..."}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/9/>
{'author': 'J.K. Rowling',
 'tags': ['integrity'],
 'text': '“Remember, if the time should come when you have t...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/9/>
{'author': 'Jane Austen',
 'tags': ['books', 'library', 'reading'],
 'text': '“I declare after all there is no enjoyment like re...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/9/>
{'author': 'Jane Austen',
 'tags': ['elizabeth-bennet', 'jane-austen'],
 'text': '“There are few people whom I really love, and stil...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/9/>
{'author': 'C.S. Lewis',
 'tags': ['age', 'fairytales', 'growing-up'],
 'text': '“Some day you will be old enough to start reading...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/9/>
{'author': 'C.S. Lewis',
 'tags': ['god'],
 'text': '“We are not necessarily doubting that God will do...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/9/>
{'author': 'Mark Twain',
 'tags': ['death', 'life'],
 'text': '“The fear of death follows from the fear of life....'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/9/>
{'author': 'Mark Twain',
 'tags': ['misattributed-mark-twain', 'truth'],
 'text': '“A lie can travel half way around the world while...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/9/>
{'author': 'C.S. Lewis',
 'tags': ['christianity', 'faith', 'religion', 'sun'],
 'text': '“I believe in Christianity as I believe that the s...'}
2019-04-26 13:53:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes
.toscrape.com/page/10/> (referer: http://quotes.toscrape.com/page/9/)
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/10/>
{'author': 'J.K. Rowling',
 'tags': ['truth'],
 'text': '“The truth." Dumbledore sighed. "It is a beautiful...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/10/>
{'author': 'Jimi Hendrix',
 'tags': ['death', 'life'],
 'text': "“I'm the one that's got to die when it's time for..."}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/10/>
{'author': 'J.M. Barrie',
 'tags': ['adventure', 'love'],
 'text': '“To die will be an awfully big adventure.”'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/10/>
{'author': 'E.E. Cummings',
 'tags': ['courage'],
 'text': '“It takes courage to grow up and become who you re...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/10/>
{'author': 'Khaled Hosseini',
 'tags': ['life'],
 'text': '“But better to get hurt by the truth than comforte...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/10/>
{'author': 'Harper Lee',
 'tags': ['better-life-empathy'],
 'text': '“You never really understand a person until you co...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/10/>
{'author': "Madeleine L'Engle",
 'tags': ['books',
          'children',
          'difficult',
          'grown-ups',
          'write',
          'writers',
          'writing'],
 'text': '“You have to write the book that wants to be writt...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/10/>
{'author': 'Mark Twain',
 'tags': ['truth'],
 'text': '“Never tell the truth to people who are not worthy...'}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/10/>
{'author': 'Dr. Seuss',
 'tags': ['inspirational'],
 'text': "“A person's a person, no matter how small.”"}
2019-04-26 13:53:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes
.toscrape.com/page/10/>
{'author': 'George R.R. Martin',
 'tags': ['books', 'mind'],
 'text': '“... a mind needs books as a sword needs a whetsto...'}
2019-04-26 13:53:52 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET
 http://quotes.toscrape.com/page/10/> - no more duplicates will be shown (see DU
PEFILTER_DEBUG to show all duplicates)
2019-04-26 13:53:52 [scrapy.core.engine] INFO: Closing spider (finished)
2019-04-26 13:53:52 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 2870,
 'downloader/request_count': 11,
 'downloader/request_method_count/GET': 11,
 'downloader/response_bytes': 24812,
 'downloader/response_count': 11,
 'downloader/response_status_count/200': 10,
 'downloader/response_status_count/404': 1,
 'dupefilter/filtered': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 4, 26, 5, 53, 52, 939800),
 'item_scraped_count': 100,
 'log_count/DEBUG': 112,
 'log_count/INFO': 9,
 'request_depth_max': 10,
 'response_received_count': 11,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/404': 1,
 'scheduler/dequeued': 10,
 'scheduler/dequeued/memory': 10,
 'scheduler/enqueued': 10,
 'scheduler/enqueued/memory': 10,
 'start_time': datetime.datetime(2019, 4, 26, 5, 53, 38, 767800)}
2019-04-26 13:53:52 [scrapy.core.engine] INFO: Spider closed (finished)
 
C:\Users\Administrator\Desktop\py\scrapytutorial>

【解释】结果：

a) 首先Scrap输出了当前的版本号以及正在启动的项目名称。
b) 接着输出了当前 settings. py中一些重写后的配置。
c) 然后输出了当前所应用的 Middlewares和 Pipelines Middlewares默认是启用的,可以在 settings.py中修改。
d) Pipelines默认是空,同样也可以在 settings. py中配置。后面会对它们进行讲解。
e) 接下来就是输出各个页面的抓取结果了,可以看到爬虫一边解析,一边翻页,直至将所有内容抓取完毕,然后终止。
f) 最后, Scrap输出了整个抓取过程的统计信息,如请求的字节数、请求次数、响应次数、完成原因等。

整个 Scrap程序成功运行。我们通过非常简单的代码就完成了一个网站内容的爬取,这样相比之前一点点写程序简洁很多。

8.1保存到文件json csv xml pickle marshal

【说明】运行完 Scrap后,我们只在控制台看到了输出结果。如果想保存结果可直接用, Scrap提供的【 Feed Exports】轻松将抓取结果输出。

【操作a】 将上面的结果【保存成JSON文件】,可以执行如下命令：

#CMD命令，完成后，目录下多出一个quotes.json文件即结果
scrapy crawl quotes -o quotes.json

【操作b】还可以【每一个Item输出一行】JSON,输出后缀为jl,为 jsonline的缩写,命令如下所示:

#cmd命令：每个item导出为一行的joson文件。写法1
scrapy crawl quotes -o quotes.jl
 
#-----或-------：
 
#cmd命令：每个item导出为一行的joson文件.写法2
scrapy crawl quotes -o quotes.jsonlines

【操作c】其它格式输出：

1.输出格式还支持很多种,例如csv、xml、 pickle、 marshal等,还支持仰、s3等远程输出；
2.另外还可以通过自定义 ItemExporter来实现其他的输出。
【操作】下面命令对应的输出分别为【csv、xml、 pickle、 marshal格式、ftp远程输出】：

#cmd命令：其它格式输出
scrapy crawl quotes -o quotes.csv
scrapy crawl quotes -o quotes.xml
scrapy crawl quotes -o quotes.pickle
scrapy crawl quotes -o quotes.marshal
 
#ftp输出需要正确配置用户名、密码、地址、输岀路径,否则会报错。其它ftp文件格式参考上一段
scrapy crawl quotes -o ftp://user:pass@ftp.example.com/path/to/quotes.csv

8.2其它复杂格式输出（用Pipline.py 输出到数据库）

【引言】通过 Scrap提供的 Feed Exports,我们可以轻松地输出抓取结果到文件。对于一些小型项目来说,这应该足够了。如想要更复杂的输出,如输出到数据库等,则可用 Item Pipeline来完成

【目标】如果想进行更复杂的操作,如【将结果保存到 MongoDB数据库】,或者筛选某些有用的Iem,则我们可以定义 Item Pipeline来实现

【操作说明】Item Pipeline为项目管道。当Item生成后,它会自动被送到 Item Pipeline进行处理,我们【常用ltem Pipeline来做如下操作】：
§ 清理HTML数据。
§ 验证爬取数据,检查爬取字段。
§ 查重并丢弃重复内容。
§ 将爬取结果保存到数据库。

【用到的方法】很简单,只需要定义一个类并实现 process_item()方法即可。启用 Item Pipeline后, Item Pipeline会自动调用这个方法：

process_item()方法必须返回包含数据的字典或Item对象 或 者抛出 DropItem异常。

process_item()方法有两个参数。一个参数是item,每次 Spider生成的Item都会作为参数传递过来。

另一个参数是 spider,就是 Spider的实例。

【实操】接下来,我们实现一个 Item Pipeline,筛掉text长度大于50的Item,并将结果保存到 MongoDB：

【步骤1】修改项目里的  [ pipelines.py ] 文件,之前用命令行自动生成的文件内容可以删掉,增加一个TextPipeline类,内容如下所示：

import pymongo
from scrapy.exceptions import DropItem
 
#新写部分
class TextPipeline(object):
    def __init__(self):
        self.limit = 50
 
    def process_item(self, item, spider):
        if item['text']:
            if len(item['text']) > self.limit:
                item['text'] = item['text'][0:self.limit].rstrip() + '...'
            return item
        else:
            return DropItem('Missing Text')

【代码说明】：这段代码在构造方法里定义了限制长度为50,实现了 process item()方法,其参数是item和spider。首先该方法判断item的text属性是否存在,如果不存在,则抛出 DropItem异常;如果存在,再判断长度是否大于50,如果大于,那就截断然后拼接省略号,再将item返回即可。

【步骤2】接下来,我们将处理后的【item存入 MongoDB】,定义另外一个 Pipeline。同样在 pipelines. py中,我们实现另一个类 MongoPipeline,内容如下所示:

class MongoPipeline(object):
    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db
 
    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get('MONGO_URI'),
            mongo_db=crawler.settings.get('MONGO_DB')
        )
 
    def open_spider(self, spider):
        self.client = pymongo.MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]
 
    def process_item(self, item, spider):
        name = item.__class__.__name__
        self.db[name].insert(dict(item))
        return item
 
    def close_spider(self, spider):
        self.client.close()

【步骤3】MongoPipeline类实现了API定义的另外几个方法。

§ from crawler。它是一个类方法,用@ classmethod标识,是一种依赖注入的方式。它的参数就是 crawler,通过 crawler我们可以拿到全局配的每个配置信息。在全局配置 settings.py中,我们可以定义 MONGO URI和MoN0DB来指定 MongoDB连接需要的地址和数据库名称,拿到配置信息之后返回类对象即可。所以这个方法的定义主要是用来获取 settings. py中的配置的。
§ open spider。当 Spider开启时,这个方法被调用。上文程序中主要进行了一些初始化操作
§ close spider。当 Spider关闭时,这个方法会调用。上文程序中将数据库连接关闭。

最主要的 process item()方法则执行了数据插入操作。定义好 TextPipeline和 MongoPipeline这两个类后,我们需要在 [ settings.py ] 中使用它们 MongoDB的连接信息还需要定义，在其中加入如下信息：

ITEM_PIPELINES = {
   'tutorial.pipelines.TextPipeline': 300,
   'tutorial.pipelines.MongoPipeline': 400,
}
MONGO_URI='localhost'
MONGO_DB='tutorial'

【代码说明】赋值 ITEM PIPELINES字典,键名是 Pipeline的类名称,键值是调用优先级,是一个数字,数字越小则对应的 Pipeline越先被调用。

【步骤4】再重新执行爬取,命令如下所示:

#重新运行，爬取
scrapy crawl quotes

【代码说明】爬取结束后, MongoDB中创建了一个 tutorial的数据库、 Quoteltem的表,如图所示：

【说明】其中text长的部分已经被追加省略号…短的不变，其它author,tags也有加入其中

【结语】至此已经完成最简单用scrapy爬取quotes网站。但更复杂的功能还需进一步学习。

实例代码

quote.py

# -*- coding: utf-8 -*-
import scrapy
from tutorial.items import TutorialItem
 
class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/']
 
    def parse(self, response):
        quotes=response.css(".quote")
        for quote in quotes:
            item=TutorialItem()
            item['text']=quote.css(".text::text").extract_first()
            item['author']=quote.css('.author::text').extract_first()
 
            #tags=quote.css('.tags .tag::text').extract()
            #或<meta class="keywords" itemprop="keywords" content="change,deep-thoughts,thinking,world">
            item['tags']=quote.css('.keywords::attr(content)').extract_first()
            yield item
            #print(text,author,tags)
        #下一页
        next=response.css('.next >a::attr(href)').extract_first()
        #函数.urljoin()把当前网址：http://quotes.toscrape.com/+next获取的 拼接成完整网址
        url=response.urljoin(next)
        yield scrapy.Request(url=url, callback=self.parse)

settings.py

# -*- coding: utf-8 -*-
 
# Scrapy settings for tutorial project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html
 
BOT_NAME = 'tutorial'
 
SPIDER_MODULES = ['tutorial.spiders']
NEWSPIDER_MODULE = 'tutorial.spiders'
 
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'tutorial (+http://www.yourdomain.com)'
 
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
 
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
 
# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
 
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
 
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
 
# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}
 
# Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'tutorial.middlewares.TutorialSpiderMiddleware': 543,
#}
 
# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'tutorial.middlewares.TutorialDownloaderMiddleware': 543,
#}
 
# Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}
 
# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
#    'tutorial.pipelines.TutorialPipeline': 300,
#}
ITEM_PIPELINES = {
   'tutorial.pipelines.TutorialPipeline': 300,
   'tutorial.pipelines.MongoPipeline': 400,
}
MONGO_URI='localhost'
MONGO_DB='tutorial2'
 
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
 
# Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

piplines.py

# -*- coding: utf-8 -*-
 
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
import pymongo
from scrapy.exceptions import DropItem
 
class TutorialPipeline(object):
    def __init__(self):
        self.limit=50
 
    def process_item(self, item, spider):
        if item['text']:
            if len(item['text']) > self.limit:
                item['text']=item['text'][0:self.limit].rstrip()+"..."
            return item
        else:
            return DropItem("text were missing!")
 
class MongoPipeline(object):
    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db
 
    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get('MONGO_URI'),
            mongo_db=crawler.settings.get('MONGO_DB')
        )
 
    def open_spider(self, spider):
        self.client = pymongo.MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]
 
    def process_item(self, item, spider):
        name = item.__class__.__name__
        self.db[name].insert(dict(item))
        return item
 
    def close_spider(self, spider):
        self.client.close()

items.py

# -*- coding: utf-8 -*-
 
# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html
 
import scrapy
 
class TutorialItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    text=scrapy.Field()
    author=scrapy.Field()
    tags=scrapy.Field()
    #pass

main.py

# -*- coding: utf-8 -*-
__author__='pasaulis'
#在程序中运行命令行，方法调试，如：在jobbole.py中打个断点，运行就会停在那
 
from scrapy.cmdline import execute
import sys,os
 
#获取到当前目录：E:\a\scrapy_test\Aticle，方便后面cmd命令运行不必去找目录
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
#测试目录获取是否正确
# print(os.path.dirname(os.path.abspath(__file__)))
 
#调用命令运行爬虫scrapy crawl quotes
execute(["scrapy","crawl","quotes"])