【scrapy】创建第一个项目

1)创建项目命令：

scrapy startproject tutorial

该命令将在当前目录下创建tutorial文件夹

2）定义Item

Items are containers that will be loaded with the scraped data;They are declared by creating a scrapy.Item class and defining its attibutes as scrapy.Field objects.

import scrapy

class DmozItem(scrapy.Item):
        title=scrapy.Field()
        link=scrapy.Field()
        desc=scrapy.Field()

3)定义spider

To create a spider,you must subclass scrapy.Spider and define the three main mandatory attributes:

name:identifies the spiders

start_urls:a list of urls where the spider will begin to crawl from.

parse():a method of the spider,which will be called with the downloaded Response object of each start url.The response is passed to the method as the first and only argument.

The parse() methods is in charge of processing the response and returning scraped data(as Item object) and more urls to follow(as Request object).

import scrapy

class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        filename = response.url.split("/")[-2]
        with open(filename, 'wb') as f:
            f.write(response.body)

4)爬取

命令：scrapy crawl dmoz

5)storing the scraped data

命令：scrapy crawl dmoz -o items.json

that will generate a items.json file containing all scraped items,serialized in JSON

If you want to perform more complex things with the scraped items,you can write an Item Pipeline.

------------------------------------------------------------------------------------------------------------------------------------

Introduction to Selectors

There are several ways to extract data from web pages.Scrapy uses a mechanism based on XPath or CSS expressions called Scrapy Selectors.

You can see selectors as objects that represent nodes in the document structure.

Selectors have four basic methods:

xpath():returns a list of selectors

css():returns a list of selectors

extract():returns a unicode string with the selected data

re():returns a list of Unicode strings extracted by applying the regular expression given as argument.

Trying Selectors in the Shell:

Start a shell:

scrapy shell "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/"

After the shell loads,you will have the response fetched in a local response variable,so if you type response.body() you will see the body of the response,or you can type response.headers to see its headers.