关于Scrapy框架的基本概念

Scrapy爬取网页基本概念

Scrapy爬取网页基本概念

怎么样用Scrapy生成project？

scrapy startproject xxx

如何用Scrapy爬取网页？

import scrapy
from scrapy.contrib.spiders import CrawlSpider
from scrapy.http import Request
from scrapy.selector import Selector

xxx=selector.xpath(xxxxx).extract()

Scrapy的文件结构

Project中包含：

• items.py
• settings.py
• pipelines.py

1. items.py

---

Item objects are simple containers used to collect the scraped data. They provide a dictionary-like API with a convenient syntax for declaring their available fields.——Scrapy官方手册

---

items.py定义需要抓取并需要后期处理的数据

2. settings.py

---

The Scrapy settings allows you to customize the behaviour of all Scrapy components, including the core, extensions, pipelines and spiders themselves.——Scrapy官方手册

---

settings.py文件配置Scrapy，从而修改user-agent，设定爬取时间间隔，设置代理，配置各种中间件等等

3. pipelines.py

---

After an item has been scraped by a spider, it is sent to the Item Pipeline which process it through several components that are executed sequentially.——Scrapy官方手册

---

pipelines.py用于存放执行后期数据处理的功能，从而使得数据的爬取和处理分开。