Scrapy代码实战

1.Spider爬虫代码

 # -*- coding: utf-8 -*-
 import scrapy
 from yszd.items import YszdItem

 class YszdSpiderSpider(scrapy.Spider):
     # 爬虫名称，启动爬虫时必须的参数
     name = 'yszd_spider'
     # 爬取域范围，运行爬虫在这个域名下爬取数据（可选）
     allowed_domains = ['itcast.cn']
     # 起始url列表，爬虫执行后第一批请求将从这个列表里获取
     start_urls = ['http://www.itcast.cn/channel/teacher.shtml']

     def parse(self, response):
         # //表示跳级定位，即对当前元素的所有子节点进行查找，一般开头都是使用跳级定位
         # div[@class='li_txt'] : 查找div且属性class='li_txt'的
         node_list = response.xpath("//div[@class='li_txt']")
         # 存储所有item字段
         # items = []
         for node in node_list:
             # 创建item字段对象用来存储信息
             item = YszdItem()
             # extract() : 将xpath对象转换为Unicode字符串
             name = node.xpath("./h3/text()").extract()
             title = node.xpath("./h4/text()").extract()
             info = node.xpath("./p/text()").extract()

             item['name'] = name[0]
             item['title'] = title[0]
             item['info'] = info[0]

             yield item
             # items.append(item)

2.Item代码（定义爬取的字段）

 # -*- coding: utf-8 -*-

 # Define here the models for your scraped items
 #
 # See documentation in:
 # https://doc.scrapy.org/en/latest/topics/items.html

 import scrapy

 class YszdItem(scrapy.Item):
     name = scrapy.Field()
     title = scrapy.Field()
     info = scrapy.Field()

3.Pipelines管道代码

 # -*- coding: utf-8 -*-

 # Define your item pipelines here
 #
 # Don't forget to add your pipeline to the ITEM_PIPELINES setting
 # See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html

 import json

 class YszdPipeline(object):
     def __init__(self):
         self.f = open("yszd.json", "w")

     def process_item(self, item, spider):
         # ensure_ascii默认是True，会把内容转换为unicode
         text = json.dumps(dict(item), ensure_ascii=False) + "\n"
         self.f.write(text)
         return item

     def close_spider(self, spider):
         self.f.close()

4.setting代码（开启管道，300表示优先级，越小优先级越高）

5.运行爬虫

　　执行命令：scrapy crawl yszd_spider

　　注意：yszd_spider为你定义爬虫的名称，与1中的第8行代码对应！

6.执行结果