Scrapy 应用之爬取《盗墓笔记》

爬取《盗墓笔记》和爬取《宦海沉浮》原理一样，但是使用了两种不同的追踪链接的方式，《盗墓笔记》使用的是跟踪下一页链接，直至没有下一页为止，《宦海沉浮》则是将所有的url都放在start_urls中进行爬取

第一步：新建项目

KeysdeMacBook:Desktop keys$ scrapy startproject Novel
New Scrapy project 'Novel', using template directory '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/scrapy/templates/project', created in:
    /Users/keys/Desktop/Novel
You can start your first spider with:
    cd Novel
    scrapy genspider example example.com

　　

第二步：创建爬虫

KeysdeMacBook:Desktop keys$ cd Novel/
KeysdeMacBook:MyCrawl keys$ scrapy genspider BiJi www.mossiella.com/html/255.html

第三步：配置item.py

# -*- coding: utf-8 -*-

import scrapy

class NovelItem(scrapy.Item):
    url = scrapy.Field()
    title = scrapy.Field()
    content = scrapy.Field()

　　

第四步：编写爬虫

# -*- coding: utf-8 -*-
import scrapy
from Novel.items import NovelItem

class BijiSpider(scrapy.Spider):
    name = 'BiJi'
    allowed_domains = ['www.mossiella.com']
    start_urls = ['http://www.mossiella.com/html/255.html/']

    def parse(self, response):
        url = response.url
        title = response.xpath('//h1/text()').extract_first('')
        content = ','.join(response.xpath('//div[@class="zwcent"]/p/text()').extract())
        myitem = NovelItem()
        myitem['url'] = url
        myitem['title'] = title
        myitem['content'] = content

        yield myitem
        next_url = response.xpath('//a[@rel="next"]')

        if next_url:
            url = next_url.css('::attr(href)').extract_first('')
            yield scrapy.Request(url=url, callback=self.parse)

　　

第五步：配置pipeline.py

import pymysql

class MysqlPipeline(object):
    # 采用同步的机制写入mysql
    def __init__(self):
        self.conn = pymysql.connect(
            '127.0.0.1',
            'root',
            'rootkeys',
            'Article',
            charset="utf8",
            use_unicode=True)
        self.cursor = self.conn.cursor()

    def process_item(self, item, spider):
        insert_sql = """
            insert into BiJi(url, title, content)
            VALUES (%s, %s, %s)
        """
        # 使用VALUES实现传值
        self.cursor.execute(
            insert_sql,
            (item["url"],
             item["title"],
             item["content"]))
        self.conn.commit()

　　

第六步：配置setting.py

BOT_NAME = 'Novel'
SPIDER_MODULES = ['Novel.spiders']
NEWSPIDER_MODULE = 'Novel.spiders'
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
ROBOTSTXT_OBEY = False

　　

第七步：运行爬虫

import os
import sys
from scrapy.cmdline import execute

sys.path.append(os.path.dirname(os.path.abspath(__file__)))

run_spider = 'BiJi'

if __name__ == '__main__':
    print('Running Spider of ' + run_spider)
    execute(['scrapy', 'crawl', run_spider])