scrapy爬虫框架爬取招聘网站

目录结构

BossFace.py文件中代码：

# -*- coding: utf-8 -*-
import scrapy
from ..items import BossfaceItem
import json
class BossfaceSpider(scrapy.Spider):
    name = 'BossFace'
    allowed_domains = ['www.zhipin.com/c101010100-p100109/?page=2']
    start_urls = ['https://fe-api.zhaopin.com/c/i/sou?start=90&pageSize=90&cityId=天津&salary=0,0&workExperience=-1&education=-1&companyType=-1&employmentType=-1&jobWelfareTag=-1&kw=Python&kt=3&=0&at=081a4411244a4e9c80d393212650f005&rt=6cc8df0863c944a88cbc303fa5d7dd40&_v=0.56963230&userCode=1041847897&x-zp-page-request-id=1dba50fde35b475b99fc09aa009dbee1-1568818384291-188677&x-zp-client-id=412ece5e-7595-4148-8838-3b957ac4202a']
    def parse(self, response):

        item = BossfaceItem()
        rs =  json.loads(response.text)['data']
        res = dict(rs)
        lis = res['results']

        for i in range(0,len(lis)):
            item['workName'] = lis[i]['jobName']
            item['workPay'] = lis[i]['salary']
            item['workPosition'] = lis[i]['city']['display']
            item['degree'] = lis[i]['eduLevel']['name']
            item['Company'] = lis[i]['company']['name']
            yield item

items.py中的代码：

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy

class BossfaceItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    workName = scrapy.Field()
    workPay = scrapy.Field()
    workPosition = scrapy.Field()
    degree = scrapy.Field()
    Company = scrapy.Field()
    workDetail = scrapy.Field()
    pass

将这些开启，建立延迟，防止服务器封掉ip

在命令行创建的命令依次是：

1.scrapy startproject bossFace

2.scrapy genspider BossFace www.zhipin.com   #进入spider中执行

3.scrapy crawl BossFace    #进入第二级别的bossFace中执行

4.scrapy crawl BossFace -o item.json  #执行目录与3相同