什么是scrapy-redis

虽然 scrapy 框架是异步加多线程的，但是我们只能在一台主机上运行，爬取效率还是有限的，scrapy-redis 库是基于 scrapy 修改，为我们提供了 scrapy分布式的队列，调度器，去重等等功能，并且原有的 scrapy 单机版爬虫代码只需做很小的改动。有了它，就可以将多台主机组合起来，共同完成一个爬取任务，抓取的效率又提高了。再配合 Scrapyd 与 Gerapy 可以很方便的实现爬虫的分布式部署与运行。

目标任务

使用scrapy-redis爬取 https://hr.tencent.com/position.php?&start= 招聘信息，爬取的内容包括：职位名、详情连接 、职位类别、招聘人数、工作地点、发布时间、具体要求信息。

安装爬虫

  1. pip install scrapy
  2. pip install scrapy-redis
  • python 版本 3.7， scrapy 版本 1.6.0， scrapy-redis 版本 0.6.8

创建爬虫

  1. #  创建工程
  2. scrapy startproject TencentSpider
  4. # 创建爬虫
  5. cd TencentSpider
  6. scrapy genspider -t crawl tencent tencent.com
  • 爬虫名称 tencent , 作用域 tencent.com，爬虫类型 crawl

编写 items.py

  1. # -*- coding: utf-8 -*-
  3. # Define here the models for your scraped items
  4. #
  5. # See documentation in:
  6. # https://doc.scrapy.org/en/latest/topics/items.html
  8. import scrapy
  10. class TencentspiderItem(scrapy.Item):
  11.     # define the fields for your item here like:
  12.     # name = scrapy.Field()
  14.     # 职位名
  15.     positionname = scrapy.Field()
  17.     # 详情连接
  18.     positionlink = scrapy.Field()
  20.     # 职位类别
  21.     positionType = scrapy.Field()
  23.     # 招聘人数
  24.     peopleNum = scrapy.Field()
  26.     # 工作地点
  27.     workLocation = scrapy.Field()
  29.     # 发布时间
  30.     publishTime = scrapy.Field()
  32.     # 职位详情
  33.     positiondetail = scrapy.Field()
  • 定义需求爬取的 item 项

编写 spiders/tencent.py

  1. # -*- coding: utf-8 -*-
  2. import scrapy
  3. from scrapy_redis.spiders import RedisCrawlSpider
  4. # 导入CrawlSpider类和Rule
  5. from scrapy.spiders import CrawlSpider, Rule
  6. # 导入链接规则匹配类，用来提取符合规则的连接
  7. from scrapy.linkextractors import LinkExtractor
  8. from TencentSpider.items import TencentspiderItem
  10. class TencentSpider(RedisCrawlSpider):	# 普通的scrapy爬虫继承自CrawlSpider
  11.     name = 'tencent'
  12.     # allowed_domains = ['tencent.com']
  13.     allowed_domains = ['hr.tencent.com']
  15.     # 普通的scrapy爬虫需要在这里定义start_urls，并且无redis_key变量
  16.     # start_urls = ['https://hr.tencent.com/position.php?&start=0#a']
  17.     redis_key = 'tencent:start_urls'
  19.     # Response里链接的提取规则，返回的符合匹配规则的链接匹配对象的列表
  20.     pagelink = LinkExtractor(allow=("start=\d+"))
  22.     rules = (
  23.         # 获取这个列表里的链接，依次发送请求，并且继续跟进，调用指定回调函数处理
  24.         Rule(pagelink, callback='parse_item', follow=True),
  25.     )
  27.     # CrawlSpider的rules属性是直接从response对象的文本中提取url，然后自动创建新的请求。
  28.     # 与Spider不同的是，CrawlSpider已经重写了parse函数
  29.     # scrapy crawl spidername开始运行，程序自动使用start_urls构造Request并发送请求，
  30.     # 然后调用parse函数对其进行解析，在这个解析过程中使用rules中的规则从html（或xml）文本中提取匹配的链接，
  31.     # 通过这个链接再次生成Request，如此不断循环，直到返回的文本中再也没有匹配的链接，或调度器中的Request对象用尽，程序才停止。
  32.     # 如果起始的url解析方式有所不同，那么可以重写CrawlSpider中的另一个函数parse_start_url(self, response)用来解析第一个url返回的Response，但这不是必须的。
  34.     def parse_item(self, response):
  35.    	    # print(response.request.headers)
  36.         items = []
  37.         url1 = "https://hr.tencent.com/"
  38.         for each in response.xpath("//tr[@class='even'] | //tr[@class='odd']"):
  39.             # 初始化模型对象
  40.             item = TencentspiderItem()
  41.             # 职位名称
  42.             try:
  43.                 item['positionname'] = each.xpath("./td[1]/a/text()").extract()[0].strip()
  44.             except BaseException:
  45.                 item['positionname'] = ""
  47.             # 详情连接
  48.             try:
  49.                 item['positionlink'] = "{0}{1}".format(url1, each.xpath("./td[1]/a/@href").extract()[0].strip())
  50.             except BaseException:
  51.                 item['positionlink'] = ""
  53.             # 职位类别
  54.             try:
  55.                 item['positionType'] = each.xpath("./td[2]/text()").extract()[0].strip()
  56.             except BaseException:
  57.                 item['positionType'] = ""
  59.             # 招聘人数
  60.             try:
  61.                 item['peopleNum'] = each.xpath("./td[3]/text()").extract()[0].strip()
  62.             except BaseException:
  63.                 item['peopleNum'] = ""
  65.             # 工作地点
  66.             try:
  67.                 item['workLocation'] = each.xpath("./td[4]/text()").extract()[0].strip()
  68.             except BaseException:
  69.                 item['workLocation'] = ""
  71.             # 发布时间
  72.             try:
  73.                 item['publishTime'] = each.xpath("./td[5]/text()").extract()[0].strip()
  74.             except BaseException:
  75.                 item['publishTime'] = ""
  77.             items.append(item)
  78.             # yield item
  79.         for item in items:
  80.             yield scrapy.Request(url=item['positionlink'], meta={'meta_1': item}, callback=self.second_parseTencent)
  82.     def second_parseTencent(self, response):
  83.         item = TencentspiderItem()
  84.         meta_1 = response.meta['meta_1']
  85.         item['positionname'] = meta_1['positionname']
  86.         item['positionlink'] = meta_1['positionlink']
  87.         item['positionType'] = meta_1['positionType']
  88.         item['peopleNum'] = meta_1['peopleNum']
  89.         item['workLocation'] = meta_1['workLocation']
  90.         item['publishTime'] = meta_1['publishTime']
  92.         tmp = []
  93.         tmp.append(response.xpath("//tr[@class='c']")[0])
  94.         tmp.append(response.xpath("//tr[@class='c']")[1])
  95.         positiondetail = ''
  96.         for i in tmp:
  97.             positiondetail_title = i.xpath("./td[1]/div[@class='lightblue']/text()").extract()[0].strip()
  98.             positiondetail = positiondetail + positiondetail_title
  99.             positiondetail_detail = i.xpath("./td[1]/ul[@class='squareli']/li/text()").extract()
  100.             positiondetail = positiondetail + ' '.join(positiondetail_detail) + ' '
  102.         # positiondetail_title = response.xpath("//div[@class='lightblue']").extract()
  103.         # positiondetail_detail = response.xpath("//ul[@class='squareli']").extract()
  104.         # positiondetail = positiondetail_title[0] + '\n' + positiondetail_detail[0] + '\n' + positiondetail_title[1] + '\n' + positiondetail_detail[1]
  106.         item['positiondetail'] = positiondetail.strip()
  108.         yield item
  • 爬虫的主逻辑

编写 pipelines.py

  1. # -*- coding: utf-8 -*-
  3. # Define your item pipelines here
  4. #
  5. # Don't forget to add your pipeline to the ITEM_PIPELINES setting
  6. # See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
  7. import json
  9. class TencentspiderPipeline(object):
  10.     """
  11.     功能：保存item数据
  12.     """
  13.     def __init__(self):
  14.         self.filename = open("tencent.json", "w", encoding='utf-8')
  16.     def process_item(self, item, spider):
  17.         try:
  18.             text = json.dumps(dict(item), ensure_ascii=False) + "\n"
  19.             self.filename.write(text)
  20.         except BaseException as e:
  21.             print(e)
  22.         return item
  24.     def close_spider(self, spider):
  25.         self.filename.close()
  • 处理每个页面爬取得到的 item 项

编写 middlewares.py

  1. # -*- coding: utf-8 -*-
  3. # Define here the models for your spider middleware
  4. #
  5. # See documentation in:
  6. # https://doc.scrapy.org/en/latest/topics/spider-middleware.html
  8. import scrapy
  9. from scrapy import signals
  10. from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
  11. import random
  13. class TencentspiderSpiderMiddleware(object):
  14.     # Not all methods need to be defined. If a method is not defined,
  15.     # scrapy acts as if the spider middleware does not modify the
  16.     # passed objects.
  18.     @classmethod
  19.     def from_crawler(cls, crawler):
  20.         # This method is used by Scrapy to create your spiders.
  21.         s = cls()
  22.         crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
  23.         return s
  25.     def process_spider_input(self, response, spider):
  26.         # Called for each response that goes through the spider
  27.         # middleware and into the spider.
  29.         # Should return None or raise an exception.
  30.         return None
  32.     def process_spider_output(self, response, result, spider):
  33.         # Called with the results returned from the Spider, after
  34.         # it has processed the response.
  36.         # Must return an iterable of Request, dict or Item objects.
  37.         for i in result:
  38.             yield i
  40.     def process_spider_exception(self, response, exception, spider):
  41.         # Called when a spider or process_spider_input() method
  42.         # (from other spider middleware) raises an exception.
  44.         # Should return either None or an iterable of Response, dict
  45.         # or Item objects.
  46.         pass
  48.     def process_start_requests(self, start_requests, spider):
  49.         # Called with the start requests of the spider, and works
  50.         # similarly to the process_spider_output() method, except
  51.         # that it doesn’t have a response associated.
  53.         # Must return only requests (not items).
  54.         for r in start_requests:
  55.             yield r
  57.     def spider_opened(self, spider):
  58.         spider.logger.info('Spider opened: %s' % spider.name)
  60. class TencentspiderDownloaderMiddleware(object):
  61.     # Not all methods need to be defined. If a method is not defined,
  62.     # scrapy acts as if the downloader middleware does not modify the
  63.     # passed objects.
  65.     @classmethod
  66.     def from_crawler(cls, crawler):
  67.         # This method is used by Scrapy to create your spiders.
  68.         s = cls()
  69.         crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
  70.         return s
  72.     def process_request(self, request, spider):
  73.         # Called for each request that goes through the downloader
  74.         # middleware.
  76.         # Must either:
  77.         # - return None: continue processing this request
  78.         # - or return a Response object
  79.         # - or return a Request object
  80.         # - or raise IgnoreRequest: process_exception() methods of
  81.         #   installed downloader middleware will be called
  82.         return None
  84.     def process_response(self, request, response, spider):
  85.         # Called with the response returned from the downloader.
  87.         # Must either;
  88.         # - return a Response object
  89.         # - return a Request object
  90.         # - or raise IgnoreRequest
  91.         return response
  93.     def process_exception(self, request, exception, spider):
  94.         # Called when a download handler or a process_request()
  95.         # (from other downloader middleware) raises an exception.
  97.         # Must either:
  98.         # - return None: continue processing this exception
  99.         # - return a Response object: stops process_exception() chain
  100.         # - return a Request object: stops process_exception() chain
  101.         pass
  103.     def spider_opened(self, spider):
  104.         spider.logger.info('Spider opened: %s' % spider.name)
  106. class MyUserAgentMiddleware(UserAgentMiddleware):
  107.     """
  108.     随机设置User-Agent
  109.     """
  110.     def __init__(self, user_agent):
  111.         self.user_agent = user_agent
  113.     @classmethod
  114.     def from_crawler(cls, crawler):
  115.         return cls(
  116.             user_agent=crawler.settings.get('MY_USER_AGENT')
  117.         )
  119.     def process_request(self, request, spider):
  120.         agent = random.choice(self.user_agent)
  121.         request.headers['User-Agent'] = agent

编写 settings.py

  1. # -*- coding: utf-8 -*-
  3. # Scrapy settings for TencentSpider project
  4. #
  5. # For simplicity, this file contains only settings considered important or
  6. # commonly used. You can find more settings consulting the documentation:
  7. #
  8. #     https://doc.scrapy.org/en/latest/topics/settings.html
  9. #     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
  10. #     https://doc.scrapy.org/en/latest/topics/spider-middleware.html
  12. BOT_NAME = 'TencentSpider'
  14. SPIDER_MODULES = ['TencentSpider.spiders']
  15. NEWSPIDER_MODULE = 'TencentSpider.spiders'
  17. # 普通scrapy无下面5项关于redis的配置
  18. # 使用了scrapy_redis的去重组件，在redis数据库里做去重（必须）
  19. DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
  21. # 使用了scrapy_redis的调度器，在redis里分配请求（必须）
  22. SCHEDULER = "scrapy_redis.scheduler.Scheduler"
  24. # 在redis中保持scrapy-redis用到的各个队列，从而允许暂停和暂停后恢复，也就是不清理redis queues（可选参数）
  25. SCHEDULER_PERSIST = True
  27. # 指定redis数据库的连接参数（必须）
  28. REDIS_HOST = '127.0.0.1'
  29. REDIS_PORT = 6379
  30. DUPEFILTER_DEBUG = True
  32. # scrapy-redis在redis中都是用key-value形式存储数据，其中有几个常见的key-value形式：
  33. # 1、 “项目名:items”  -->list 类型，保存爬虫获取到的数据item 内容是 json 字符串
  34. # 2、 “项目名:dupefilter”   -->set类型，用于爬虫访问的URL去重 内容是 40个字符的 url 的hash字符串
  35. # 3、 “项目名:start_urls”   -->List 类型，用于获取spider启动时爬取的第一个url
  36. # 4、 “项目名:requests”   -->zset类型，用于scheduler调度处理 requests 内容是 request 对象的序列化 字符串
  38. # Crawl responsibly by identifying yourself (and your website) on the user-agent
  39. #USER_AGENT = 'TencentSpider (+http://www.yourdomain.com)'
  40. # 设置useragent随机列表
  41. MY_USER_AGENT = [
  42.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36",
  43.     "Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4",
  44.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.21 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.21",
  45.     "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)",
  46.     "Mozilla/5.0 (Windows NT 6.2; rv:30.0) Gecko/20150101 Firefox/32.0",
  47.     "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko",
  48.     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36",
  49.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36",
  50.     "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36",
  51.     "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2)",
  52.     "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)",
  53.     "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0",
  54.     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36",
  55.     "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36",
  56.     "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko",
  57.     "Mozilla/4.0 (compatib1e; MSIE 6.1; Windows NT)",
  58.     "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36",
  59.     "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618)",
  60.     "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)",
  61.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36",
  62.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36",
  63.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36",
  64.     "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36",
  65.     "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; Media Center PC 6.0)",
  66.     "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36",
  67.     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36",
  68.     "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:23.0) Gecko/20100101 Firefox/23.0",
  69.     "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)",
  70.     "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31",
  71.     "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36",
  72.     "Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/20100101 Firefox/17.0",
  73.     "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36",
  74.     "Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.1)",
  75.     "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)",
  76.     "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36",
  77.     "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0",
  78.     "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; InfoPath.2)",
  79.     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134",
  80.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0",
  81.     "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0)",
  82.     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36",
  83.     "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0",
  84.     "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36",
  85.     "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0",
  86.     "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36",
  87.     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763",
  88.     "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.10 Safari/537.36",
  89.     "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11",
  90.     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
  91.     "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)",
  92.     "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0",
  93.     "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0",
  94.     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36",
  95.     "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0",
  96.     "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)",
  97.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36",
  98.     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36",
  99.     "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12) AppleWebKit/602.1.21 (KHTML, like Gecko) Version/9.2 Safari/602.1.21",
  100.     "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko",
  101.     "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729)",
  102.     "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36"
  103. ]
  105. # Obey robots.txt rules
  106. ROBOTSTXT_OBEY = True
  108. # Configure maximum concurrent requests performed by Scrapy (default: 16)
  109. CONCURRENT_REQUESTS = 32
  111. # Configure a delay for requests for the same website (default: 0)
  112. # See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
  113. # See also autothrottle settings and docs
  114. #DOWNLOAD_DELAY = 3
  115. # The download delay setting will honor only one of:
  116. #CONCURRENT_REQUESTS_PER_DOMAIN = 16
  117. #CONCURRENT_REQUESTS_PER_IP = 16
  119. # Disable cookies (enabled by default)
  120. #COOKIES_ENABLED = False
  122. # Disable Telnet Console (enabled by default)
  123. TELNETCONSOLE_ENABLED = False
  125. # Override the default request headers:
  126. #DEFAULT_REQUEST_HEADERS = {
  127. #   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  128. #   'Accept-Language': 'en',
  129. #}
  130. DEFAULT_REQUEST_HEADERS = {
  131.     'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
  132.     'Accept-Encoding': 'gzip,deflate,br',
  133.     'accept-language': 'zh-CN,zh;q=0.9',
  134.     'cache-control': 'no-cache',
  135.     'pragma': 'no-cache',
  136.     'upgrade-insecure-requests': '1',
  137.     'host': 'hr.tencent.com'
  138. }
  140. # Enable or disable spider middlewares
  141. # See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
  142. #SPIDER_MIDDLEWARES = {
  143. #    'TencentSpider.middlewares.TencentspiderSpiderMiddleware': 543,
  144. #}
  146. # Enable or disable downloader middlewares
  147. # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
  148. DOWNLOADER_MIDDLEWARES = {
  149.     'TencentSpider.middlewares.TencentspiderDownloaderMiddleware': None,
  150.     'TencentSpider.middlewares.MyUserAgentMiddleware': 543，
  151.     'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None
  152. }
  154. # Enable or disable extensions
  155. # See https://doc.scrapy.org/en/latest/topics/extensions.html
  156. #EXTENSIONS = {
  157. #    'scrapy.extensions.telnet.TelnetConsole': None,
  158. #}
  160. # Configure item pipelines
  161. # See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
  162. ITEM_PIPELINES = {
  163.     'TencentSpider.pipelines.TencentspiderPipeline': 300,
  164.     # 通过配置RedisPipeline将item写入key为 spider.name : items 的redis的list中，供后面的分布式处理item 这个已经由 scrapy-redis 实现，不需要我们写代码，直接使用即可
  165.     'scrapy_redis.pipelines.RedisPipeline': 100
  166. }
  168. # Enable and configure the AutoThrottle extension (disabled by default)
  169. # See https://doc.scrapy.org/en/latest/topics/autothrottle.html
  170. #AUTOTHROTTLE_ENABLED = True
  171. # The initial download delay
  172. #AUTOTHROTTLE_START_DELAY = 5
  173. # The maximum download delay to be set in case of high latencies
  174. #AUTOTHROTTLE_MAX_DELAY = 60
  175. # The average number of requests Scrapy should be sending in parallel to
  176. # each remote server
  177. #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
  178. # Enable showing throttling stats for every response received:
  179. #AUTOTHROTTLE_DEBUG = False
  181. # Enable and configure HTTP caching (disabled by default)
  182. # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
  183. #HTTPCACHE_ENABLED = True
  184. #HTTPCACHE_EXPIRATION_SECS = 0
  185. #HTTPCACHE_DIR = 'httpcache'
  186. #HTTPCACHE_IGNORE_HTTP_CODES = []
  187. #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
  188. LOG_LEVEL = 'DEBUG'

搭建 redis

这里搭建单机版 windows 版本，需要 linux 版本的自行百度。 下载地址：https://github.com/rgl/redis/downloads 选择最新版和你电脑的对应版本下载安装，这里我选择 redis-2.4.6-setup-64-bit.exe，双击安装，然后将 C:\Program Files\Redis 加入系统环境变量。配置文件为 C:\Program Files\Redis\conf\redis.conf 运行 redis 服务器的命令： redis-server 运行 redis 客户端的命令： redis-cli

运行爬虫

启动爬虫

  1. cd TencentSpider
  2. scrapy crawl tencent
  • TencentSpider 为项目文件夹， tencent 为爬虫名
  • 这时候爬虫会处于等待状态。
  • 可以在本机或者其他主机启动多个爬虫实例，只有所处的主机能够连接 redis 即可

设置 start_urls

  1. # redis-cli
  2. redis 127.0.0.1:6379> lpush tencent:start_urls https://hr.tencent.com/position.php?&start=0#a
  3. (integer) 1
  4. redis 127.0.0.1:6379>

或者运行以下脚本:

  1. # -*- coding: utf-8 -*-
  3. # Define here the models for your scraped items
  4. #
  5. # See documentation in:
  6. # https://doc.scrapy.org/en/latest/topics/items.html
  8. import redis
  10. if __name__ == '__main__':
  11.     conn = redis.Redis(host='127.0.0.1',port=6379)
  12.     # settings 中 REDIS_START_URLS_AS_SET = False  #默认是false，true的话，就是集合，false的话，就为列表
  14.     # 列表
  15.     conn.lpush('tencent:start_urls','https://hr.tencent.com/position.php?&start=0#a')
  17.     # 集合
  18.     # conn.sadd('tencent:start_urls','https://hr.tencent.com/position.php?&start=0#a')
  20.     # conn.close() 无需关闭连接
  • tencent:start_urls 为 spiders/tencent.py 中变量 redis_key 的值
  • 稍等片刻后，所有爬虫会运行，爬取完成后 ctrl + c 停止

结果会保存在 redis 数据库的key tencent:items 中与项目文件夹根目录下的 tencent.json 文件中，内容如下：

  1. {"positionname": "29302-服务采购商务岗", "positionlink": "https://hr.tencent.com/position_detail.php?id=49345&keywords=&tid=0&lid=0", "positionType": "职能类", "peopleNum": "1", "workLocation": "深圳", "publishTime": "2019-04-12", "positiondetail": "工作职责：• 负责相关产品和品类采购策略的制订及实施； • 负责相关产品及品类的采购运作管理，包括但不限于需求理解、供应商开发及选择、供应资源有效管理、商务谈判、成本控制、交付管理、组织验收等 • 支持业务部门的采购需求； • 收集、分析市场及行业相关信息，为采购决策提供依据。 工作要求：• 认同腾讯企业文化理念，正直、进取、尽责；  • 本科或以上学历，管理、传媒、经济或其他相关专业，市场营销及内容类产品运营工作背景者优先； • 五年以上工作经验，对采购理念和采购过程管理具有清晰的认知和深刻的理解；拥有二年以上营销/设计采购、招标相关类管理经验； • 熟悉采购运作及管理，具有独立管理重大采购项目的经验，具有较深厚的采购专业知识；  • 具备良好的组织协调和沟通能力、学习能力和团队合作精神强，具有敬业精神，具备较强的分析问题和解决问题的能力；  • 了解IP及新文创行业现状及发展，熟悉市场营销相关行业知识和行业运作特点； • 具有良好的英语听说读写能力，英语可作为工作语言；同时有日语听说读写能力的优先； • 具备良好的文档撰写能力。计算机操作能力强，熟练使用MS OFFICE办公软件和 ERP 等软件的熟练使用。"}
  2. {"positionname": "CSIG16-自动驾驶高精地图（地图编译）", "positionlink": "https://hr.tencent.com/position_detail.php?id=49346&keywords=&tid=0&lid=0", "positionType": "技术类", "peopleNum": "1", "workLocation": "北京", "publishTime": "2019-04-12", "positiondetail": "工作职责：地图数据编译工具软件开发 工作要求： 硕士以上学历，2年以上工作经验，计算机、测绘、GIS、数学等相关专业；  精通C++编程，编程基础扎实；  熟悉常见数据结构，有较复杂算法设计经验；  精通数据库编程，如MySQL、sqlite等；  有实际的地图项目经验，如地图tile、大地坐标系、OSM等；   至少熟悉一种地图数据规格，如MIF、NDS、OpenDrive等；   有较好的数学基础，熟悉几何和图形学基本算法，；   具备较好的沟通表达能力和团队合作意识。"}
  3. {"positionname": "32032-资深特效美术设计师（上海）", "positionlink": "https://hr.tencent.com/position_detail.php?id=49353&keywords=&tid=0&lid=0", "positionType": "设计类", "peopleNum": "1", "workLocation": "上海", "publishTime": "2019-04-12", "positiondetail": "工作职责：负责游戏3D和2D特效制作，制作规范和技术标准的制定； 与项目组开发人员深入沟通，准确实现项目开发需求。 工作要求：5年以上端游、手游特效制作经验，熟悉UE4引擎； 能熟练使用相关软件和引擎工具制作高品质的3D特效； 善于使用第三方软件制作高品质序列资源，用于引擎特效； 可以总结自己的方法论和经验用于新人和带领团队； 对游戏开发和技术有热情和追求，有责任心，善于团队合作，沟通能力良好，应聘简历须附带作品。"}
  4. ......
  5. ......
  6. ......
  7.  
  8. 此爬虫不保证时效性，如果源站调整就会失效。

scrapy-redis分布式爬取tencent社招信息的更多相关文章

  1. scrapy-redis + Bloom Filter分布式爬取tencent社招信息

    scrapy-redis + Bloom Filter分布式爬取tencent社招信息 什么是scrapy-redis 什么是 Bloom Filter 为什么需要使用scrapy-redis + B ...

  2. 爬虫--scrapy+redis分布式爬取58同城北京全站租房数据

    作业需求: 1.基于Spider或者CrawlSpider进行租房信息的爬取 2.本机搭建分布式环境对租房信息进行爬取 3.搭建多台机器的分布式环境,多台机器同时进行租房数据爬取 建议:用Pychar ...

  3. python爬虫项目(scrapy-redis分布式爬取房天下租房信息)

    python爬虫scrapy项目(二) 爬取目标:房天下全国租房信息网站(起始url:http://zu.fang.com/cities.aspx) 爬取内容:城市:名字:出租方式:价格:户型:面积: ...

  4. Python爬虫基础--分布式爬取贝壳网房屋信息(Client)

    1. client_code01 2. client_code02 3. 这个时候运行多个client就可以分布式进行数据爬取.

  5. Python爬虫基础--分布式爬取贝壳网房屋信息(Server)

    1. server_code01 2. server_code02 3. server_code03

  6. Scrapy 分布式爬取

    由于受到计算机能力和网络带宽的限制,单台计算机运行的爬虫咋爬取数据量较大时,需要耗费很长时间.分布式爬取的思想是“人多力量大”,在网络中的多台计算机同时运行程序,公童完成一个大型爬取任务, Scrap ...

  7. scrapy-redis实现爬虫分布式爬取分析与实现

    本文链接:http://blog.csdn.net/u012150179/article/details/38091411 一 scrapy-redis实现分布式爬取分析 所谓的scrapy-redi ...

  8. 利用 Scrapy 爬取知乎用户信息

    思路:通过获取知乎某个大V的关注列表和被关注列表,查看该大V和其关注用户和被关注用户的详细信息,然后通过层层递归调用,实现获取关注用户和被关注用户的关注列表和被关注列表,最终实现获取大量用户信息. 一 ...

  9. 使用python scrapy爬取知乎提问信息

    前文介绍了python的scrapy爬虫框架和登录知乎的方法. 这里介绍如何爬取知乎的问题信息,并保存到mysql数据库中. 首先,看一下我要爬取哪些内容: 如下图所示,我要爬取一个问题的6个信息: ...

随机推荐

  1. jsp引擎是什么

    1.JSP引擎 执行JSP代码需要在服务器上安装JSP引擎,比较常见的引擎有webLogic和Tomcat.把这些支持JSP的web服务器配置好后,就可以在客户端通过浏览器来访问JSP页面了. 2.J ...

  2. python eval()内置函数

    python有一个内置函数eval(),可以将字符串进行运行. 通过help(eval)查看帮助文档 Help on built-in function eval in module builtins ...

  3. XML 介绍

    XML eXtensible Markup language:可扩展的标记语言 解决HTML不可扩展的问题, 作用:保存或传输数据,不是用来显示数据的. XML介绍 1.  基于文本格式的 2.  标 ...

  4. PAT_A1095#Cars on Campus

    Source: PAT A1095 Cars on Campus (30 分) Description: Zhejiang University has 8 campuses and a lot of ...

  5. Django框架（六）—— 视图层：HttpRequest、HTTPResponse、JsonResponse、CBV和FBV、文件上传

    目录 视图层 一.视图函数 二.视图层之HttpRequest对象 三.视图层之HttpResponse对象 四.视图层之JsonResponse对象 五.CBV和FBV 六.文件上传 视图层 一.视 ...

  6. 20140513 matlab画图

    1.matlab画图 x1=[1.00E-06,2.00E-06,4.00E-06,9.00E-06,2.00E-05,4.00E-05,8.00E-05,2.00E-04,4.00E-04,7.00 ...

  7. 标准 IO 测试 标准输出，输入，出错缓冲大小；全缓冲文本流大小

    例子:测试缓冲区大小 #include <stdio.h> int main(int argc, const char *argv[]) { //标准输入大小,没有输入内容时,标准输入缓冲 ...

  8. 16_TLB与流水线

    1 前面做的实验起始有缺陷 访问内存之后,后面执行两句代码后:并不能保证刚才访问的代码还在TLB中:有可能被刷新出去了: 实验验证缺陷: 代码 不连续 TLB 被淘汰: 2万次中有1次被淘汰:由于访问 ...

  9. 读书笔记一、numpy基础--创建数组

    创建ndarray   (1)使用array函数 接受一切序列型的对象(包括其他数组),然后产生一个新的含有传入数据的numpy数组. import numpy as np #将一个由数值组成列表作为 ...

  10. 链表list

    Don't  lost link! list与vector不同之处在于元素的物理地址可以任意. 为保证对列表元素访问的可行性,逻辑上互为前驱和后继的元素之间,应维护某种索引关系.这种索引关系,可抽象地 ...