Scrapy 源代码分析系列-4 scrapy.commands 子包

子包scrapy.commands定义了在命令scrapy中使用的子命令(subcommand): bench, check, crawl, deploy, edit, fetch,

genspider, list, parse, runspider, settings, shell, startproject, version, view。 所有的子命令模块都定义了一个继承自

类ScrapyCommand的子类Command。

首先来看一下子命令crawl, 该子命令用来启动spider。

1. crawl.py

关注的重点在方法run(self, args, opts):

 def run(self, args, opts):
if len(args) < 1:
raise UsageError()
elif len(args) > 1:
raise UsageError("running 'scrapy crawl' with more than one spider is no longer supported")
spname = args[0] crawler = self.crawler_process.create_crawler() # A
spider = crawler.spiders.create(spname, **opts.spargs) # B
crawler.crawl(spider) # C
self.crawler_process.start() # D

那么问题来啦,run接口方法是从哪里调用的呢? 让我们回到 Python.Scrapy.11-scrapy-source-code-analysis-part-1

中 "1.2 cmdline.py command.py" 关于"_run_print_help() "的说明。

A: 创建类Crawler对象crawler。在创建Crawler对象时, 同时将创建Crawler对象的实例属性spiders(SpiderManager)。如下所示:

 class Crawler(object):

     def __init__(self, settings):
self.configured = False
self.settings = settings
self.signals = SignalManager(self)
self.stats = load_object(settings['STATS_CLASS'])(self)
self._start_requests = lambda: ()
self._spider = None
# TODO: move SpiderManager to CrawlerProcess
spman_cls = load_object(self.settings['SPIDER_MANAGER_CLASS'])
self.spiders = spman_cls.from_crawler(self) # spiders 的类型是: SpiderManager

Crawler对象对应一个SpiderManager对象,而SpiderManager对象管理多个Spider。

B: 获取Sipder对象。

C: 为Spider对象安装Crawler对象。(为蜘蛛安装爬行器)

D: 类CrawlerProcess的start()方法如下:

     def start(self):
if self.start_crawling():
self.start_reactor() def start_crawling(self):
log.scrapy_info(self.settings)
return self._start_crawler() is not None def start_reactor(self):
if self.settings.getbool('DNSCACHE_ENABLED'):
reactor.installResolver(CachingThreadedResolver(reactor))
reactor.addSystemEventTrigger('before', 'shutdown', self.stop)
reactor.run(installSignalHandlers=False) # blocking call def _start_crawler(self):
if not self.crawlers or self.stopping:
return name, crawler = self.crawlers.popitem()
self._active_crawler = crawler
sflo = log.start_from_crawler(crawler)
crawler.configure()
crawler.install()
crawler.signals.connect(crawler.uninstall, signals.engine_stopped)
if sflo:
crawler.signals.connect(sflo.stop, signals.engine_stopped)
crawler.signals.connect(self._check_done, signals.engine_stopped)
crawler.start() # 调用类Crawler的start()方法
return name, crawler

类Crawler的start()方法如下:

     def start(self):
yield defer.maybeDeferred(self.configure)
if self._spider:
yield self.engine.open_spider(self._spider, self._start_requests()) # 和Engine建立了联系 (ExecutionEngine)
yield defer.maybeDeferred(self.engine.start)

关于类ExecutionEngine将在子包scrapy.core分析涉及。

2. startproject.py

3. subcommand是如何加载的

在cmdline.py的方法execute()中有如下几行代码:

     inproject = inside_project()
cmds = _get_commands_dict(settings, inproject)
cmdname = _pop_command_name(argv)

_get_commands_dict():

 def _get_commands_dict(settings, inproject):
cmds = _get_commands_from_module('scrapy.commands', inproject)
cmds.update(_get_commands_from_entry_points(inproject))
cmds_module = settings['COMMANDS_MODULE']
if cmds_module:
cmds.update(_get_commands_from_module(cmds_module, inproject))
return cmds

_get_commands_from_module():

 def _get_commands_from_module(module, inproject):
d = {}
for cmd in _iter_command_classes(module):
if inproject or not cmd.requires_project:
cmdname = cmd.__module__.split('.')[-1]
d[cmdname] = cmd()
return d

To Be Continued

接下来解析settings相关的逻辑。Python.Scrapy.15-scrapy-source-code-analysis-part-5

Python.Scrapy.14-scrapy-source-code-analysis-part-4的更多相关文章

  1. Memcached source code analysis (threading model)--reference

    Look under the start memcahced threading process memcached multi-threaded mainly by instantiating mu ...

  2. Golang Template source code analysis(Parse)

    This blog was written at go 1.3.1 version. We know that we use template thought by followed way: fun ...

  3. Memcached source code analysis -- Analysis of change of state--reference

    This article mainly introduces the process of Memcached, libevent structure of the main thread and w ...

  4. Apache Commons Pool2 源码分析 | Apache Commons Pool2 Source Code Analysis

    Apache Commons Pool实现了对象池的功能.定义了对象的生成.销毁.激活.钝化等操作及其状态转换,并提供几个默认的对象池实现.在讲述其实现原理前,先提一下其中有几个重要的对象: Pool ...

  5. Redis source code analysis

    http://zhangtielei.com/posts/blog-redis-dict.html http://zhangtielei.com/assets/photos_redis/redis_d ...

  6. linux kernel & source code analysis& hacking

    https://kernelnewbies.org/ http://www.tldp.org/LDP/lki/index.html https://kernelnewbies.org/ML https ...

  7. 2018.6.21 HOLTEK HT49R70A-1 Source Code analysis

    Cange note: “Reading TMR1H will latch the contents of TMR1H and TMR1L counter to the destination”? F ...

  8. The Ultimate List of Open Source Static Code Analysis Security Tools

    https://www.checkmarx.com/2014/11/13/the-ultimate-list-of-open-source-static-code-analysis-security- ...

  9. Top 40 Static Code Analysis Tools

    https://www.softwaretestinghelp.com/tools/top-40-static-code-analysis-tools/ In this article, I have ...

  10. Source Code Reading for Vue 3: How does `hasChanged` work?

    Hey, guys! The next generation of Vue has released already. There are not only the brand new composi ...

随机推荐

  1. jquery 复制DIV与相关事件

    <div class="pages_img fix" name="fixpages">                <div class=& ...

  2. 【Linux】浅谈段页式内存管理

    让我们来回顾一下历史,在早期的计算机中,程序是直接运行在物理内存上的.换句话说,就是程序在运行的过程中访问的都是物理地址.如果这个系统只运行一个程序,那么只要这个程序所需的内存不要超过该机器的物理内存 ...

  3. 万恶的ViewBag

    作个记录,鼓捣了一晚上,他竟然把订单事情给我篡改了 ViewBag.orderNo="20150613202108844"; 页面上出来的竟然是20150613202108840 ...

  4. [ActionScript 3.0] AS3.0 火焰效果

    package { import flash.display.Sprite; import flash.display.* import flash.events.Event; import flas ...

  5. 黑马程序员_JAVA基础知识总结3

    ------- android培训.java培训.期待与您交流! ---------- Java源文件的扩展名是.java,编译之后生成.class的文件.所有的类都有一个共同的继承祖先Object类 ...

  6. 8. String to Integer (atoi)

    题目: Implement atoi to convert a string to an integer. Hint: Carefully consider all possible input ca ...

  7. 54. Search a 2D Matrix && Climbing Stairs (Easy)

    Search a 2D Matrix Write an efficient algorithm that searches for a value in an m x n matrix. This m ...

  8. Android将Activity 打 jar包 (解决资源文件不能打包的问题)

    转载地址:http://blog.csdn.net/xiaanming/article/details/9257853 最近有一个需要,我们公司做了一个apk客户端,然后其他的公司可以根据自己的需要来 ...

  9. 洛谷P3367 【模板】并查集

    P3367 [模板]并查集 293通过 551提交 题目提供者HansBug 标签 难度普及- 提交  讨论  题解 最新讨论 不知道哪错了 为啥通不过最后三个节点 题解 不懂为什么MLE 最后一个数 ...

  10. jQuery ajax()使用serialize()提交form数据

    jQuery的serialize()方法通过序列化表单值,创建URL编码文本字符串,我们就可以选择一个或多个表单元素,也可以直接选择form将其序列化 <form action="&q ...