Python学习总结 13 Scrapy

当前环境是 Win8 64位的，使用的Python 3.5 版本。

一 安装Scrapy

1，安装 lxml

pip install lxml -i https://pypi.douban.com/simple

2， 安装 Scrapy

pip install Scrapy -i https://pypi.douban.com/simple

　　在windows下安装有可能失败，报以下异常。

building 'twisted.test.raiser' extension
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

解决方案：

　　http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted 下载twisted对应版本的whl文件（如我的Twisted-17.5.0-cp35-cp35m-win_amd64.whl），cp后面是Python版本，amd64代表64位，运行命令：

pip install Twisted-17.5.-cp35-cp35m-win_amd64.whl

　　安装完成后，再次运行 pip install Scrapy -i https://pypi.douban.com/simple , 就可以安装成功 Scrapy.

3, 验证安装

　　在DOS 状态下输入 Scrapy命令，如果提示如下命令，就证明安装成功啦，如果失败了，请检查上述步骤有何疏漏。

C:\Users\si\Desktop\>Scrapy
Scrapy 1.4. - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command

参考资料：

https://doc.scrapy.org/en/latest/intro/tutorial.html