Scrapy爬虫入门系列1 安装

安装python2.7

参见CentOS升级python 2.6到2.7

安装pip

参见CentOS安装python setuptools and pip‎

依赖

https://docs.scrapy.org/en/latest/intro/install.html

• lxml, an efficient XML and HTML parser
• parsel, an HTML/XML data extraction library written on top of lxml,
• w3lib, a multi-purpose helper for dealing with URLs and web page encodings
• twisted, an asynchronous networking framework
• cryptography and pyOpenSSL, to deal with various network-level security needs

安装lxml

pip install lxml 

如果报错：

AttributeError: 'module' object has no attribute 'HTTPSConnection' 

需要在安装python之前先安装openssl,此外还要安装libxml2 and libxslt，以及libffi

yum install -y libxml2 libxml2-devel  libxslt libxslt-devel  libffi-devel  python-devel openssl-devel

pip install cryptography
pip install pyopenssl

pip install parsel


pip install twisted

安装Scrapy

pip install Scrapy 

输入scrapy命令验证一下

报错：

Traceback (most recent call last):

  File "/usr/local/bin/scrapy", line 5, in <module>

    from pkg_resources import load_entry_point

  File "/usr/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 2607, in <module>

  File "/usr/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 565, in resolve

pkg_resources.DistributionNotFound: setuptools>=1.0

解决

pip install --upgrade scrapy

艺搜参考

http://doc.scrapy.org/en/latest/intro/install.html#intro-install