scrapy与scrapyd安装

Scrapy是用python编写的爬虫程序。

Scrapyd是一个部署与运行scrapy爬虫的应用，提供JSON API的调用方式来部署与控制爬虫 。

本文验证在fedora与centos是安装成功。

运行需要如下程序：

l  python2.7

l  pip setuptools

l  lxml

l  OpenSSL(pyopenssl)

其中scrapyd需占用6800端口

安装依赖：

yum -y install readline-devel

yum -y install openssl-devel

yum -y install sqlite sqlite-devel

安装python2.7

wget https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz

tar zxvf Python-2.7.9.tgz

cd Python-2.7.9

./configure --with-zlib-dir=/usr/local/lib

make && make install

安装 pip：

wget https://bootstrap.pypa.io/get-pip.py

python2.7 get-pip.py

顺便也会安装 setuptools

安装lxml及依赖

yum install libffi-devel

yum install libxml2

yum install libxslt

yum install libxml2-devel

yum install libxslt-devel

pip install lxml

安装pyOpenSSL

https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.15.1.tar.gz#md5=f447644afcbd5f0a1f47350fec63a4c6 --no-check-certificate

tar zxvf pyOpenSSL-0.15.1.tar

cd pyOpenSSL-0.15.1

python2.7 setup.py install

安装Twisted

yum -y install bzip2-devel

wget https://pypi.python.org/packages/source/T/Twisted/Twisted-14.0.0.tar.bz2#md5=9625c094e0a18da77faa4627b98c9815

tar xf Twisted-14.0.0.tar.bz2

cd Twisted-14.0.0

python2.7 setup.py install

安装scrapy:

pip install Scrapy

测试安装：

scrappy startproject testProject， 当前目录下若能生成scrapy格式的路径，说明安装成功

安装scrapyd

pip install scrapyd

安装 scrapyd-client

wget https://github.com/scrapy/scrapyd-client/archive/master.zip

unzip master.zip

cd scrapyd-client-master

python2.7 setup.py install

scrapyd-client主要是便于在服务端用脚本来部署scrapy程序：scrapyd-deploy.