centos7 安装 pyspider 出现的一系列问题及解决方案集合

先安装python3 和 pip3

wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz

安装zlib-devel包(后面安装pip需要用到,这里先下载，后面就不用重复编译)

yum install zlib-devel

tar -xvf Python-3.6.5.tgz

yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel

./configure --prefix=/usr/local/python3/ --with-zlib=/usr/include --enable-loadable-sqlite-extensions

make && make install
 ln -s /usr/local/python3/bin/python3 /usr/local/bin/python3

pip3 安装

wget --no-check-certificate https://pypi.python.org/packages/source/p/pip/pip-8.0.2.tar.gz#md5=3a73c4188f8dbad6a1e6f6d44d117eeb

tar -zxvf pip-8.0.2.tar.gz
cd pip-8.0.2
sudo python3 setup.py build
sudo python3 setup.py install
sudo ln -s /usr/local/python3.4/bin/pip3 /usr/bin/pip3

出现的错误

Collecting pycurl (from pyspider)
Downloading https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz (214kB)
100% |████████████████████████████████| 217kB 172kB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "/tmp/pip-build-ffrwekm7/pycurl/setup.py", line 223, in configure_unix
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/usr/local/python3/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/usr/local/python3/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'curl-config': 'curl-config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-ffrwekm7/pycurl/setup.py", line 913, in <module>
ext = get_extension(sys.argv, split_extension_source=split_extension_source)
File "/tmp/pip-build-ffrwekm7/pycurl/setup.py", line 582, in get_extension
ext_config = ExtensionConfiguration(argv)
File "/tmp/pip-build-ffrwekm7/pycurl/setup.py", line 99, in __init__
self.configure()
File "/tmp/pip-build-ffrwekm7/pycurl/setup.py", line 227, in configure_unix
raise ConfigurationError(msg)
__main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory: 'curl-config': 'curl-config'

在安装 pyspider 的时候我就遇到了这个问题， pyspider 依赖 pycurl 这个库，而 pycurl 要求系统中存在相对应的库。(如果系统自动安装我们要先卸载)

Collecting pycurl
Using cached https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz
Complete output from command python setup.py egg_info:
Using curl-config (libcurl 7.29.0)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-5y2xk7rz/pycurl/setup.py", line 913, in <module>
ext = get_extension(sys.argv, split_extension_source=split_extension_source)
File "/tmp/pip-install-5y2xk7rz/pycurl/setup.py", line 582, in get_extension
ext_config = ExtensionConfiguration(argv)
File "/tmp/pip-install-5y2xk7rz/pycurl/setup.py", line 99, in __init__
self.configure()
File "/tmp/pip-install-5y2xk7rz/pycurl/setup.py", line 316, in configure_unix
specify the SSL backend manually.''')
__main__.ConfigurationError: Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using. Please see PycURL documentation for how to specify the SSL backend manually.

---------------------------------------------------------------------

pip remove pycurl

`yum install libcurl-devel`

export PYCURL_SSL_LIBRARY=openssl >> ~/.bashrc

source ~/.bashrc

pip3 install pycurl

--------------------------------------------------------------------------

ubuntu的方案

sudo apt-get install libssl-dev libcurl4-openssl-dev python-dev

pip3 install pyspider

引用页面

https://blog.csdn.net/blueheart20/article/details/81626516

https://github.com/binux/pyspider

http://www.cnblogs.com/yunlongaimeng/p/9947228.html