深度学习环境搭建部署（DeepLearning 神经网络）

工作环境

系统：Ubuntu  LTS
显卡：GPU
NVIDIA驱动：410.93
CUDA：10.0
Python：.x

CUDA以及NVIDIA驱动安装，详见https://www.cnblogs.com/orzs/p/10951473.html

需要部署的软件

conda环境
nccl2环境
openmpi环境
horovod环境

1. 创建conda环境

官网下载地址：https://www.anaconda.com/distribution/#download-section

下载合适的安装文件，然后运行。

 cd init
 sudo wget https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh
 bash Anaconda3-2019.03-Linux-x86_64.sh

根据提示操作，并选择安装目录，默认安装在~/anaconda3/ 目录下。

注：初始化操作

1、如果默认不初始化，则安装之后，没有conda命令，需要手动初始化

注：为避免用户名泄露，此处的用户名均已$USER替代

installation finished.
Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]
[no] >>>

You have chosen to not have conda modify your shell scripts at all.
To activate conda's base environment in your current shell session:

eval "$(/home/$USER/anaconda3/bin/conda shell.YOUR_SHELL_NAME hook)"

To install conda's shell functions for easier access, first activate, then:

conda init

If you'd prefer that conda's base environment not be activated on startup,
set the auto_activate_base parameter to false:

conda config --set auto_activate_base false

Thank you for installing Anaconda3!

===========================================================================

Anaconda and JetBrains are working together to bring you Anaconda-powered
environments tightly integrated in the PyCharm IDE.

PyCharm for Anaconda is available at:
https://www.anaconda.com/pycharm

2、如果选择初始化，则会修改~/.bashrc文件，并创建conda命令

installation finished.
Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]
"deeplearning" 105L, 3558C written

installation finished.
Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]
[no] >>> yes
WARNING: The conda.compat module is deprecated and will be removed in a future release.
no change /home/$USER/anaconda3/condabin/conda
no change /home/$USER/anaconda3/bin/conda
no change /home/$USER/anaconda3/bin/conda-env
no change /home/$USER/anaconda3/bin/activate
no change /home/$USER/anaconda3/bin/deactivate
no change /home/$USER/anaconda3/etc/profile.d/conda.sh
no change /home/$USER/anaconda3/etc/fish/conf.d/conda.fish
no change /home/$USER/anaconda3/shell/condabin/Conda.psm1
no change /home/$USER/anaconda3/shell/condabin/conda-hook.ps1
no change /home/$USER/anaconda3/lib/python3./site-packages/xonsh/conda.xsh
no change /home/$USER/anaconda3/etc/profile.d/conda.csh
modified /home/$USER/.bashrc

==> For changes to take effect, close and re-open your current shell. <==

If you'd prefer that conda's base environment not be activated on startup,
set the auto_activate_base parameter to false:

conda config --set auto_activate_base false

Thank you for installing Anaconda3!

===========================================================================

Anaconda and JetBrains are working together to bring you Anaconda-powered
environments tightly integrated in the PyCharm IDE.

PyCharm for Anaconda is available at:
https://www.anaconda.com/pycharm

执行以下命令，使conda环境生效

 source ~/.bashrc

2. 进入conda py3.6

 conda create -n py36 python=3.6
 conda activate py36

3. 安装必要包

#修改清华的pip源

 mkdir ~/.pip
 touch ~/.pip/pip.conf

#pip.conf中写入以下内容

[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple

安装包

 pip
 pip install opencv-python==4.1.0.25
 pip
 pip 

4. 安装nccl2

下载地址：https://docs.nvidia.com/deeplearning/sdk/nccl-install-guide/index.html

根据系统和cuda版本下载对应的nccl2

 -ga-cuda10.0_1-1_amd64.deb
 -ga-cuda10./7fa2af80.pub（根据提示执行）
 sudo apt update
 -+cuda10. libnccl-dev=-+cuda10.

5、安装libcudnn

根据版本，下载对应的文件：https://developer.nvidia.com/rdp/cudnn-download

 -+cuda10.0_amd64.deb
 -+cuda10.0_amd64.deb

6. 安装openmpi

下载地址：https://www.open-mpi.org/faq/?category=building#easy-build

 sudo wget https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.1.tar.gz
 .tar.gz | tar xf -
 cd openmpi-/
 sudo ./configure --prefix=/usr/local
 sudo make all install

7. 安装horovod

文档说明：https://github.com/horovod/horovod/blob/master/docs/gpus.rst

 HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir horovod

注：HOROVOD_WITH_TENSORFLOW=1  可开启debug模式。

至此，深度学习环境安装完成，接下来即可做深度训练。

conda环境常用命令

如何默认不使用conda环境

 conda config --set auto_activate_base false

退出conda环境

 conda deactivate

进入conda环境

 conda activate

安装过程中可能出现的问题：

1、

ImportError: libcudnn.so.: cannot open shared object file: No such file or directory

原因：cudann未安装或者版本错误

解决：根据版本，下载对应的文件：https://developer.nvidia.com/rdp/cudnn-download

 -+cuda10.0_amd64.deb
 -+cuda10.0_amd64.deb

2、

ImportError: libcuda.so.: cannot open shared object file: No such file or directory

原因：一般是cuda版本不对导致

解决：安装对应的cuda版本即可

3、

ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

原因：一般情况是cuda链接库的问题

解决：执行以下命令即可

 sudo ldconfig /usr/local/cuda/lib64

4、奇葩问题：

ModuleNotFoundError: No module named 'cv2'

如果未安装opencv-python，直接执行以下命令安装即可

 pip install opencv-python==4.1.0.25

如果已经安装，依然错误提示，我遇到的情况是，Python被劫持

执行命令

 which python

回显提示

~/anaconda3/envs/py36/bin/python

执行

 ~/anaconda3/envs/py36/bin/python

看到的版本是3.6.8

但是直接python看到是3.6.6

原因：python被劫持

解决：将~/.bashrc里的python环境变量清除即可

# alias python=/usr/bin/python3.

5、执行以下命令报错

 conda create -n py36 python=3.6

WARNING: The conda.compat module is deprecated and will be removed in a future release.
Collecting package metadata: failed

UnavailableInvalidChannel: The channel is not accessible or is invalid.
  channel name: anaconda/pkgs/free
  channel url: https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
  error code: 

You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.

检查conda配置（以前曾经安装过conda）

 conda config --show-sources

channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
  - defaults
show_channel_urls: True

原因：conda已经不支持外部源

解决：删除清华的源即可

 conda config --remove channels 'https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/'
 conda config --remove channels 'https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/'

6、

[$USER-nmg-:] mca_base_component_repository_open: unable to open mca_oob_ud: libibverbs.so.: cannot open shared object file: No such file or directory (ignored)
[$USER-nmg-:] mca_base_component_repository_open: unable to open mca_oob_ud: libibverbs.so.: cannot open shared object file: No such file or directory (ignored)
[$USER-nmg-:] mca_base_component_repository_open: unable to open mca_btl_openib: libibverbs.so.: cannot open shared object file: No such file or directory (ignored)

原因：缺少libibverbs.so.1导致

解决：安装libibverbs1即可

 apt-cache search libibverbs
 sudo apt-get install libibverbs1

7、

python: symbol lookup error: /usr/local/lib/openmpi/mca_coll_cuda.so: undefined symbol: opal_cuda_check_bufs

原因：openmpi安装有问题或者版本冲突导致

解决：卸载并重新安装openmpi即可。

 cd /where/your/old_mpi/sources/are   //进入其他版本的安装目录
 sudo make uninstall
 sudo rm -rf /usr/local/lib/openmpi /usr/local/lib/libmca* /usr/local/lib/libmpi* /usr/local/lib/libompitrace* /usr/local/lib/libopen* /usr/local/lib/liboshmem* /usr/local/lib/mpi_*
 cd /where/your/mpi/sources/are   //进入需要安装的版本的目录
 sudo ./configure --prefix=/usr/local
 sudo make all install

8、

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

我遇到的情况是，已经安装了对应的版本（cuda10.0、libcudnn7-dev_7.6.0.64、tensorflow-gpu-1.13.1），但是被/usr/local/cuda-9.0/空目录影响到了，删除此目录即可。

 sudo rm -rf /usr/local/cuda-9.0/