超算云(GPU服务器)环境配置

最近在用并行超算云GPU服务器（中国国家网格12区）搭建毕设的环境，这里记录一下。

首先，超算云服务器的登录可以采用网页版、也可以采用客户端（超算云地址：https://cloud.paratera.com/ ）。需要注意的是，并行超算云只提供windows和mac的客户端，Linux用户可能只有使用网页版的界面了（或者用pappcloud直接远程练ssh用vim写:( 哈哈，pappcloud的用法可参见官网下载的《papp_cloud使用手册》）。

超算云上最常见的是用module进行包管理（文档：https://modules.readthedocs.io/en/latest/module.html）。我们可以用module avail命令来查看现有的包：

[macong@paratera01 ~]$ module avail 

------------------------- /usr/share/Modules/modulefiles -------------------------
dot         module-git  module-info modules     null        use.own

-------------------------------- /etc/modulefiles --------------------------------
mpi/compat-openmpi16-x86_64 mpi/mpich-x86_64
mpi/mpich-3.0-x86_64        mpi/openmpi-x86_64
mpi/mpich-3.2-x86_64

----------------------------- /software/modulefiles ------------------------------
alphafold/2.0
anaconda/2.7
anaconda/3.7(default)
anaconda/3.7.4
bcftools/1.10.1

具体新建环境相关操作大家可以参见module的文档，此处不再赘述。大家需要注意的是，因为Pytorch和Tensorflow对应的CUDA版本有出入，我们建议Pytorch和Tensorflow分别装两个不同的环境。

然后根据下列不同的命令对Pytorch和Tensorflow进行装载。

1. Pytorch环境装载与测试

Pytorch 1.9.0 环境装载：

[macong@paratera01 project]$ module load anaconda/3.7.4(tensflow)
[macong@paratera01 project]$ source activate torch
(torch) [macong@paratera01 project]$

可以查看此时的torch版本：

(torch) [macong@paratera01 ~]$ pip list |grep torch
torch                   1.9.0+cu111
torchvision             0.10.0+cu111

接下来我们编写test_torch.py测试文件：

# test_torch.py
import torch
print(torch.cuda.is_available())

采用以下的sub_torch.sh脚本提交到GPU运算节点运行(注意，提交脚本里面一定要有装载环境操作（在计算节点装载），在用户节点里装载环境没用（用户节点只能用于安装依赖包）)

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 5
#SBATCH -p gpu
#SBATCH --gres=gpu:1
#SBATCH --no-requeue
module load anaconda/3.7.4
source activate torch
export PYTHONUNBUFFERED=1
python test_torch.py

提交命令为

sbatch sub_torch.sh

(注意，不是bash sub_torch.sh，bash不能提交到计算节点)

用squeue查看队列情况

[macong@paratera01 project]$ squeue
CLUSTER: priv
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

CLUSTER: swarm
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          16601003       gpu sub_tens   macong  R    INVALID      1 g0002

一段时间后，查看同目录下的slurm-16601003.out输出文件，我们看到

True

说明Pytorch环境配置成功。

2. Tensorflow 2.5.1 环境装载与测试：

Tensorflow 2.5.1 环境装载：

[macong@paratera01 project]$ module load anaconda/3.7.4
[macong@paratera01 project]$ export LD_LIBRARY_PATH=/home/macong/project/cuda/lib64:$LD_LIBRARY_PATH
[macong@paratera01 project]$ source activate tensflow
(tensflow) [macong@paratera01 project]$

加载完毕后，可以查看此时的tensorflow版本

(tensflow) [macong@paratera01 project]$ pip list |grep tensorflow
tensorflow-estimator    2.4.0
tensorflow-gpu          2.4.1

接下来我们编写以下test_tensorflow.py文件：

# test_tensorflow.py
import tensorflow as tf
print(tf.test.is_gpu_available())

采用以下的sub_tensorflow.sh脚本提交到GPU运算节点运行(同样地，提交脚本里面一定要有装载环境操作。另外注意，因为Tensorflow需要cudnn，这里要额外地增加cuda动态链接库的加载地址）

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 5
#SBATCH -p gpu
#SBATCH --gres=gpu:1
#SBATCH --no-requeue
module load anaconda/3.7.4
export LD_LIBRARY_PATH=/home/macong/project/cuda/lib64:$LD_LIBRARY_PATH
source activate tensflow
export PYTHONUNBUFFERED=1
python test_tensorflow.py

提交命令为

sbatch sub_tensorflow.sh

输出显示

job 16601097 on cluster swarm

同样，我们可以用squeue查看队列情况

[macong@paratera01 project]$ squeue
CLUSTER: priv
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

CLUSTER: swarm
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          16601097       gpu sub_tens   macong  R    INVALID      1 g0039

一段时间后，查看同目录下的slurm-16601097.out输出文件，我们看到一长串打印输出

2021-11-28 15:29:22.848812: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:From test_tensorflow.py:2: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2021-11-28 15:30:04.558903: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-28 15:30:04.592168: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-11-28 15:30:04.596694: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-11-28 15:30:04.736951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:84:00.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2021-11-28 15:30:04.737540: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-11-28 15:30:05.810351: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-11-28 15:30:05.810525: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-11-28 15:30:06.033285: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-11-28 15:30:06.193055: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-11-28 15:30:06.630374: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-11-28 15:30:06.820341: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-11-28 15:30:06.847036: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-11-28 15:30:06.850769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-11-28 15:30:06.850852: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-11-28 15:30:09.592923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-11-28 15:30:09.593017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2021-11-28 15:30:09.593043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2021-11-28 15:30:09.628099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/device:GPU:0 with 14761 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:84:00.0, compute capability: 7.0)
True

当然，我们只需要关注最后一行的“True”，说明Tensorflow环境配置成功。

3. 常用命令：

(1) squeue

squeue可用于查看当前的任务队列信息，如之前我们看到的：

[macong@paratera01 project]$ squeue
CLUSTER: priv
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

CLUSTER: swarm
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          16601097       gpu sub_tens   macong  R    INVALID      1 g0039

(2) scancel

scancel+任务id可用于将正在运行的任务杀掉，如杀掉正在运行的16601167任务

scancel  16601167

对应的slurm-16601167.out文件中会显示:

slurmstepd: error: *** JOB 16601167 ON g0011 CANCELLED AT 2021-11-28T10:10:00 ***

更多命令可详见官网《中国国家网格12区用户手册v2.4》