安装k8s和NVIDIA环境

安装环境

系统要求

CPU： 2个核心

内存： 2GB

显卡：NVIDIA系列

安装docker

apt install docker.io

安装k8s

添加软件源

方便起见，将Ubuntu的软件管理中的下载地址修改为阿里云。

在/etc/apt/source.list添加k8s的软件源

deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main

更新apt update

问题: NO_PUBKEY

NO_PUBKEY BA300B7755AFCFAE

apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 6A030B21BA07F4FB

问题: depend on sth

执行 apt update

或者 apt --fix-broken install

修改HOST

vim /etc/hosts

注释掉

127.0.0.1 computer_name

根据想要形成的集群的IP地址添加

192.168.9.103 master

192.168.9.104 node1

192.168.9.105 node2 ......

#127.0.0.1      localhost
#127.0.1.1      dell3

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

192.168.9.103 master

安装kubeadm

apt install kubeadm

安装kubeadm时会自动安装kubectl、kubelet。

列出需要的镜像

kubeadm config images list

结果是：

k8s.gcr.io/kube-apiserver:v1.17.0
k8s.gcr.io/kube-controller-manager:v1.17.0
k8s.gcr.io/kube-scheduler:v1.17.0
k8s.gcr.io/kube-proxy:v1.17.0
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.5

使用国内源下载这些镜像

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.17.0
  docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.17.0
  docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.0
  docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.17.0
  docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
  docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
  docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.5

使用tag命令打标

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
  docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.17.0 k8s.gcr.io/kube-apiserver:v1.17.0
  docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.17.0 k8s.gcr.io/kube-controller-manager:v1.17.0
  docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.0 k8s.gcr.io/kube-scheduler:v1.17.0
  docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.17.0 k8s.gcr.io/kube-proxy:v1.17.0
  docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0 k8s.gcr.io/etcd:3.4.3-0
  docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.5 k8s.gcr.io/coredns:1.6.5

配置 master

先关闭swap

swapoff -a

进行初始化:

root@dell3:~# kubeadm init --kubernetes-version=v1.17.0 --pod-network-cidr 192.168.0.0/16
W1218 14:48:40.560734   20883 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1218 14:48:40.560767   20883 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
	[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

信息提示关闭swap

swapoff -a

之后再执行安装

安装完成的信息是：

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.9.103:6443 --token tn3e9a.6fgbdbu3vvus8ia9 \
    --discovery-token-ca-cert-hash sha256:ce5aa219f8fd1da40646997f2c3d27ee905989812b115146356ecfc9304036ba 

按照提示执行三个命令：

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

添加网络配置

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

完成的信息是：

podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

查看pod

root@dell3:~# kubectl get pod
No resources found in default namespace.
root@dell3:~# kubectl get pod -n kube-system
NAME                            READY   STATUS     RESTARTS   AGE
coredns-6955765f44-gccbp        0/1     Pending    0          2m33s
coredns-6955765f44-gl7zg        0/1     Pending    0          2m33s
etcd-dell3                      1/1     Running    0          2m33s
kube-apiserver-dell3            1/1     Running    0          2m33s
kube-controller-manager-dell3   1/1     Running    0          2m33s
kube-flannel-ds-amd64-rrhng     0/1     Init:0/1   0          70s
kube-proxy-srnvg                1/1     Running    0          2m33s
kube-scheduler-dell3            1/1     Running    0          2m33s

如果k8s的核心组件都在运行中了，说明k8s安装成功。

安装cuda

NVIDIA建议先安装cuda再安装NVIDIA驱动。

./cuda.run --override 使用--override参数来取消安装时对gcc版本的检查。

安装 NVIDIA 驱动

寻找合适的版本

根据自己的GPU型号，到英伟达网站寻找合适的版本。

安装

方便起见，直接使用Ubuntu19提供的驱动管理软件Additional Drivers来安装

成功的标志

打开 NVIDIA X Server 看到显卡的详细信息。如果打开是空白的，说明当前没有安装NVIDIA驱动。

输入 nvidia-smi 可以看到驱动的详细信息。

root@dell3:~# nvidia-smi
Wed Dec 18 15:07:25 2019
+------------------------------------------------------+
| NVIDIA-SMI 340.107    Driver Version: 340.107        |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 610      Off  | 0000:01:00.0     N/A |                  N/A |
|100%   56C    P8    N/A /  N/A |    129MiB /  1023MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GT 610      Off  | 0000:06:00.0     N/A |                  N/A |
|100%   46C    P8    N/A /  N/A |      3MiB /  1023MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0            Not Supported                                               |
|    1            Not Supported                                               |
+-----------------------------------------------------------------------------+

问题

重复登录

开机之后按Ctrl+Alt+F[1-6]进入字符界面，卸载NVIDIA驱动。

apt remove --purge nvidia-*

如果有驱动的安装包，也可以执行

./nvidia-*.run --uninstall

重启，再次进入原生的图形界面，然后在设置中关闭密码登录。

再次安装NVIDIA驱动。

进不去图形界面

重新安装NVIDIA驱动。

安装 NVIDIA对k8s的插件

安装 nvidia-docker2

# Add the package repositories
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

$ sudo apt-get update && sudo apt-get install -y nvidia-docker2
$ sudo systemctl restart docker

测试nvidia-docker2

使用nvidia-docker2来运行cuda:

docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

第一次运行需下载镜像，如果镜像下载太慢，可以添加加速器。

运行结果是:

root@dell:~# docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
Wed Dec 18 08:55:00 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 750 Ti  Off  | 00000000:01:00.0  On |                  N/A |
| 33%   30C    P8     1W /  38W |    359MiB /  1999MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

添加配置

$ kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/1.0.0-beta4/nvidia-device-plugin.yml

修改runtime

修改/etc/docker/daemon.json文件,添加default-runtime键。

{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

重启docker

# systemctl daemon-reload

# systemctl restart docker

使用

查看k8s是否识别出了GPU

执行kubectl describe node node_name来查看本节点的详细信息:

root@dell:~/mypod# kubectl describe nodes
Name:               dell
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    disktype=ssd
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=dell
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/master=
Annotations:        flannel.alpha.coreos.com/backend-data: {"VtepMAC":"c6:9a:2d:50:03:4b"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.8.52
                    kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 12 Dec 2019 10:25:16 +0800
Taints:             <none>
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Wed, 18 Dec 2019 17:17:55 +0800   Thu, 12 Dec 2019 18:00:39 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 18 Dec 2019 17:17:55 +0800   Thu, 12 Dec 2019 18:00:39 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 18 Dec 2019 17:17:55 +0800   Thu, 12 Dec 2019 18:00:39 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Wed, 18 Dec 2019 17:17:55 +0800   Mon, 16 Dec 2019 10:30:19 +0800   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  192.168.8.52
  Hostname:    dell
Capacity:
 cpu:                4
 ephemeral-storage:  479152840Ki
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             24568140Ki
 nvidia.com/gpu:     1
 pods:               110
Allocatable:
 cpu:                4
 ephemeral-storage:  441587256613
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             24465740Ki
 nvidia.com/gpu:     1
 pods:               110
System Info:
 Machine ID:                 833fac65cd12401db017c0b0033439e7
 System UUID:                28d52460-d7da-11dd-9d00-40167e218cad
 Boot ID:                    7a4a6548-28da-447c-845a-fab20ed82181
 Kernel Version:             5.3.0-24-generic
 OS Image:                   Ubuntu 19.10
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://19.3.2
 Kubelet Version:            v1.16.3
 Kube-Proxy Version:         v1.16.3
PodCIDR:                     192.168.0.0/24
PodCIDRs:                    192.168.0.0/24
Non-terminated Pods:         (13 in total)

此时如果看到capacity中包含了nvidia.com/gpu:1的信息，说明k8s识别出了本机含有一块GPU。

调用GPU

创建一个调用GPU的pod

创建一个文件gpu-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: tf-pod
spec:
  containers:
    - name: tf-container
      image: tensorflow/tensorflow:latest-gpu
      resources:
        limits:
          nvidia.com/gpu: 1 # requesting 1 GPUs

然后执行kubectl apply -f gpu-pod.yaml

使用kubectl get pod查看pod的状态。

root@dell:~/mypod# kubectl get pod
NAME            READY   STATUS              RESTARTS   AGE
busybox-6t962   0/1     Completed           0          6d2h
cuda3           0/1     Completed           47         2d7h
gpu-cuda        0/1     Completed           0          5d
gpu-pod         0/1     Completed           0          5d1h
gpu-pod23       0/2     Pending             0          2d6h
hello-world     0/1     ContainerCreating   0          6d5h
myjob-k9hx5     0/1     Completed           0          6d3h
myjob2-xmdm8    0/1     Completed           0          6d1h
pi-9cttz        0/1     Completed           0          6d2h

使用kubectl describe pod pod_name来查看pod的详细信息。

问题：kubectl 命令无效

像这样的报错：

root@dell3:~# kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/1.0.0-beta4/nvidia-device-plugin.yml The connection to the server 192.168.9.103:6443 was refused - did you specify the right host or port?

多是重启之后连不上docker，解决方法：

# swapoff -a
# systemctl daemon-reload
# systemctl restart docker
# systemctl restart kubelet

其中最重要的就是禁用swap。