Kubernets二进制安装(11)之部署Node节点服务的kubelet

集群规划

主机名	角色	IP地址
mfyxw30.mfyxw.com	kubelet	192.168.80.30
mfyxw40.mfyxw.com	kubelet	192.168.80.40

注意：这里部署文档以mfyxw30.mfyxw.com主机为例，另外一台运算节点安装部署方法类似

1.创建生成kubelet证书签名请求(csr)的JSON配置文件

在运维主机mfyxw50.mfyxw.com上操作

[root@mfyxw50 cert]#cat > /opt/certs/kubelet-csr.json << EOF
{
    "CN": "kubelet-node",
    "hosts": [
    "127.0.0.1",
    "192.168.80.100",
    "192.168.80.10",
    "192.168.80.20",
    "192.168.80.30",
    "192.168.80.40",
    "192.168.80.50"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "GuangDong",
            "L": "GuangZhou",
            "O": "od",
            "OU": "ops"
        }
    ]
}
EOF

2.生成kubelet证书和私钥

[root@mfyxw50 ~]#cd /opt/certs/
[root@mfyxw50 certs]#cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssljson -bare kubelet

3.复制证书至各运算节点(Node节点)

在运维主机mfyxw50.mfyxw.com上操作

[root@mfyxw50 certs]# scp -r kubelet.pem kubelet-key.pem mfyxw30:/opt/kubernetes/server/bin/cert/
[root@mfyxw50 certs]# scp -r kubelet.pem kubelet-key.pem mfyxw40:/opt/kubernetes/server/bin/cert/

4.查看复制过去的私钥的权限是否是600

#分别在mfyxw30.mfyxw.com和mfyxw40.mfyxw.com主机上查看kubelet-key.pem的权限是否是600,图片以mfyxw30主机为例
[root@mfyxw30 ~]# ls -l /opt/kubernetes/server/bin/cert/

[root@mfyxw40 ~]# ls -l /opt/kubernetes/server/bin/cert/

5.创建配置

在mfyxw30.mfyxw.com上操作，只需要在mfyxw30或mfyxw40任一台主机中执行如下步骤(（1）-（7）)即可

(1)设置set-cluster #创建需要连接的集群信息，可以创建多个k8s集群信息

注意在/opt/kubernetes/server/conf目录下

[root@mfyxw30 ~]# mkdir -p /opt/kubernetes/server/conf/
[root@mfyxw30 ~]# cd /opt/kubernetes/server/conf/
[root@mfyxw30 conf]#kubectl config set-cluster myk8s \
  --certificate-authority=/opt/kubernetes/server/bin/cert/ca.pem \
  --embed-certs=true \
  --server=https://192.168.80.100:7443 \
  --kubeconfig=kubelet.kubeconfig

(2)设置set-credentials #创建用户账号，即用户登陆使用的客户端私有和证书，可以创建多个证书

注意在/opt/kubernetes/server/conf目录下

[root@mfyxw30 conf]#kubectl config set-credentials k8s-node \
--client-certificate=/opt/kubernetes/server/bin/cert/client.pem \
--client-key=/opt/kubernetes/server/bin/cert/client-key.pem \
--embed-certs=true --kubeconfig=kubelet.kubeconfig

(3)set-context # 设置context，即确定账号和集群对应关系

注意在/opt/kubernetes/server/conf目录下

[root@mfyxw30 conf]#kubectl config set-context myk8s-context \
  --cluster=myk8s \
  --user=k8s-node \
  --kubeconfig=kubelet.kubeconfig

(4)use-context # 设置当前使用哪个context

注意在/opt/kubernetes/server/conf目录下

[root@mfyxw30 conf]#kubectl config use-context myk8s-context \
--kubeconfig=kubelet.kubeconfig

(5)创建资源配置文件k8s-node.yaml

[root@mfyxw30 conf]# cat > /opt/kubernetes/server/bin/conf/k8s-node.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: k8s-node
EOF

(6)应用资源配置文件

[root@mfyxw30 ~]#cd /opt/kubernetes/server/bin/conf/
[root@mfyxw30 conf]#kubectl create -f k8s-node.yaml

(7)检查

[root@mfyxw30 ~]#kubectl get clusterrolebinding k8s-node

(8)将生成的kubelet.kubeconfig文件复制到mfyxw40主机上的/opt/kubernetes/server/conf目录下

#在mfyxw40主机上创建目录
[root@mfyxw40 ~]# mkdir -p  /opt/kubernetes/server/conf/

#在mfyxw30主机上把kubelet-kubeconfig文件复制至mfyxw40主机下
[root@mfyxw30 ~]#cd /opt/kubernetes/server/conf/
[root@mfyxw30 conf]# scp -r kubelet.kubeconfig mfyxw40:/opt/kubernetes/server/conf/

6.创建基础镜像pause

在运维主机mfyxw50.mfyxw.com上运行

(1)下载pause镜像

[root@mfyxw50 ~]# docker pull kubernetes/pause
[root@mfyxw50 ~]# docker images | grep pause

(2)给pause重新打标签

[root@mfyxw50 ~]# docker tag kubernetes/pause:latest harbor.od.com/public/pause:latest

(3)登录到harbor.od.com私有仓库 并 上传重新打标签的pause到私有仓库

[root@mfyxw50 ~]# docker login harbor.od.com     #会提示输入私有仓库的用户名和密码
[root@mfyxw50 ~]# docker push harbor.od.com/public/pause:latest

(4)登录到网页端的harbor.od.com查看pause是否已经上传

在登录harbor.od.com遇到的故障

错误提示：502 Bad Gateway，那是因为harbor没有启动

解决方法

进入到harbor的目录启动harbor即可

[root@mfyxw50 ~]#cd /opt/src/harbor
[root@mfyxw50 harbor]#docker-compose start

7.创建kubelet启动脚本

在mfyxw30.mfyxw.com主机上创建kubelet启动脚本

[root@mfyxw30 ~]#cat > /opt/kubernetes/server/bin/kubelet.sh << EOF
#!/bin/sh
./kubelet \\
  --anonymous-auth=false \\
  --cgroup-driver systemd \\
  --cluster-dns 172.16.0.2 \\
  --cluster-domain cluster.local \\
  --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice \\
  --fail-swap-on="false" \\
  --client-ca-file ./cert/ca.pem \\
  --tls-cert-file ./cert/kubelet.pem \\
  --tls-private-key-file ./cert/kubelet-key.pem \\
  --hostname-override mfyxw30.mfyxw.com \\
  --image-gc-high-threshold 20 \\
  --image-gc-low-threshold 10 \\
  --kubeconfig /opt/kubernetes/server/conf/kubelet.kubeconfig \\
  --log-dir /data/logs/kubernetes/kube-kubelet \\
  --pod-infra-container-image harbor.od.com/public/pause:latest \\
  --root-dir /data/kubelet
EOF

在mfyxw40.mfyxw.com主机上创建kubelet启动脚本

[root@mfyxw40 ~]#cat > /opt/kubernetes/server/bin/kubelet.sh << EOF
#!/bin/sh
./kubelet \\
  --anonymous-auth=false \\
  --cgroup-driver systemd \\
  --cluster-dns 172.16.0.2 \\
  --cluster-domain cluster.local \\
  --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice \\
  --fail-swap-on="false" \\
  --client-ca-file ./cert/ca.pem \\
  --tls-cert-file ./cert/kubelet.pem \\
  --tls-private-key-file ./cert/kubelet-key.pem \\
  --hostname-override mfyxw40.mfyxw.com \\
  --image-gc-high-threshold 20 \\
  --image-gc-low-threshold 10 \\
  --kubeconfig /opt/kubernetes/server/conf/kubelet.kubeconfig \\
  --log-dir /data/logs/kubernetes/kube-kubelet \\
  --pod-infra-container-image harbor.od.com/public/pause:latest \\
  --root-dir /data/kubelet
EOF

8.调整权限和目录

在mfyxw30.mfyxw.com主机上调整kubelet.sh的权限并创建/data/logs/kubernetes/kube-apiserver目录

[root@mfyxw30 ~]#chmod +x /opt/kubernetes/server/bin/kubelet.sh
[root@mfyxw30 ~]#mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet

在mfyxw40.mfyxw.com主机上调整kubelet.sh的权限并创建/data/logs/kubernetes/kube-apiserver目录

[root@mfyxw40 ~]#chmod +x /opt/kubernetes/server/bin/kubelet.sh
[root@mfyxw40 ~]#mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet

9.给kubelet创建软链接和目录

#分别在mfyxw30.mfyxw.com和mfyxw40.mfyxw.com主机上创建kubelet的软链接,图片以mfyxw30主机为例
[root@mfyxw30 ~]# ln -s /opt/kubernetes/server/bin/kubelet /usr/bin/kubelet

[root@mfyxw40 ~]# ln -s /opt/kubernetes/server/bin/kubelet /usr/bin/kubelet

10.为kubelet创建supervisor配置文件

在mfyxw30.mfyxw.com主机上为kubelet创建supervisor配置文件

[root@mfyxw30 ~]#cat > /etc/supervisord.d/kube-kubelet.ini << EOF
[program:kube-kubelet-80-30]
command=/opt/kubernetes/server/bin/kubelet.sh                            ; the program (relative uses PATH, can take args)
numprocs=1                                                               ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                     ; directory to cwd to before exec (def no cwd)
autostart=true                                                           ; start at supervisord start (default: true)
autorestart=true                                                         ; retstart at unexpected quit (default: true)
startsecs=30                                                            ; number of secs prog must stay running (def. 1)
startretries=3                                                           ; max # of serial start failures (default 3)
exitcodes=0,2                                                            ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                          ; signal used to kill process (default TERM)
stopwaitsecs=10                                                          ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                                ; setuid to this UNIX account to run the program
redirect_stderr=false                                                    ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                             ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                                 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                              ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                              ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stderr.log ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                             ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                                 ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB                                              ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false                                              ; emit events on stderr writes (default false)
EOF

在mfyxw40.mfyxw.com主机上为kubelet创建supervisor配置文件

[root@mfyxw40 ~]#cat > /etc/supervisord.d/kube-kubelet.ini << EOF
[program:kube-kubelet-80-40]
command=/opt/kubernetes/server/bin/kubelet.sh                            ; the program (relative uses PATH, can take args)
numprocs=1                                                               ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                     ; directory to cwd to before exec (def no cwd)
autostart=true                                                           ; start at supervisord start (default: true)
autorestart=true                                                         ; retstart at unexpected quit (default: true)
startsecs=30                                                            ; number of secs prog must stay running (def. 1)
startretries=3                                                           ; max # of serial start failures (default 3)
exitcodes=0,2                                                            ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                          ; signal used to kill process (default TERM)
stopwaitsecs=10                                                          ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                                ; setuid to this UNIX account to run the program
redirect_stderr=false                                                    ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                             ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                                 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                              ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                              ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stderr.log ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                             ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                                 ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB                                              ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false                                              ; emit events on stderr writes (default false)
EOF

11.启动服务并检查

在mfyxw30.mfyxw.com主机上启动服务并检查

[root@mfyxw30 ~]#supervisorctl update
[root@mfyxw30 ~]#supervisorctl status

在mfyxw40.mfyxw.com主机上启动服务并检查

[root@mfyxw40 ~]#supervisorctl update
[root@mfyxw40 ~]#supervisorctl status

12.查看Node节点状态

在mfyxw30.mfyxw.com主机上操作

[root@mfyxw30 ~]#kubectl get nodes

在mfyxw40.mfyxw.com主机上操作

[root@mfyxw40 ~]#kubectl get nodes

鉴于在使用kubectl get nodes命令查出的结果中ROLES上显示，正常使用kubeadm会显示master或None，故在此修改下标签

在mfyxw30.mfyxw.com主机上操作

[root@mfyxw30 ~]# kubectl label node mfyxw30.mfyxw.com node-role.kubernetes.io/master=
[root@mfyxw30 ~]# kubectl label node mfyxw30.mfyxw.com node-role.kubernetes.io/node=
[root@mfyxw30 ~]# kubectl get nodes

在mfyxw40.mfyxw.com主机上操作

[root@mfyxw40 ~]# kubectl label node mfyxw30.mfyxw.com node-role.kubernetes.io/master=
[root@mfyxw40 ~]# kubectl label node mfyxw30.mfyxw.com node-role.kubernetes.io/node=
[root@mfyxw40 ~]# kubectl get nodes