k8s集群部署初体验

██ 环境准备【所有节点】
██ 安装Docker/kubeadm/kubelet【所有节点】
██ 部署 k8s master
██ 部署 k8s node
██ 部署网络插件【CNI】
██ 以下是node6的操作日志，可以看到网络插件部署、初始化、成功的全过程。

██ 环境准备【所有节点】

■ 关闭防火墙、selinux

systemctl stop firewalld

setenforce 0

■ 关闭 swap

swapoff -a

fstab 注释掉 swap 的自动挂载

确认 swap 为0

free -m

■ 设置主机名

hostnamectl set-hostname

在master添加hosts：

cat >> /etc/hosts << EOF

192.168.222.21 node5

192.168.222.22 node6

192.168.222.23 node7

EOF

■ 将桥接的IPv4流量传递到iptables的链：

cat > /etc/sysctl.d/k8s.conf << EOF

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

EOF

sysctl --system

■ 设置时区、时间同步等

██ 安装Docker/kubeadm/kubelet【所有节点】

■ 修改系统自带的源镜像地址为阿里云地址

备份源镜像地址

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

改为阿里云地址

wget http://mirrors.aliyun.com/repo/Centos-7.repo -O /etc/yum.repos.d/CentOS-Base.repo

■ 安装Docker

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

yum -y install docker-ce-18.06.1.ce-3.el7

systemctl enable docker && systemctl start docker

docker --version

查看可用的版本，安装最新版，安装时间2022-05-30

yum list docker-ce --showduplicates

yum -y remove docker-ce-18.06.1.ce-3.el7

yum -y install docker-ce-20.10.16-3.el7

yum -y remove docker-ce-20.10.16-3.el7

yum -y install docker-ce-19.03.15-3.el7

■ 配置镜像仓库地址

cat > /etc/docker/daemon.json << EOF

{

"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],

"exec-opts":["native.cgroupdriver=systemd"]

}

EOF

■ 重启docker

systemctl status docker

systemctl daemon-reload

systemctl restart docker

systemctl enable docker

■ 添加 k8s yum 源

cat > /etc/yum.repos.d/kubernetes.repo << EOF

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64

enabled=1

gpgcheck=0

repo_gpgcheck=0

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

■ 安装 kubelet,kubeadm,kubectl

yum list kubelet --showduplicates

yum list kubeadm --showduplicates

yum list kubectl --showduplicates

yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0

yum remove -y kubelet kubeadm kubectl

yum install -y kubelet-1.20.1 kubeadm-1.20.1 kubectl-1.20.1

yum install -y kubelet-1.23.1 kubeadm-1.23.1 kubectl-1.23.1

systemctl status kubelet

systemctl enable kubelet

systemctl start kubelet

TIPS：k8s集群还未拉起，故这里的kubelet是无法启动的，等master初始化时会自动拉起

██ 部署 k8s master

kubeadm init

--apiserver-advertise-address=192.168.222.21

--image-repository registry.aliyuncs.com/google_containers

--kubernetes-version v1.23.1

--service-cidr=10.92.0.0/12

--pod-network-cidr=10.220.0.0/16

--ignore-preflight-errors=all

初始化成功会输出类似如下信息：

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube

  cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

  chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.222.21:6443 --token plfz4s.m574ak0ryby28q6o \

  --discovery-token-ca-cert-hash sha256:ea7fe9e638b97215c3f656c4cb7988ef876a9a69217b7663ef33680e414df6e5 \

  --ignore-preflight-errors=all

使用kubectl命令查看状态:

kubectl get nodes

node节点使用 kubectl 命令，需用scp命令分别拷贝config文件至对应目录方可:

mkdir -p $HOME/.kube

scp node5:/etc/kubernetes/admin.conf $HOME/.kube/config

chown $(id -u)(id -g) $HOME/.kube/config

██ 部署 k8s node

在node节点中执行 master init 后产生的加入命令：

【如上】

TIPS：这里加入node后会看到node状态为NotReady，是因为没有安装CNI，kubelet无法通过网络给apiserver上报node状态，安装CNI后即可恢复

kubectl get nodes

NAME    STATUS     ROLES    AGE     VERSION

node5   NotReady   master   16h     v1.18.0

node6   NotReady   <none>   6m55s   v1.18.0

node7   NotReady   <none>   42s     v1.18.0

██ 部署网络插件【CNI】

wget https://docs.projectcalico.org/manifests/calico.yaml --no-check-certificate

下载完成后，修改、定义Pod网络：CALICO_IPV4POOL_CIDR, 需与前面master初始化时配置一样（参数 pod-network-cidr）

- name: CALICO_IPV4POOL_CIDR

value: "192.168.0.0/16"

修改完成后应用报错如下：

kubectl apply -f calico.yaml

报错如下：【2022-05-28】

[root@node6:0 ~]# kubectl apply -f calico.yaml

error: unable to recognize "calico.yaml": no matches for kind "PodDisruptionBudget" in version "policy/v1"

可见网络插件无法部署，看上去是calico版本问题导致，后续根据错误日志，反复测试不同版本，最终确认docker版本19.03.15与kubelet版本1.23.1是匹配的，可以顺利配置calico网络接口。

应用成功后，可以看到CNI POD正在初始化中，静待拉起~~~

calico running 状态后，查看节点已全部 Ready

██ 以下是node6的操作日志，可以看到网络插件部署、初始化、成功的全过程。

[root@node6:0 ~]# kubectl apply -f calico.yaml

configmap/calico-config created

customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created

clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created

clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created

clusterrole.rbac.authorization.k8s.io/calico-node created

clusterrolebinding.rbac.authorization.k8s.io/calico-node created

daemonset.apps/calico-node created

serviceaccount/calico-node created

deployment.apps/calico-kube-controllers created

serviceaccount/calico-kube-controllers created

poddisruptionbudget.policy/calico-kube-controllers created

[root@node6:0 ~]# kubectl get nodes

NAME    STATUS     ROLES                  AGE     VERSION

node5   NotReady   control-plane,master   5m50s   v1.23.1

node6   NotReady   <none>                 3m43s   v1.23.1

[root@node6:0 ~]# kubectl get pods -n kube-system

NAME                                      READY   STATUS     RESTARTS   AGE

calico-kube-controllers-6b77fff45-cwtd2   0/1     Pending    0          61s

calico-node-5nw2r                         0/1     Init:0/2   0          61s

calico-node-fl272                         0/1     Init:0/2   0          61s

coredns-6d8c4cb4d-fwxk2                   0/1     Pending    0          6m17s

coredns-6d8c4cb4d-ql29c                   0/1     Pending    0          6m17s

etcd-node5                                1/1     Running    0          6m30s

kube-apiserver-node5                      1/1     Running    0          6m32s

kube-controller-manager-node5             1/1     Running    0          6m31s

kube-proxy-gzdfk                          1/1     Running    0          6m17s

kube-proxy-k8vmn                          1/1     Running    0          4m27s

kube-scheduler-node5                      1/1     Running    0          6m30s

[root@node6:0 ~]# kubectl get pods -n kube-system

NAME                                      READY   STATUS              RESTARTS   AGE

calico-kube-controllers-6b77fff45-cwtd2   0/1     ContainerCreating   0          3m48s

calico-node-5nw2r                         0/1     PodInitializing     0          3m48s

calico-node-fl272                         0/1     PodInitializing     0          3m48s

coredns-6d8c4cb4d-fwxk2                   0/1     ContainerCreating   0          9m4s

coredns-6d8c4cb4d-ql29c                   0/1     ContainerCreating   0          9m4s

etcd-node5                                1/1     Running             0          9m17s

kube-apiserver-node5                      1/1     Running             0          9m19s

kube-controller-manager-node5             1/1     Running             0          9m18s

kube-proxy-gzdfk                          1/1     Running             0          9m4s

kube-proxy-k8vmn                          1/1     Running             0          7m14s

kube-scheduler-node5                      1/1     Running             0          9m17s

[root@node6:0 ~]# kubectl get pods -n kube-system

NAME                                      READY   STATUS    RESTARTS   AGE

calico-kube-controllers-6b77fff45-cwtd2   1/1     Running   0          9m48s

calico-node-5nw2r                         1/1     Running   0          9m48s

calico-node-fl272                         1/1     Running   0          9m48s

coredns-6d8c4cb4d-fwxk2                   1/1     Running   0          15m

coredns-6d8c4cb4d-ql29c                   1/1     Running   0          15m

etcd-node5                                1/1     Running   0          15m

kube-apiserver-node5                      1/1     Running   0          15m

kube-controller-manager-node5             1/1     Running   0          15m

kube-proxy-gzdfk                          1/1     Running   0          15m

kube-proxy-k8vmn                          1/1     Running   0          13m

kube-scheduler-node5                      1/1     Running   0          15m

[root@node6:0 ~]# kubectl get nodes

NAME    STATUS   ROLES                  AGE   VERSION

node5   Ready    control-plane,master   17m   v1.23.1

node6   Ready    <none>                 15m   v1.23.1