kubernates的集群安装-kubadm

kubernates的集群安装-kubadm

环境准备工作（CentOS）

• 准备三台或以上的虚拟机
• 停用防火墙
  
  sudo systemctl stop firewalld

sudo systemctl disable firewalld
• 修改主机名（命名规则符合 DNS 标准）

cat <<EOF | sudo tee /etc/hostname
k8sv23-n03-p102
EOF
#立即生效
sudo hostnamectl set-hostname k8sv23-n12-p104

• 安装 docker（参照官网即可）

#移除历史
sudo yum remove docker \
                docker-client \
                docker-client-latest \
                docker-common \
                docker-latest \
                docker-latest-logrotate \
                docker-logrotate \
                docker-engine
# yum 添加仓库
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
#安装
sudo yum -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

• docker配置 cgroupdriver ,并启动

 cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "registry-mirrors": [
    "http://mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com",
     "https://uy35zvn6.mirror.aliyuncs.com",
     "https://cr.console.aliyun.com",
     "https://docker.mirrors.ustc.edu.cn",
        "https://registry.docker-cn.com",
     "http://mirror.azure.cn/"
   ],
   "exec-opts": ["native.cgroupdriver=systemd"],
   "log-driver": "json-file",
   "log-opts": { "max-size": "100m" },
   "storage-driver": "overlay2"
}
EOF
#启动并添加自启动
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker

• 配置网络模块

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1 # better than modify /etc/sysctl.conf
EOF

sudo sysctl --system

• 关闭分区

sudo swapoff -a
sudo sed -ri '/\sswap\s/s/^#?/#/' /etc/fstab

以上就是所有节点需要准备的工作。

安装 master 节点

本次安装k8s 版本为：1.23.17,该版本是运行时容器默认 docker，后面版本需要安装插件 cri-docker.

• 安装组件
  
  三个非容器化的组件 kubectl ，kubeadm,kubelet;具体含义看官网

#移除历史记录
yum remove -y kubelet kubeadm kubectl
#指定版本 1.23.17
sudo yum install -y kubelet-1.23.17 kubeadm-1.23.17 kubectl-1.23.17 --disableexcludes=kubernetes

• k8s 集群初始化（kubeadm init）

#开启 kubelet
systemctl enable kubelet.service
#安装 kubeadm init，apiserver-advertise-address参数是本机 ip：192.168.10.109 ，image-repository：镜像国内仓库，1.24.版本后仓库 CNCF 接管
#service-cidr：网段，pod-network-cidr
kubeadm init \
--apiserver-advertise-address=192.168.10.109 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.23.17 \
--service-cidr=10.10.0.0/16 \
--pod-network-cidr=10.244.0.0/16

如果安装拉取镜像失败，可以查看 k8s 需要的组件或插件镜像列表`kubeadm config images list --kubernetes-version v1.23.17`

#查看需要那些镜像，这些镜像都是在谷歌仓库，需要指向国内镜像源
[root@k8s ~]# kubeadm config images list --kubernetes-version v1.28.1
registry.k8s.io/kube-apiserver:v1.28.1
registry.k8s.io/kube-controller-manager:v1.28.1
registry.k8s.io/kube-scheduler:v1.28.1
registry.k8s.io/kube-proxy:v1.28.1
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.10.1

可配置国内yum源：

#如果 yum 安装失败，找不到包，配置 yum 阿里云源
cat <<EOF | sudo  tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
          https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装成功会输出结果：

#启动成功输出结果
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.109:6443 --token l85iz5.bg08rhr8x8gmhain \
        --discovery-token-ca-cert-hash sha256:4a1634f8c06e2c47cb06196df8ae59e6997cdfc9e08848ace5fb968c02b790a9

参照他给出的建议执行(复制管理配置文件)：

 mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

此时，检查集群

#检查集群情况 ，可以看到是只有一个 master节点，但是状态是NotReady
 [root@k8s ~]# kubectl get node
NAME         STATUS     ROLES                  AGE     VERSION
k8s.master   NotReady   control-plane,master   4m42s   v1.23.17

• 安装网络插件 Flannel(其他网络插件),需要去 github 下载kube-flannel.yml，并修改network 与集群 init 设置 --pod-network-cidr字段一致

#kubectl apply -f kube-flannel.yml
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
-------------------------------
#注意 kube-flannel.yml 文件网段Network修改成  --pod-network-cidr
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
-----------------------------    

#安装完成之后 节点就是 Ready 状态了
[root@k8s ~]# kubectl get node
NAME         STATUS   ROLES                  AGE   VERSION
k8s.master   Ready    control-plane,master   28m   v1.23.17  

安装 worer 节点

• 安装组件
  
  三个非容器化的组件 kubectl ，kubeadm,kubelet;具体含义看官网

#移除历史记录
yum remove -y kubelet kubeadm kubectl
#指定版本 1.23.17
sudo yum install -y kubelet-1.23.17 kubeadm-1.23.17 kubectl-1.23.17 --disableexcludes=kubernetes

• 节点加入集群 kubeadm join

#join加入 到主节点 ip,token有时效，可以在主节点生成：kubeadm token create，证书需要自己保存
kubeadm join 192.168.10.109:6443 --token l85iz5.bg08rhr8x8gmhain \
        --discovery-token-ca-cert-hash sha256:4a1634f8c06e2c47cb06196df8ae59e6997cdfc9e08848ace5fb968c02b790a9

• 验证

#在master节点检查节点信息
[root@k8s ~]# kubectl get nodes
NAME                 STATUS   ROLES                  AGE     VERSION
k8s.192.168.10.108   Ready    <none>                 8m17s   v1.23.17
k8s.master           Ready    control-plane,master   42m     v1.23.17

至此，k8s 既安装成功，后续添加节点 执行kubeadm join 即可

其他

• 想要其他虚拟机操作集群
  
  比如： 把kubectl，迁移到console服务器

#它只需要安装一个 kubectl，然后复制“config”文件就行，你可以直接在 Master 节点上用“scp”远程拷贝，例如
scp `which kubectl` chrono@192.168.10.208:~/
#或者：scp /usr/bin/kubectl root@192.168.10.120:/usr/bin/
scp ~/.kube/config chrono@192.168.10.208:~/.kube
#或者：scp ~/.kube/config  root@192.168.10.120:~/.kube

• 官网问题集 https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/

#重置节点
kubeadm reset
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

总结

1. kubeadm 是一个方便易用的 Kubernetes 工具，能够部署生产级别的 Kubernetes 集群。
2. 安装 Kubernetes 之前需要修改主机的配置，包括主机名、Docker 配置、网络设置、交换分区等。
3. Kubernetes 的组件镜像存放在 gcr.io，国内下载比较麻烦，需要国内镜像网站获取。指定国内镜像。
4. 安装 Master 节点需要使用命令 kubeadm init，安装 Worker 节点需要使用命令 kubeadm join，还要部署 Flannel 等网络插件才能让集群正常工作。
5. kubeadm join 命令有时效性，如果失效了 kubeadm token create 新创建一个

[root@k8s ~]# kubeadm token create --print-join-command
kubeadm join 192.168.10.109:6443 --token i26bzd.v8bctu5zzgrj8fs7 --discovery-token-ca-cert-hash sha256:349254d493aee98b96718848f8d72a01454fa717383fc69ccc27ad13fee38cb8
[root@k8s ~]#

其他操作

[root@localhost ~]# kubectl get nodes
NAME              STATUS     ROLES                  AGE   VERSION
k8s.master        Ready      control-plane,master   18h   v1.23.17
k8sv23-n03-p102   Ready      <none>                 15h   v1.23.17
k8sv23-n03-p115   Ready      <none>                 15h   v1.23.17
k8sv23-n07-124    NotReady   <none>                 15h   v1.23.17
k8sv23-n08-p132   Ready      <none>                 15h   v1.23.17
k8sv23-n10-p133   Ready      <none>                 15h   v1.23.17
k8sv23-n11-136    Ready      <none>                 15h   v1.23.17
k8sv23-n11-p108   Ready      <none>                 16h   v1.23.17

##查看守护 pod,有一个没有准备好
[root@localhost ~]# kubectl get daemonsets --all-namespaces
NAMESPACE      NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-flannel   kube-flannel-ds   8         8         7       8            7           <none>                   79m
kube-system    kube-proxy        8         8         8       8            8           kubernetes.io/os=linux   18h

##查看该 pod 详细信息
[root@localhost ~]# kubectl describe daemonset kube-flannel-ds -n kube-flannel
Name:           kube-flannel-ds
Selector:       app=flannel,k8s-app=flannel
Node-Selector:  <none>
Labels:         app=flannel
                k8s-app=flannel
                tier=node
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 8
Current Number of Nodes Scheduled: 8
Number of Nodes Scheduled with Up-to-date Pods: 8
Number of Nodes Scheduled with Available Pods: 7
Number of Nodes Misscheduled: 0
Pods Status:  7 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  	........

#查看所有 pod 有个 node flannel 镜像拉取失败Init:ImagePullBackOff
[root@localhost ~]# kubectl get pods --all-namespaces
NAMESPACE      NAME                                 READY   STATUS                  RESTARTS   AGE
kube-flannel   kube-flannel-ds-5mxvl                1/1     Running                 0          77m
kube-flannel   kube-flannel-ds-5wsz7                1/1     Running                 0          77m
kube-flannel   kube-flannel-ds-c4k2s                0/1     Init:ImagePullBackOff   0          77m
kube-flannel   kube-flannel-ds-h7c5d                1/1     Running                 0          77m
kube-flannel   kube-flannel-ds-j8tlh                1/1     Running                 0          77m
kube-flannel   kube-flannel-ds-kp9kn                1/1     Running                 0          77m
kube-flannel   kube-flannel-ds-l7g2v                1/1     Running                 0          77m
kube-flannel   kube-flannel-ds-tcklb                1/1     Running                 0          77m
kube-system    coredns-7bff545f9f-szb6n             1/1     Running                 0          53m
kube-system    coredns-7bff545f9f-vwp85             1/1     Running                 0          53m
kube-system    etcd-k8s.master                      1/1     Running                 0          18h
kube-system    kube-apiserver-k8s.master            1/1     Running                 0          18h
kube-system    kube-controller-manager-k8s.master   1/1     Running                 0          18h
kube-system    kube-proxy-24v9j                     1/1     Running                 0          15h
kube-system    kube-proxy-7gvqk                     1/1     Running                 0          15h
kube-system    kube-proxy-bl6zf                     1/1     Running                 0          15h
kube-system    kube-proxy-fwtkk                     1/1     Running                 0          18h
kube-system    kube-proxy-gqlqc                     1/1     Running                 0          16h
kube-system    kube-proxy-lxbcr                     1/1     Running                 0          15h
kube-system    kube-proxy-mphk5                     1/1     Running                 0          15h
kube-system    kube-proxy-wjfp6                     1/1     Running                 1          16h
kube-system    kube-scheduler-k8s.master            1/1     Running