kubernetes：v1.25 + containerd

由于kubernets从v1.24开始停止支持dockershim，kubernets不再支持通过docker来创建和管理容器。本文记录安装kubernetes v1.25 + containerd

1、环境

IP	主机名	节点角色	操作系统
10.0.8.101	master01,k8s-master01,k8s-api.ilinux.io	master	Ubuntu 22.04.1 LTS
10.0.8.111	node01,k8s-node01	node	Ubuntu 22.04.1 LTS
10.0.8.112	node02,k8s-node2	node	Ubuntu 22.04.1 LTS
10.0.8.113	node03,k8s-node03	node	Ubuntu 22.04.1 LTS

2、时区调整

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

3、配置镜像源 这里是有aliyun

sed -i 's@http://cn.archive.ubuntu.com@https://mirrors.aliyun.com@g' /etc/apt/sources.list
apt-get update

网卡配置：
cat /etc/netplan/00-installer-config.yaml
network:
  ethernets:
    ens33:
      dhcp4: false
      addresses: [192.168.3.101/24]
      nameservers:
        addresses: [192.168.3.1]
      gateway4: 192.168.3.1
  ethernets:
    ens37:
      dhcp4: false
      addresses: [10.0.8.101/24]
  version: 2

4、配置域名

#cat /etc/hosts
10.0.8.101 master01 k8s-master01 k8s-api.ilinux.io
10.0.8.111 node01 k8s-node01
10.0.8.112 node02 k8s-node02
10.0.8.113 node03 k8s-node03

5、关闭防火墙

#systemctl disable --now ufw
#systemctl stop ufw

6、关闭swap

swapoff -a # 临时swap
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久关闭swap

7、优化资源限制参数

ulimit -SHn 65535

cat >>/etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 131072
* soft nproc 65535
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

8、安装ipvs等工具并优化内存

apt -y install ipvsadm ipset sysstat conntrack libseccomp-dev

　加载ipvs模块以及配置：

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

cat >> /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

systemctl restart systemd-modules-load.service

9、内核参数调整

cat > /etc/sysctl.d/k8s.conf <<EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

sysctl --system

10、配置containerd需要的模块

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

加载模块：
modprobe -- overlay
modprobe -- br_netfilter

配置Containerd所需的内核：
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

加载内核：
sysctl --system

11、安装containerd

# step 1: 安装必要的一些系统工具
sudo apt-get update
sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common
# step 2: 安装GPG证书
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
# Step 3: 写入软件源信息
sudo add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
# Step 4: 更新并安装containerd.io
sudo apt-get -y update
sudo apt-get -y install containerd.io critcl

12、配置containerd

生成配置文件：containerd config default > /etc/containerd/config.toml
修改配置文件
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
grep 'SystemdCgroup' -B 11 /etc/containerd/config.toml   #检查是否修改成功

将sandbox_image镜像地址更改为阿里云：
sed -i 's@k8s.gcr.io@registry.aliyuncs.com/google_containers@g' /etc/containerd/config.toml
grep 'sandbox_image' /etc/containerd/config.toml 

镜像加速：
配置文件：/etc/containerd/config.toml，增加配置路径
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

创建加速配置文件
如：docker.io
mkdir -p /etc/containerd/certs.d/docker.io
配置文件：hosts.toml
cat /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://docker.io"
[host."http://hub-mirror.c.163.com"]
  capabilities = ["pull","resolve"]

#增加私有仓库：
如：harbor.myland.com
mkdir -p /etc/containerd/certs.d/harbor.myland.com

#配置文件：hosts.toml,如果不需要验证服务端证书，skip_verify 设置为 true即可，ca.crt为签发harbor的server.crt的 CA的证书
cat /etc/containerd/certs.d/harbor.myland.com/hosts.toml
server = "https://harbor.myland.com"
[host."https://harbor.myland.com"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = false
  ca = "/etc/containerd/certs.d/harbor.myland.com/ca.crt"

systemctl daemon-reload && systemctl enable --now containerd
systemctl restart containerd

#可以安装nerdctl工具管理containerd，改命令风格与docker命令相似
https://github.com/containerd/nerdctl

13、安装kubelet kubeadm kubectl

apt update && apt install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
echo "deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list

apt update -y
apt install -y kubelet kubeadm kubectl

14、kubernetes初始化　

kubeadm init \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.25.2 \
--control-plane-endpoint k8s-api.ilinux.io \
--apiserver-advertise-address 10.0.8.101 \
--pod-network-cidr 10.201.0.0/16 \
--service-cidr 10.97.0.0/16 \
--token-ttl 0　

13、验证

查看k8s集群

ctr查看containerd

nerdctl查看

netctl检查containerd配置，与docker一致：如检查PID

也可以设置默认的nerdctl使用的namespace

mkdir /etc/nerdctl && cat  'namespace = "k8s.io"' >/ etc/nerdctl/nerdctl.toml