k8s学习笔记（一）

你将学到什么

• 如何部署k8s集群

网络拓扑

主机名	网络地址	角色
study	92.0.0.50(内网) 192.168.203.250/19(外网)	Master
fnode	92.0.0.16(内网) 192.168.203.216/19(外网)	Node
gnode	92.0.0.17(内网) 192.168.203.217/19(外网)	Node

配置代理

安装shadowsocks

# yum install python-pip -y
# pip install shadowsocks

创建shadowsocks配置文件

# tee /etc/shadowsocks.json << EOF
{
"server":"45.78.XX.XX",
"server_port": XXXX,
"local_port": 1080,
"password":"XXXX",
"timeout":600,
"method":"rc4-md5"
}
EOF

启动shadowsocks客户端

# sslocal -c /etc/shadowsocks.json -d start
# ss -lntup|grep sslocal

所有节点配置

安装docker

# tee /etc/yum.repos.d/docker.repo <<-'EOF'
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF
# yum makecache fast
# yum install -y docker-engine-1.12.0
# mkdir -p /etc/systemd/system/docker.service.d
# tee /etc/systemd/system/docker.service.d/https-proxy.conf << EOF
[Service]
Environment="ALL_PROXY=socks5://127.0.0.1:1080/" "NO_PROXY=localhost,127.0.0.1,docker.io,yanzhe919.mirror.aliyuncs.com,99nkhzdo.mirror.aliyuncs.com,*.aliyuncs.com,*.mirror.aliyuncs.com,registry.docker-cn.com,hub.c.163.com,hub-auth.c.163.com,"
EOF
# systemctl daemon-reload
# systemctl restart docker
### 测试下代理是否生效
# docker pull k8s.gcr.io/kube-apiserver-amd64:v1.10.3

安装kubeadm、kubelet 和 kubectl

# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
proxy=socks5://127.0.0.1:1080
EOF
# setenforce 0
# yum install -y kubelet kubeadm kubectl
# cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
# sysctl --system
# swapoff -a && sed -i '/swap/d' /etc/fstab
# systemctl enable kubelet && systemctl start kubelet

配置kubelet的cgroup驱动

# docker info | grep -i cgroup
# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
### 如果不相同
# sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# systemctl daemon-reload
# systemctl restart kubelet

Master节点配置

初始化Master节点

### 我们后面要创建Flannel网络，所以必须设置--pod-network-cidr，注意记住下面的输出，这个是用来添加其他node节点到集群的命令
# kubeadm init --apiserver-advertise-address=92.0.0.50 --pod-network-cidr=10.244.0.0/16
You can now join any number of machines by running the following on each node
as root:

   kubeadm join 92.0.0.50:6443 --token ro2pkh.3myahmoituohkbs7 --discovery-token-ca-cert-hash sha256:cbdfe4bf3cc4b15b0f47f3a8913638b0d3401de84c0e8cd21b647950b9e015fc

### 如果想用非root用户运行kubectl，则执行，否则跳过
# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config

### 如果已经是root用户
# export KUBECONFIG=/etc/kubernetes/admin.conf

安装pod网络

# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

设置允许Pod调度到Master节点

# kubectl describe node k8s |grep Taints
Taints:             node-role.kubernetes.io/master:NoSchedule
# kubectl taint nodes --all node-role.kubernetes.io/master-
node "k8s" untainted
# kubectl describe node k8s |grep Taints
Taints:             <none>

Node节点配置

# kubeadm join 92.0.0.50:6443 --token ro2pkh.3myahmoituohkbs7 --discovery-token-ca-cert-hash sha256:cbdfe4bf3cc4b15b0f47f3a8913638b0d3401de84c0e8cd21b647950b9e015fc

问题处理

初始化报错

### 报错一（屏幕输出日志）
[preflight] Some fatal errors occurred:
    [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
    [ERROR Swap]: running with swap on is not supported. Please disable swap

### 解决办法
# tee -a /etc/sysctl.conf << EOF
net.bridge.bridge-nf-call-iptables = 1
EOF
# sysctl -p
# swapoff -a && sed -i '/swap/d' /etc/fstab

### 报错二（/var/log/message）
May 29 13:44:12 k8s kubelet: F0529 13:44:12.807374   23932 server.go:233] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

### 解决办法（降低docker版本为1.12.0）
# docker info |grep -i cgroup
Cgroup Driver: cgroupfs
# vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
# systemctl daemon-reload
# systemctl restart kubelet

kubectl命令执行失败

The connection to the server 172.16.196.2:8443 was refused - did you specify the right host or port?

### 解决办法
# export KUBECONFIG=/etc/kubernetes/admin.conf

kubelet服务告警

### systemctl status kubelet告警
cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 29 06:30:28 fnode kubelet[4136]: E0529 06:30:28.935309    4136 kubelet.go:2130] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

### 解决办法
### 删除 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 的 KUBELET_NETWORK_ARGS，然后重启kubelet服务

参考文档

Installing kubeadm
Troubleshooting kubeadm
Using kubeadm to Create a Cluster
Kubernetes集群之清除集群