K8S中POD节点状态ContainerCreating原因排查

现象：

# kubectl get pods -n kube-system |grep dashboard
kubernetes-dashboard-6685cb584f-dqkwk   0/1     ContainerCreating   0          13m

状态一直是ContainerCreating。

查看pod日志：

# kubectl describe pod kubernetes-dashboard-6685cb584f-dqkwk --namespace=kube-system
...
 Warning  FailedCreatePodSandBox  13m                     kubelet, k8s-220   Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fe44dea0870b79dbf2ae7d6de0eff1583c8ba99566775e70bfc44ca2088a74ce" network for pod "kubernetes-dashboard-6685cb584f-dqkwk": NetworkPlugin cni failed to set up pod "kubernetes-dashboard-6685cb584f-dqkwk_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.2.1/24
...

可以看到是网络问题。

原因是node之前反复添加过,添加之前需要清除下网络
node上执行：

# kubeadm reset
# systemctl stop kubelet
# systemctl stop docker
# rm -rf /var/lib/cni/ /var/lib/kubelet/* /etc/cni/
# ifconfig cni0 down
# ifconfig flannel.1 down
# ifconfig docker0 down
# ip link delete cni0
# ip link delete flannel.1
# systemctl start docker
# # kubeadm join xxx

重新加入后问题即可解决。

# kubectl get pods -n kube-system |grep dashboard
kubernetes-dashboard-6685cb584f-dqkwk   1/1     Running   0          14m

可以看到状态变为了Running。