kubernetes排错系列：（二）、运行很久的kubernetes集群，创建出来的pod都是pending状态

1.查看pod信息

# 查看pod 报错信息
kubectl get pods
发现pod的ip没有 生成，也没有分配到某个node节点

# 查看pod详细时间
kubectl describe pods
发现pod事件为空

2.查看集群信息

kubectl get nodes

发现集群状态正常

kubectl cluster-info

Kubernetes master is running at https://xx.xx.55.113
KubeDNS is running at https://xx.xx.55.113:6443/api

3. 查看kubelet日志发现

journalctl -xefu kubelet

Sep  :: cbov10-sso55- kubelet[]: E0917 ::34.750672     kuberuntime_manager.go:] getPodContainerStatuses for pod "kube-scheduler-k8s-master_kube-system(a29556bf1d34f898bf5d0ce3c15a5948)" failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/s=13b9f39777ba4419b309b8b3ae11760e;i=121b8;b=37ab1d18c20d41b2a88be512b10aba22;m=559ecd: no such file or directory
Sep  :: cbov10-sso55- kubelet[]: E0917 ::35.260404     remote_runtime.go:] ContainerStatus "fad2baa3755429e94c4b79549ce2a2d2a2b018e1c55bf282d1b5b0be0b4022fe" from runtime service failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/s=13b9f39777ba4419b309b8b3ae11760e;i=121b8;b=37ab1d18c20d41b2a88be512b10aba22;m=559ecd: no such file or directory
Sep  :: cbov10-sso55- kubelet[]: E0917 ::35.260498     kuberuntime_gc.go:] Failed to remove container "fad2baa3755429e94c4b79549ce2a2d2a2b018e1c55bf282d1b5b0be0b4022fe": failed to get container status "fad2baa3755429e94c4b79549ce2a2d2a2b018e1c55bf282d1b5b0be0b4022fe": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/s=13b9f39777ba4419b309b8b3ae11760e;i=121b8;b=37ab1d18c20d41b2a88be512b10aba22;m=559ecd: no such file or directory
Sep  :: cbov10-sso55- kubelet[]: E0917 ::44.674953     pod_workers.go:] Error syncing pod a29556bf1d34f898bf5d0ce3c15a5948 ("kube-scheduler-cbov10-sso55-113_kube-system(a29556bf1d34f898bf5d0ce3c15a5948)"), skipping: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/s=13b9f39777ba4419b309b8b3ae11760e;i=121b8;b=37ab1d18c20d41b2a88be512b10aba22;m=559ecd: no such file or directory

4.清理docker占用的磁盘空间

#可以用于清理磁盘，删除关闭的容器、无用的数据卷和网络
docker system prune -a

5.之后再次部署，发现pod能正常启动。