k8s错误集合

1、etcd没有启动的

[root@mcwk8s03 ~]# kubectl get nodes
Unable to connect to the server: context deadline exceeded

启动之后就可以使用了

[root@mcwk8s03 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
mcwk8s05 NotReady <none> 404d v1.15.12
mcwk8s06 NotReady <none> 404d v1.15.12
[root@mcwk8s03 ~]#

2、pod一直重启。首先排查oom,上次重启原因

Containers:
  fab-browser-api:
    Container ID:  docker://9xx
    Image:         roc.xx79
    Image ID:      docxxd
    Port:          9090/TCP
    Host Port:     0/TCP
    Command:
      java
      -Xms1G
      -Xmx2G
      -XX:MetaspaceSize=64M
      -XX:MaxMetaspaceSize=128M
      -Xss256K
      -XX:+UseConcMarkSweepGC
      -XX:CMSFullGCsBeforeCompaction=5
      -XX:+UseCMSCompactAtFullCollection
      -XX:CMSInitiatingOccupancyFraction=80
      -XX:+HeapDumpOnOutOfMemoryError
      -XX:HeapDumpPath=./logs/
      -DserverName=fxx-api
      -jar
      /app.jar
    State:          Running
      Started:      Mon, 18 Dec 2023 10:12:11 +0800
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 18 Dec 2023 04:09:38 +0800
      Finished:     Mon, 18 Dec 2023 10:12:09 +0800
    Ready:          True
    Restart Count:  36
    Limits:
      cpu:     1
      memory:  2000Mi
    Requests:
      cpu:      300m
      memory:   800Mi
    Liveness:   http-get http://:9090/argus/health delay=120s timeout=1s period=20s #success=1 #failure=6
    Readiness:  http-get http://:9090/argus/health delay=120s timeout=1s period=20s #success=1 #failure=6

3、k8s排查问题思路

https://zhuanlan.zhihu.com/p/421693641

https://zhuanlan.zhihu.com/p/651299187

https://blog.csdn.net/weixin_45727359/article/details/128024686

4、有些服务启动慢，默认的健康检查时间需要调整长一些，等服务启动之后再健康检查

现象：发布应用，新起的pod，一直没有启动起来，反复重启。

kubectl describe pod ,查看上次报错，并且事件里提示健康检查没有通过

    State:          Running
      Started:      Tue, 09 Jan 2024 10:32:56 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Tue, 09 Jan 2024 10:29:03 +0800
      Finished:     Tue, 09 Jan 2024 10:32:55 +0800
    Ready:          True
    Restart Count:  1

Warning  Unhealthy  6m30s (x22 over 10m)  kubelet, qa-kube003.xx.x.com  Readiness probe failed: 
Get http://10.96.x.x:9090/argus/health: dial tcp 10.x.x.x:9090: connect: connection refused

解决方法：容器的健康检查失败，容器多次重启，偶尔才起来。这次这个是服务启动需要3分钟多，而健康检查200s的时候就开始了。服务还没正常启动起来就检测，导致探测失败。将探测时间延长到250s之后，再次发布，就没有发生重启的现象了，直接就好了

kubectl edit deploy ai-xxl-deploy

复制配置，然后在服务发布里面添加上配置，并且将时间改为250s,发布应用