k8s错误集合

1、etcd没有启动的

[root@mcwk8s03 ~]# kubectl get nodes
Unable to connect to the server: context deadline exceeded

启动之后就可以使用了

[root@mcwk8s03 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
mcwk8s05 NotReady <none> 404d v1.15.12
mcwk8s06 NotReady <none> 404d v1.15.12
[root@mcwk8s03 ~]#

2、pod一直重启。首先排查oom,上次重启原因

Containers:

  fab-browser-api:

    Container ID:  docker://9xx

    Image:         roc.xx79

    Image ID:      docxxd

    Port:          9090/TCP

    Host Port:     0/TCP

    Command:

      java

      -Xms1G

      -Xmx2G

      -XX:MetaspaceSize=64M

      -XX:MaxMetaspaceSize=128M

      -Xss256K

      -XX:+UseConcMarkSweepGC

      -XX:CMSFullGCsBeforeCompaction=5

      -XX:+UseCMSCompactAtFullCollection

      -XX:CMSInitiatingOccupancyFraction=80

      -XX:+HeapDumpOnOutOfMemoryError

      -XX:HeapDumpPath=./logs/

      -DserverName=fxx-api

      -jar

      /app.jar

    State:          Running

      Started:      Mon, 18 Dec 2023 10:12:11 +0800

    Last State:     Terminated

      Reason:       OOMKilled

      Exit Code:    137

      Started:      Mon, 18 Dec 2023 04:09:38 +0800

      Finished:     Mon, 18 Dec 2023 10:12:09 +0800

    Ready:          True

    Restart Count:  36

    Limits:

      cpu:     1

      memory:  2000Mi

    Requests:

      cpu:      300m

      memory:   800Mi

    Liveness:   http-get http://:9090/argus/health delay=120s timeout=1s period=20s #success=1 #failure=6

    Readiness:  http-get http://:9090/argus/health delay=120s timeout=1s period=20s #success=1 #failure=6

3、k8s排查问题思路

https://zhuanlan.zhihu.com/p/421693641

https://zhuanlan.zhihu.com/p/651299187

https://blog.csdn.net/weixin_45727359/article/details/128024686

4、有些服务启动慢，默认的健康检查时间需要调整长一些，等服务启动之后再健康检查

现象：发布应用，新起的pod，一直没有启动起来，反复重启。

kubectl describe pod ,查看上次报错，并且事件里提示健康检查没有通过

    State:          Running

      Started:      Tue, 09 Jan 2024 10:32:56 +0800

    Last State:     Terminated

      Reason:       Error

      Exit Code:    143

      Started:      Tue, 09 Jan 2024 10:29:03 +0800

      Finished:     Tue, 09 Jan 2024 10:32:55 +0800

    Ready:          True

    Restart Count:  1

Warning  Unhealthy  6m30s (x22 over 10m)  kubelet, qa-kube003.xx.x.com  Readiness probe failed: 
Get http://10.96.x.x:9090/argus/health: dial tcp 10.x.x.x:9090: connect: connection refused

解决方法：容器的健康检查失败，容器多次重启，偶尔才起来。这次这个是服务启动需要3分钟多，而健康检查200s的时候就开始了。服务还没正常启动起来就检测，导致探测失败。将探测时间延长到250s之后，再次发布，就没有发生重启的现象了，直接就好了

kubectl edit deploy ai-xxl-deploy

复制配置，然后在服务发布里面添加上配置，并且将时间改为250s,发布应用