Kubernetes 集群中 Ingress 故障的根因诊断

作者：scwang18，主要负责技术架构，在容器云方向颇有研究。

前言

KubeSphere 是青云开源的基于 Kubernetes 的云原生分布式操作系统，提供了比较炫酷的 Kubernetes 集群管理界面，我们团队用 KubeSphere 来作为开发平台。

本文记录了一次 KubeSphere 环境下的网络故障的解决过程。

现象

开发同学反馈自己搭建的 Harbor 仓库总是出问题，偶尔会报 net/http: TLS handshake timeout ，通过 curl 的方式访问 harbor.xxxx.cn ，也会随机频繁挂起。但是 ping 的反馈一切正常。

原因分析

接到错误报障后，经过了多轮分析，才最终定位到原因，应该是安装 KubeSphere 时，使用了最新版的 Kubernetes 1.23.1 。

虽然使用 ./kk version --show-supported-k8s 可以看到 KubeSphere 3.2.1 可以支持 Kubernetes 1.23.1 ，但实际上只是试验性支持，有坑的。

分析过程如下：

出现 Harbor registry 访问问题，下意识以为是 Harbor 部署有问题，但是在检查 Harbor core 的日志的时候，没有看到异常时有相应错误信息，甚至 info 级别的日志信息都没有。
又把目标放在 Harbor portal，查看访问日志，一样没有发现异常信息。
根据访问链，继续追查 kubesphere-router-kubesphere-system ，即 KubeSphere 版的 nginx ingress controller ，同样没有发现异常日志。
尝试在集群内其他 Pod 里访问 Harbor 的集群内 Service 地址，发现不会出现访问超时问题。初步判断是 KubeSphere 自带的 Ingress 的问题。
把 kubeSphere 自带的 Ingress Controller 关闭，安装 Kubernetes 官方推荐的 ingress-nginx-controller 版本，故障依旧，而且 Ingress 日志里也没有发现异常信息。
综合上面的分析，问题应该出现在客户端到 Ingress Controller 之间，我的 Ingress Controller 是通过 NodePort 方式暴露到集群外面。因此，测试其他通过 NodePort 暴露到集群外的 service，发现是一样的故障，至此，可以完全排除 Harbor 部署问题了，基本确定是客户端到 Ingress Controller 的问题。
外部客户端通过 NodePort 访问 Ingress Controller 时，会通过 kube-proxy 组件，分析 kube-proxy 的日志，发现告警信息

can’t set sysctl net/ipv4/vs/conn_reuse_mode, kernel version must be at least 4.1

这个告警信息是因为我的 centos 7.6 的内核版本过低，当前是 3.10.0-1160.21.1.el7.x86_64 ，与 Kubernetes 新版的 ipvs 存在兼容性问题。

可以通过升级操作系统的 kernel 版本可以解决。

升级完 kernel 后，Calico 启动不了，报以下错误信息

ipset v7.1: kernel and userspace incompatible: settype hash:ip,port with revision 6 not supported by userspace.

原因是安装 KubeSphere 时默认安装的 Calico 版本是 v3.20.0 , 这个版本不支持最新版的 Linux Kernel ，升级后的内核版本是 5.18.1-1.el7.elrepo.x86_64，calico 需要升级到 v3.23.0 以上版本。

升级完 Calico 版本后，Calico 继续报错

user "system:serviceaccount:kube-system:calico-node" cannot list resource "caliconodestatuses" in api group "crd.projectcalico.org"

还有另外一个错误信息，都是因为 clusterrole 的资源权限不足，可以通过修改 clusterrole 来解决问题。

至此，该莫名其妙的网络问题解决了。

解决过程

根据上面的分析，主要解决方案如下：

升级操作系统内核

使用阿里云的 yum 源

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

yum clean all && yum -y update

启用 elrepo 仓库

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm

安装最新版本内核

yum --enablerepo=elrepo-kernel install kernel-ml

查看系统上的所有可用内核

awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg

设置新的内核为 grub2 的默认版本

查看第4步返回的系统可用内核列表，不出意外第1个应该是最新安装的内核。

grub2-set-default 0

生成 grub 配置文件并重启

grub2-mkconfig -o /boot/grub2/grub.cfg

reboot now

验证

uname -r

升级 Calico

Kubernetes 上的 Calico 一般是使用 Daemonset 方式部署，我的集群里，Calico 的 Daemonset 名字是 calico-node。

直接输出为 yaml 文件，修改文件里的所有 image 版本号为最新版本 v3.23.1 。重新创建 Daemonset。

输出 yaml

kubectl -n kube-system get ds  calico-node -o yaml>calico-node.yaml

calico-node.yaml：

apiVersion: apps/v1

kind: DaemonSet

metadata:

  labels:

    k8s-app: calico-node

  name: calico-node

  namespace: kube-system

spec:

  revisionHistoryLimit: 10

  selector:

    matchLabels:

      k8s-app: calico-node

  template:

    metadata:

      creationTimestamp: null

      labels:

        k8s-app: calico-node

    spec:

      containers:

      - env:

        - name: DATASTORE_TYPE

          value: kubernetes

        - name: WAIT_FOR_DATASTORE

          value: "true"

        - name: NODENAME

          valueFrom:

            fieldRef:

              apiVersion: v1

              fieldPath: spec.nodeName

        - name: CALICO_NETWORKING_BACKEND

          valueFrom:

            configMapKeyRef:

              key: calico_backend

              name: calico-config

        - name: CLUSTER_TYPE

          value: k8s,bgp

        - name: NODEIP

          valueFrom:

            fieldRef:

              apiVersion: v1

              fieldPath: status.hostIP

        - name: IP_AUTODETECTION_METHOD

          value: can-reach=$(NODEIP)

        - name: IP

          value: autodetect

        - name: CALICO_IPV4POOL_IPIP

          value: Always

        - name: CALICO_IPV4POOL_VXLAN

          value: Never

        - name: FELIX_IPINIPMTU

          valueFrom:

            configMapKeyRef:

              key: veth_mtu

              name: calico-config

        - name: FELIX_VXLANMTU

          valueFrom:

            configMapKeyRef:

              key: veth_mtu

              name: calico-config

        - name: FELIX_WIREGUARDMTU

          valueFrom:

            configMapKeyRef:

              key: veth_mtu

              name: calico-config

        - name: CALICO_IPV4POOL_CIDR

          value: 10.233.64.0/18

        - name: CALICO_IPV4POOL_BLOCK_SIZE

          value: "24"

        - name: CALICO_DISABLE_FILE_LOGGING

          value: "true"

        - name: FELIX_DEFAULTENDPOINTTOHOSTACTION

          value: ACCEPT

        - name: FELIX_IPV6SUPPORT

          value: "false"

        - name: FELIX_HEALTHENABLED

          value: "true"

        envFrom:

        - configMapRef:

            name: kubernetes-services-endpoint

            optional: true

        image: calico/node:v3.23.1

        imagePullPolicy: IfNotPresent

        livenessProbe:

          exec:

            command:

            - /bin/calico-node

            - -felix-live

            - -bird-live

          failureThreshold: 6

          initialDelaySeconds: 10

          periodSeconds: 10

          successThreshold: 1

          timeoutSeconds: 10

        name: calico-node

        readinessProbe:

          exec:

            command:

            - /bin/calico-node

            - -felix-ready

            - -bird-ready

          failureThreshold: 3

          periodSeconds: 10

          successThreshold: 1

          timeoutSeconds: 10

        resources:

          requests:

            cpu: 250m

        securityContext:

          privileged: true

        terminationMessagePath: /dev/termination-log

        terminationMessagePolicy: File

        volumeMounts:

        - mountPath: /host/etc/cni/net.d

          name: cni-net-dir

        - mountPath: /lib/modules

          name: lib-modules

          readOnly: true

        - mountPath: /run/xtables.lock

          name: xtables-lock

        - mountPath: /var/run/calico

          name: var-run-calico

        - mountPath: /var/lib/calico

          name: var-lib-calico

        - mountPath: /var/run/nodeagent

          name: policysync

        - mountPath: /sys/fs/

          mountPropagation: Bidirectional

          name: sysfs

        - mountPath: /var/log/calico/cni

          name: cni-log-dir

          readOnly: true

      dnsPolicy: ClusterFirst

      hostNetwork: true

      initContainers:

      - command:

        - /opt/cni/bin/calico-ipam

        - -upgrade

        env:

        - name: KUBERNETES_NODE_NAME

          valueFrom:

            fieldRef:

              apiVersion: v1

              fieldPath: spec.nodeName

        - name: CALICO_NETWORKING_BACKEND

          valueFrom:

            configMapKeyRef:

              key: calico_backend

              name: calico-config

        envFrom:

        - configMapRef:

            name: kubernetes-services-endpoint

            optional: true

        image: calico/cni:v3.23.1

        imagePullPolicy: IfNotPresent

        name: upgrade-ipam

        resources: {}

        securityContext:

          privileged: true

        terminationMessagePath: /dev/termination-log

        terminationMessagePolicy: File

        volumeMounts:

        - mountPath: /var/lib/cni/networks

          name: host-local-net-dir

        - mountPath: /host/opt/cni/bin

          name: cni-bin-dir

      - command:

        - /opt/cni/bin/install

        env:

        - name: CNI_CONF_NAME

          value: 10-calico.conflist

        - name: CNI_NETWORK_CONFIG

          valueFrom:

            configMapKeyRef:

              key: cni_network_config

              name: calico-config

        - name: KUBERNETES_NODE_NAME

          valueFrom:

            fieldRef:

              apiVersion: v1

              fieldPath: spec.nodeName

        - name: CNI_MTU

          valueFrom:

            configMapKeyRef:

              key: veth_mtu

              name: calico-config

        - name: SLEEP

          value: "false"

        envFrom:

        - configMapRef:

            name: kubernetes-services-endpoint

            optional: true

        image: calico/cni:v3.23.1

        imagePullPolicy: IfNotPresent

        name: install-cni

        resources: {}

        securityContext:

          privileged: true

        terminationMessagePath: /dev/termination-log

        terminationMessagePolicy: File

        volumeMounts:

        - mountPath: /host/opt/cni/bin

          name: cni-bin-dir

        - mountPath: /host/etc/cni/net.d

          name: cni-net-dir

      - image: calico/pod2daemon-flexvol:v3.23.1

        imagePullPolicy: IfNotPresent

        name: flexvol-driver

        resources: {}

        securityContext:

          privileged: true

        terminationMessagePath: /dev/termination-log

        terminationMessagePolicy: File

        volumeMounts:

        - mountPath: /host/driver

          name: flexvol-driver-host

      nodeSelector:

        kubernetes.io/os: linux

      priorityClassName: system-node-critical

      restartPolicy: Always

      schedulerName: default-scheduler

      securityContext: {}

      serviceAccount: calico-node

      serviceAccountName: calico-node

      terminationGracePeriodSeconds: 0

      tolerations:

      - effect: NoSchedule

        operator: Exists

      - key: CriticalAddonsOnly

        operator: Exists

      - effect: NoExecute

        operator: Exists

      volumes:

      - hostPath:

          path: /lib/modules

          type: ""

        name: lib-modules

      - hostPath:

          path: /var/run/calico

          type: ""

        name: var-run-calico

      - hostPath:

          path: /var/lib/calico

          type: ""

        name: var-lib-calico

      - hostPath:

          path: /run/xtables.lock

          type: FileOrCreate

        name: xtables-lock

      - hostPath:

          path: /sys/fs/

          type: DirectoryOrCreate

        name: sysfs

      - hostPath:

          path: /opt/cni/bin

          type: ""

        name: cni-bin-dir

      - hostPath:

          path: /etc/cni/net.d

          type: ""

        name: cni-net-dir

      - hostPath:

          path: /var/log/calico/cni

          type: ""

        name: cni-log-dir

      - hostPath:

          path: /var/lib/cni/networks

          type: ""

        name: host-local-net-dir

      - hostPath:

          path: /var/run/nodeagent

          type: DirectoryOrCreate

        name: policysync

      - hostPath:

          path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds

          type: DirectoryOrCreate

        name: flexvol-driver-host

  updateStrategy:

    rollingUpdate:

      maxSurge: 0

      maxUnavailable: 1

    type: RollingUpdate

ClusterRole

还需要修改 ClusterRole ，否则 Calico 会一直报权限错。

输出 yaml

kubectl get clusterrole calico-node -o yaml >calico-node-clusterrole.yaml

calico-node-clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  name: calico-node

rules:

- apiGroups:

  - ""

  resources:

  - pods

  - nodes

  - namespaces

  verbs:

  - get

- apiGroups:

  - discovery.k8s.io

  resources:

  - endpointslices

  verbs:

  - watch

  - list

- apiGroups:

  - ""

  resources:

  - endpoints

  - services

  verbs:

  - watch

  - list

  - get

- apiGroups:

  - ""

  resources:

  - configmaps

  verbs:

  - get

- apiGroups:

  - ""

  resources:

  - nodes/status

  verbs:

  - patch

  - update

- apiGroups:

  - networking.k8s.io

  resources:

  - networkpolicies

  verbs:

  - watch

  - list

- apiGroups:

  - ""

  resources:

  - pods

  - namespaces

  - serviceaccounts

  verbs:

  - list

  - watch

- apiGroups:

  - ""

  resources:

  - pods/status

  verbs:

  - patch

- apiGroups:

  - crd.projectcalico.org

  resources:

  - globalfelixconfigs

  - felixconfigurations

  - bgppeers

  - globalbgpconfigs

  - bgpconfigurations

  - ippools

  - ipamblocks

  - globalnetworkpolicies

  - globalnetworksets

  - networkpolicies

  - networksets

  - clusterinformations

  - hostendpoints

  - blockaffinities

  - caliconodestatuses

  - ipreservations

  verbs:

  - get

  - list

  - watch

- apiGroups:

  - crd.projectcalico.org

  resources:

  - ippools

  - felixconfigurations

  - clusterinformations

  verbs:

  - create

  - update

- apiGroups:

  - ""

  resources:

  - nodes

  verbs:

  - get

  - list

  - watch

- apiGroups:

  - crd.projectcalico.org

  resources:

  - bgpconfigurations

  - bgppeers

  verbs:

  - create

  - update

- apiGroups:

  - crd.projectcalico.org

  resources:

  - blockaffinities

  - ipamblocks

  - ipamhandles

  verbs:

  - get

  - list

  - create

  - update

  - delete

- apiGroups:

  - crd.projectcalico.org

  resources:

  - ipamconfigs

  verbs:

  - get

- apiGroups:

  - crd.projectcalico.org

  resources:

  - blockaffinities

  verbs:

  - watch

- apiGroups:

  - apps

  resources:

  - daemonsets

  verbs:

  - get

总结

这次奇怪的网络故障，最终原因还是因为 KubeSphere 的版本与 Kubernetes 的版本不匹配。所以工作环境要稳字为先，不要冒进使用最新的版本。否则会耽搁很多时间来解决莫名其妙的问题。

本文由博客一文多发平台 OpenWrite 发布！