[k8s]kube-router替代kube-proxy实现svc网络和pod网络

本文讲解了kube-router部署,无需在部署kube-proxy了. kube-router采用lvs实现svc网络,采用bgp实现pod网络.

kube-router也是基于cni网络,本文是容器方式跑的kube-router

• 1.替代了kube-proxy组件,无需在部署kube-proxy了,解决了svc网络
• 2.自带cni,bgp,解决了pod网络
• 3.基于ipvs转发

• 4.路由传播依赖bgp

kuberouter结构

参考(部署步骤): https://cloudnativelabs.github.io/post/2017-04-19-kube-router/

部署步骤

要注意的是

• 1./root/bootstrap.kubeconfig文件

• 2.节点开启支持ipv6

• 1. kubelet 要有--network-plugin-dir=/opt/cni/bin --network-plugin=cni --cni-conf-dir=/etc/cni/net.d/ --allow-privileged=true

环境准备

mkdir -p /etc/cni/net.d /opt/cni/bin
wget https://github.com/containernetworking/plugins/releases/download/v0.6.0/cni-plugins-amd64-v0.6.0.tgz
tar xf cni-plugins-amd64-v0.6.0.tgz -C /opt/cni/bin

kube-apiserver --service-cluster-ip-range=10.254.0.0/16 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --admission-control=ServiceAccount     --service-account-key-file=/root/ssl/ca.key --client-ca-file=/root/ssl/ca.crt --tls-cert-file=/root/ssl/server.crt --tls-private-key-file=/root/ssl/server.key --allow-privileged=true --storage-backend=etcd2 --v=2 --enable-bootstrap-token-auth --token-auth-file=/root/token.csv

kube-controller-manager   --master=http://127.0.0.1:8080   --service-account-private-key-file=/root/ssl/ca.key  --cluster-signing-cert-file=/root/ssl/ca.crt --cluster-signing-key-file=/root/ssl/ca.key --root-ca-file=/root/ssl/ca.crt --v=2  --allocate-node-cidrs=true --cluster-cidr=10.1.0.0/16

kube-scheduler --master=http://127.0.0.1:8080 --v=2 

kubelet --allow-privileged=true --cluster-dns=10.254.0.2 --cluster-domain=cluster.local --v=2 --experimental-bootstrap-kubeconfig=/root/bootstrap.kubeconfig --kubeconfig=/root/kubelet.kubeconfig --fail-swap-on=false   --network-plugin=cni --cni-conf-dir=/etc/cni/net.d/ --allow-privileged=true

准备token.csv和bootstrap.kubeconfig文件

- 在master生成token.csv
BOOTSTRAP_TOKEN="41f7e4ba8b7be874fcff18bf5cf41a7c"
cat > token.csv<<EOF
41f7e4ba8b7be874fcff18bf5cf41a7c,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

- 将bootstrap.kubeconfig同步到所有节点
设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/root/ssl/ca.crt \
  --embed-certs=true \
  --server=http://192.168.14.11:8080 \
  --kubeconfig=bootstrap.kubeconfig

设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
  --token="41f7e4ba8b7be874fcff18bf5cf41a7c" \
  --kubeconfig=bootstrap.kubeconfig

设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

两个文件我都放在了/root下.

这里用到bootstrap.kubeconfig,同步到node各个节点.

部署kube-router

[root@n1 kube-router]# cat kube-router.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-router-cfg
  namespace: kube-system
  labels:
    tier: node
    k8s-app: kube-router
data:
  cni-conf.json: |
    {
      "name":"kubernetes",
      "type":"bridge",
      "bridge":"kube-bridge",
      "isDefaultGateway":true,
      "ipam": {
        "type":"host-local"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-router
  namespace: kube-system
  labels:
    k8s-app: kube-router
spec:
  template:
    metadata:
      labels:
        k8s-app: kube-router
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      containers:
      - name: kube-router
        image: cloudnativelabs/kube-router
        args: ["--run-router=true", "--run-firewall=true", "--run-service-proxy=true", "--kubeconfig=/var/lib/kube-router/kubeconfig"]
        securityContext:
          privileged: true
        imagePullPolicy: Always
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
        - name: lib-modules
          mountPath: /lib/modules
          readOnly: true
        - name: cni-conf-dir
          mountPath: /etc/cni/net.d
        - name: kubeconfig
          mountPath: /var/lib/kube-router/kubeconfig
          readOnly: true
      initContainers:
      - name: install-cni
        image: busybox
        imagePullPolicy: Always
        command:
        - /bin/sh
        - -c
        - set -e -x;
          if [ ! -f /etc/cni/net.d/10-kuberouter.conf ]; then
            TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;
            cp /etc/kube-router/cni-conf.json ${TMP};
            mv ${TMP} /etc/cni/net.d/10-kuberouter.conf;
          fi
        volumeMounts:
        - name: cni-conf-dir
          mountPath: /etc/cni/net.d
        - name: kube-router-cfg
          mountPath: /etc/kube-router
      hostNetwork: true
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      volumes:
      - name: lib-modules
        hostPath:
          path: /lib/modules
      - name: cni-conf-dir
        hostPath:
          path: /etc/cni/net.d
      - name: kube-router-cfg
        configMap:
          name: kube-router-cfg
      - name: kubeconfig
        hostPath:
          path: /root/bootstrap.kubeconfig

注: /root/bootstrap.kubeconfig.

[root@n1 kube-router]# kk
NAMESPACE     NAME                READY     STATUS    RESTARTS   AGE       IP              NODE        LABELS
kube-system   kube-router-989p5   1/1       Running   0          9m        192.168.14.12   n2.ma.com   controller-revision-hash=1689399381,k8s-app=kube-router,pod-template-generation=1
kube-system   kube-router-plmpv   1/1       Running   0          9m        192.168.14.13   n3.ma.com   controller-revision-hash=1689399381,k8s-app=kube-router,pod-template-generation=1

测试连通性

kubectl run -it --rm --restart=Never b10 --image=busybox sh
kubectl run -it --rm --restart=Never b20 --image=busybox sh

[root@n1 ~]# kk
NAMESPACE     NAME                  READY     STATUS        RESTARTS   AGE       IP              NODE        LABELS
default       b10                   1/1       Running       0          16s       10.1.1.26       n3.ma.com   run=b10
default       b20                   1/1       Running       0          7s        10.1.0.14       n2.ma.com   run=b20

[root@n1 yaml]# kubectl run -it --rm --restart=Never b10 --image=busybox sh
If you don't see a command prompt, try pressing enter.
/ # ping  10.1.0.14
PING 10.1.0.14 (10.1.0.14): 56 data bytes
64 bytes from 10.1.0.14: seq=0 ttl=62 time=2.018 ms
64 bytes from 10.1.0.14: seq=1 ttl=62 time=0.576 ms
^C

遇到的问题

• 1./root/bootstrap.kubeconfig文件

• 2.节点开启支持ipv6

• 1. kubelet 要有--network-plugin-dir=/opt/cni/bin --network-plugin=cni --cni-conf-dir=/etc/cni/net.d/ --allow-privileged=true