分析kube-proxy的iptables规则

NodePort service

创建一个mysql的NodePort服务，对应两个pod实例，rc和service的配置如下：

1、rc配置

apiVersion: v1

kind: ReplicationController

metadata:

  name: wordpress-mysql

spec:

  replicas:

  selector:

    name: wordpress-mysql

  template:

    metadata:

      labels:

        name: wordpress-mysql

    spec:

      containers:

        - name: wordpress-mysql

          image: 172.16.114.201/library/mysql:v1

          ports:

            - containerPort:

          volumeMounts:

            - name: "wordpress-mysql-data"

              mountPath: "/var/lib/mysql"

          env:

          - name: MYSQL_PASS

            value: ""

          - name: ON_CREATE_DB

            value: "wordpress"

      volumes:

        - name: "wordpress-mysql-data"

          hostPath:

            path: "/root/wordpress-mysql/data"

2、service配置

apiVersion: v1

kind: Service

metadata:

  name: wordpress-mysql

spec:

  ports:

    - port:

      targetPort:

      nodePort:

      protocol: TCP

  type: NodePort

  selector:

    name: wordpress-mysql

3、创建的service情况

Name:            wordpress-mysql

Namespace:        default

Labels:            <none>

Selector:        name=wordpress-mysql

Type:            NodePort

IP:            10.254.67.85

Port:            <unset>    /TCP

NodePort:        <unset>    /TCP

Endpoints:        10.0.3.2:,10.0.45.6:

Session Affinity:    None

No events.

4、kube-proxy占用端口情况

[root@test- log]# netstat -anp | grep kube-proxy

tcp               127.0.0.1:         0.0.0.0:*               LISTEN      /kube-proxy

tcp               172.16.114.209:    172.16.114.208:     ESTABLISHED /kube-proxy

tcp               172.16.114.209:    172.16.114.208:     ESTABLISHED /kube-proxy

tcp               172.16.114.209:    172.16.114.208:     ESTABLISHED /kube-proxy

tcp6              :::                :::*                    LISTEN      /kube-proxy

unix        [ ]         DGRAM                        /kube-proxy

unix        [ ]         STREAM     CONNECTED         /kube-proxy

5、对应的iptables规则

iptables -S -t nat | grep mysql

-A KUBE-NODEPORTS -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp --dport  -j KUBE-MARK-MASQ

-A KUBE-NODEPORTS -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp --dport  -j KUBE-SVC-GJ6HULPZPPQIKMS7

-A KUBE-SEP-7KXQQUXVSZ2LFV44 -s 10.0.45.6/ -m comment --comment "default/wordpress-mysql:" -j KUBE-MARK-MASQ

-A KUBE-SEP-7KXQQUXVSZ2LFV44 -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp -j DNAT --to-destination 10.0.45.6:

-A KUBE-SEP-J7SZJXRP24HRFT23 -s 10.0.3.2/ -m comment --comment "default/wordpress-mysql:" -j KUBE-MARK-MASQ

-A KUBE-SEP-J7SZJXRP24HRFT23 -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp -j DNAT --to-destination 10.0.3.2:

-A KUBE-SERVICES -d 10.254.67.85/ -p tcp -m comment --comment "default/wordpress-mysql: cluster IP" -m tcp --dport  -j KUBE-SVC-GJ6HULPZPPQIKMS7

-A KUBE-SVC-GJ6HULPZPPQIKMS7 -m comment --comment "default/wordpress-mysql:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-J7SZJXRP24HRFT23

-A KUBE-SVC-GJ6HULPZPPQIKMS7 -m comment --comment "default/wordpress-mysql:" -j KUBE-SEP-7KXQQUXVSZ2LFV44

从以上信息看出，kube-proxy为mysql服务在node节点上单独起了一个端口30010，在iptables的规则中，目的端口30010被指向KUBE-SVC-GJ6HULPZPPQIKMS7，KUBE-SVC-GJ6HULPZPPQIKMS7又被指向KUBE-SEP-J7SZJXRP24HRFT23和KUBE-SEP-7KXQQUXVSZ2LFV44（他两各50%的几率），KUBE-SEP-J7SZJXRP24HRFT23和KUBE-SEP-7KXQQUXVSZ2LFV44定义了DNAT转换规则，将访问重定向到10.0.45.6:3306和10.0.3.2:3306这两个endpoint。因此，当外部访问30010端口时，根据iptables的规则会将该消息分发给10.0.45.6:3306和10.0.3.2:3306这两个地址（分发的几率是各50%）

ClusterIP service

创建一个zookeeper的ClusterIP服务，rc和service的配置如下：

1、rc配置

apiVersion: v1

kind: ReplicationController

metadata:

  name: zookeeper1

spec:

  replicas:

  selector:

    name: zookeeper1

  template:

    metadata:

      labels:

        name: zookeeper1

    spec:

      containers:

        - name: zookeeper1

          image: 10.10.30.166/public/zookeeper:v1

          ports:

            - containerPort:

            - containerPort:

            - containerPort:

          env:

          - name: ZOOKEEPER_ID

            value: ""

          - name: ZOOKEEPER_SERVER_1

            value: "zookeeper1"

          - name: ZOOKEEPER_SERVER_2

            value: "zookeeper2"

          - name: ZOOKEEPER_SERVER_3

            value: "zookeeper3"

2、service配置

apiVersion: v1

kind: Service

metadata:

  name: zookeeper1

spec:

  ports:

    - port:

      targetPort:

      protocol: TCP

      name: ""

    - port:

      targetPort:

      protocol: TCP

      name: ""

    - port:

      targetPort:

      protocol: TCP

      name: ""

  type: ClusterIP

  selector:

    name: zookeeper1

3、创建service情况

Name:            zookeeper1

Namespace:        default

Labels:            <none>

Selector:        name=zookeeper1

Type:            ClusterIP

IP:            10.254.181.6

Port:                /TCP

Endpoints:        10.0.45.4:

Port:                /TCP

Endpoints:        10.0.45.4:

Port:                /TCP

Endpoints:        10.0.45.4:

Session Affinity:    None

No events.

4、iptables规则

iptables -S -t nat | grep zookeeper1

-A KUBE-SEP-BZJZKIUQRVYJVMQB -s 10.0.45.4/ -m comment --comment "default/zookeeper1:3" -j KUBE-MARK-MASQ

-A KUBE-SEP-BZJZKIUQRVYJVMQB -p tcp -m comment --comment "default/zookeeper1:3" -m tcp -j DNAT --to-destination 10.0.45.4:

-A KUBE-SEP-C3J2QHMJ3LTD3GR7 -s 10.0.45.4/ -m comment --comment "default/zookeeper1:2" -j KUBE-MARK-MASQ

-A KUBE-SEP-C3J2QHMJ3LTD3GR7 -p tcp -m comment --comment "default/zookeeper1:2" -m tcp -j DNAT --to-destination 10.0.45.4:

-A KUBE-SEP-RZ4H7H2HFI3XFCXZ -s 10.0.45.4/ -m comment --comment "default/zookeeper1:1" -j KUBE-MARK-MASQ

-A KUBE-SEP-RZ4H7H2HFI3XFCXZ -p tcp -m comment --comment "default/zookeeper1:1" -m tcp -j DNAT --to-destination 10.0.45.4:

-A KUBE-SERVICES -d 10.254.181.6/ -p tcp -m comment --comment "default/zookeeper1:1 cluster IP" -m tcp --dport  -j KUBE-SVC-HHEJUKXW5P7DV7BX

-A KUBE-SERVICES -d 10.254.181.6/ -p tcp -m comment --comment "default/zookeeper1:2 cluster IP" -m tcp --dport  -j KUBE-SVC-2SVOYTXLXAXVV7L3

-A KUBE-SERVICES -d 10.254.181.6/ -p tcp -m comment --comment "default/zookeeper1:3 cluster IP" -m tcp --dport  -j KUBE-SVC-KAVJ7GO67HRSOAM3

-A KUBE-SVC-2SVOYTXLXAXVV7L3 -m comment --comment "default/zookeeper1:2" -j KUBE-SEP-C3J2QHMJ3LTD3GR7

-A KUBE-SVC-HHEJUKXW5P7DV7BX -m comment --comment "default/zookeeper1:1" -j KUBE-SEP-RZ4H7H2HFI3XFCXZ

-A KUBE-SVC-KAVJ7GO67HRSOAM3 -m comment --comment "default/zookeeper1:3" -j KUBE-SEP-BZJZKIUQRVYJVMQB

从iptables的规则来看，对目的ip是10.254.181.6，端口是2181、2888或者3888的消息，规则指向了KUBE-SVC-HHEJUKXW5P7DV7BX、KUBE-SVC-2SVOYTXLXAXVV7L3、KUBE-SVC-KAVJ7GO67HRSOAM3，他们三又分别指向了KUBE-SEP-C3J2QHMJ3LTD3GR7、KUBE-SEP-RZ4H7H2HFI3XFCXZ、KUBE-SEP-BZJZKIUQRVYJVMQB，这三条规则定义了DNAT转换规则，将访问重定向到了10.0.45.4:3888、10.0.45.4:2888、10.0.45.4:2181