分析kube-proxy的iptables规则

NodePort service

创建一个mysql的NodePort服务，对应两个pod实例，rc和service的配置如下：

1、rc配置

apiVersion: v1
kind: ReplicationController
metadata:
  name: wordpress-mysql
spec:
  replicas:
  selector:
    name: wordpress-mysql
  template:
    metadata:
      labels:
        name: wordpress-mysql
    spec:
      containers:
        - name: wordpress-mysql
          image: 172.16.114.201/library/mysql:v1
          ports:
            - containerPort:
          volumeMounts:
            - name: "wordpress-mysql-data"
              mountPath: "/var/lib/mysql"
          env:
          - name: MYSQL_PASS
            value: ""
          - name: ON_CREATE_DB
            value: "wordpress"
      volumes:
        - name: "wordpress-mysql-data"
          hostPath:
            path: "/root/wordpress-mysql/data"

2、service配置

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
spec:
  ports:
    - port:
      targetPort:
      nodePort:
      protocol: TCP
  type: NodePort
  selector:
    name: wordpress-mysql

3、创建的service情况

Name:            wordpress-mysql
Namespace:        default
Labels:            <none>
Selector:        name=wordpress-mysql
Type:            NodePort
IP:            10.254.67.85
Port:            <unset>    /TCP
NodePort:        <unset>    /TCP
Endpoints:        10.0.3.2:,10.0.45.6:
Session Affinity:    None
No events.

4、kube-proxy占用端口情况

[root@test- log]# netstat -anp | grep kube-proxy
tcp               127.0.0.1:         0.0.0.0:*               LISTEN      /kube-proxy
tcp               172.16.114.209:    172.16.114.208:     ESTABLISHED /kube-proxy
tcp               172.16.114.209:    172.16.114.208:     ESTABLISHED /kube-proxy
tcp               172.16.114.209:    172.16.114.208:     ESTABLISHED /kube-proxy
tcp6              :::                :::*                    LISTEN      /kube-proxy
unix        [ ]         DGRAM                        /kube-proxy
unix        [ ]         STREAM     CONNECTED         /kube-proxy

5、对应的iptables规则

iptables -S -t nat | grep mysql
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp --dport  -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp --dport  -j KUBE-SVC-GJ6HULPZPPQIKMS7
-A KUBE-SEP-7KXQQUXVSZ2LFV44 -s 10.0.45.6/ -m comment --comment "default/wordpress-mysql:" -j KUBE-MARK-MASQ
-A KUBE-SEP-7KXQQUXVSZ2LFV44 -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp -j DNAT --to-destination 10.0.45.6:
-A KUBE-SEP-J7SZJXRP24HRFT23 -s 10.0.3.2/ -m comment --comment "default/wordpress-mysql:" -j KUBE-MARK-MASQ
-A KUBE-SEP-J7SZJXRP24HRFT23 -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp -j DNAT --to-destination 10.0.3.2:
-A KUBE-SERVICES -d 10.254.67.85/ -p tcp -m comment --comment "default/wordpress-mysql: cluster IP" -m tcp --dport  -j KUBE-SVC-GJ6HULPZPPQIKMS7
-A KUBE-SVC-GJ6HULPZPPQIKMS7 -m comment --comment "default/wordpress-mysql:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-J7SZJXRP24HRFT23
-A KUBE-SVC-GJ6HULPZPPQIKMS7 -m comment --comment "default/wordpress-mysql:" -j KUBE-SEP-7KXQQUXVSZ2LFV44

从以上信息看出，kube-proxy为mysql服务在node节点上单独起了一个端口30010，在iptables的规则中，目的端口30010被指向KUBE-SVC-GJ6HULPZPPQIKMS7，KUBE-SVC-GJ6HULPZPPQIKMS7又被指向KUBE-SEP-J7SZJXRP24HRFT23和KUBE-SEP-7KXQQUXVSZ2LFV44（他两各50%的几率），KUBE-SEP-J7SZJXRP24HRFT23和KUBE-SEP-7KXQQUXVSZ2LFV44定义了DNAT转换规则，将访问重定向到10.0.45.6:3306和10.0.3.2:3306这两个endpoint。因此，当外部访问30010端口时，根据iptables的规则会将该消息分发给10.0.45.6:3306和10.0.3.2:3306这两个地址（分发的几率是各50%）

ClusterIP service

创建一个zookeeper的ClusterIP服务，rc和service的配置如下：

1、rc配置

apiVersion: v1
kind: ReplicationController
metadata:
  name: zookeeper1
spec:
  replicas:
  selector:
    name: zookeeper1
  template:
    metadata:
      labels:
        name: zookeeper1
    spec:
      containers:
        - name: zookeeper1
          image: 10.10.30.166/public/zookeeper:v1
          ports:
            - containerPort:
            - containerPort:
            - containerPort:
          env:
          - name: ZOOKEEPER_ID
            value: ""
          - name: ZOOKEEPER_SERVER_1
            value: "zookeeper1"
          - name: ZOOKEEPER_SERVER_2
            value: "zookeeper2"
          - name: ZOOKEEPER_SERVER_3
            value: "zookeeper3"

2、service配置

apiVersion: v1
kind: Service
metadata:
  name: zookeeper1
spec:
  ports:
    - port:
      targetPort:
      protocol: TCP
      name: ""
    - port:
      targetPort:
      protocol: TCP
      name: ""
    - port:
      targetPort:
      protocol: TCP
      name: ""
  type: ClusterIP
  selector:
    name: zookeeper1

3、创建service情况

Name:            zookeeper1
Namespace:        default
Labels:            <none>
Selector:        name=zookeeper1
Type:            ClusterIP
IP:            10.254.181.6
Port:                /TCP
Endpoints:        10.0.45.4:
Port:                /TCP
Endpoints:        10.0.45.4:
Port:                /TCP
Endpoints:        10.0.45.4:
Session Affinity:    None
No events.

4、iptables规则

iptables -S -t nat | grep zookeeper1
-A KUBE-SEP-BZJZKIUQRVYJVMQB -s 10.0.45.4/ -m comment --comment "default/zookeeper1:3" -j KUBE-MARK-MASQ
-A KUBE-SEP-BZJZKIUQRVYJVMQB -p tcp -m comment --comment "default/zookeeper1:3" -m tcp -j DNAT --to-destination 10.0.45.4:
-A KUBE-SEP-C3J2QHMJ3LTD3GR7 -s 10.0.45.4/ -m comment --comment "default/zookeeper1:2" -j KUBE-MARK-MASQ
-A KUBE-SEP-C3J2QHMJ3LTD3GR7 -p tcp -m comment --comment "default/zookeeper1:2" -m tcp -j DNAT --to-destination 10.0.45.4:
-A KUBE-SEP-RZ4H7H2HFI3XFCXZ -s 10.0.45.4/ -m comment --comment "default/zookeeper1:1" -j KUBE-MARK-MASQ
-A KUBE-SEP-RZ4H7H2HFI3XFCXZ -p tcp -m comment --comment "default/zookeeper1:1" -m tcp -j DNAT --to-destination 10.0.45.4:
-A KUBE-SERVICES -d 10.254.181.6/ -p tcp -m comment --comment "default/zookeeper1:1 cluster IP" -m tcp --dport  -j KUBE-SVC-HHEJUKXW5P7DV7BX
-A KUBE-SERVICES -d 10.254.181.6/ -p tcp -m comment --comment "default/zookeeper1:2 cluster IP" -m tcp --dport  -j KUBE-SVC-2SVOYTXLXAXVV7L3
-A KUBE-SERVICES -d 10.254.181.6/ -p tcp -m comment --comment "default/zookeeper1:3 cluster IP" -m tcp --dport  -j KUBE-SVC-KAVJ7GO67HRSOAM3
-A KUBE-SVC-2SVOYTXLXAXVV7L3 -m comment --comment "default/zookeeper1:2" -j KUBE-SEP-C3J2QHMJ3LTD3GR7
-A KUBE-SVC-HHEJUKXW5P7DV7BX -m comment --comment "default/zookeeper1:1" -j KUBE-SEP-RZ4H7H2HFI3XFCXZ
-A KUBE-SVC-KAVJ7GO67HRSOAM3 -m comment --comment "default/zookeeper1:3" -j KUBE-SEP-BZJZKIUQRVYJVMQB

从iptables的规则来看，对目的ip是10.254.181.6，端口是2181、2888或者3888的消息，规则指向了KUBE-SVC-HHEJUKXW5P7DV7BX、KUBE-SVC-2SVOYTXLXAXVV7L3、KUBE-SVC-KAVJ7GO67HRSOAM3，他们三又分别指向了KUBE-SEP-C3J2QHMJ3LTD3GR7、KUBE-SEP-RZ4H7H2HFI3XFCXZ、KUBE-SEP-BZJZKIUQRVYJVMQB，这三条规则定义了DNAT转换规则，将访问重定向到了10.0.45.4:3888、10.0.45.4:2888、10.0.45.4:2181