k8s网络之calico学习

一、知识准备

1.calico主要通过ipip协议与bgp协议来实现通信。前者通过ipip隧道作为通信基础，后者则是纯三层的路由交换

2.bgp协议主要由两种方式：BGP Speaker 全互联模式(node-to-node mesh)与BGP Speaker RR模式

3.本文主要探索一下calico bgp的两种模式

二、环境准备

组件	版本
OS	Ubuntu 18.04.1 LTS
docker	18.06.0-ce
k8s	1.10.1
calico	3.1.3

ip	hostname
192.168.56.101	k8s-master
192.168.56.102	k8s-node1
192.168.56.103	k8s-node2
192.168.56.104	k8s-node3

三、安装

k8s安装

参考官网安装以及社区诸多大神的安装帖子，这里就不班门弄斧了

本文的k8s的环境打开了rbac，etcd加入了证书

calico安装

主要参考官方文档 https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/calico

1.calico rbac

kubectl apply -f \

https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/rbac.yaml

2.下载calico.yaml

curl \

https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/calico.yaml \

-O

3.填入etcd地址

ETCD_ENDPOINTS="https://192.168.56.101:2379"

sed -i "s#.*etcd_endpoints:.*#  etcd_endpoints: \"${ETCD_ENDPOINTS}\"#g" calico.yaml

sed -i "s#__ETCD_ENDPOINTS__#${ETCD_ENDPOINTS}#g" calico.yaml

4.将etcd证书信息填入。我的etcd证书在/etc/etcd/ssl下

ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n'`

ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n'`

ETCD_CA=`cat /etc/etcd/ssl/etcd-root-ca.pem | base64 | tr -d '\n'`

sed -i "s#.*etcd-cert:.*#  etcd-cert: ${ETCD_CERT}#g" calico.yaml

sed -i "s#.*etcd-key:.*#  etcd-key: ${ETCD_KEY}#g" calico.yaml

sed -i "s#.*etcd-ca:.*#  etcd-ca: ${ETCD_CA}#g" calico.yaml

sed -i 's#.*etcd_ca:.*#  etcd_ca: "/calico-secrets/etcd-ca"#g' calico.yaml

sed -i 's#.*etcd_cert:.*#  etcd_cert: "/calico-secrets/etcd-cert"#g' calico.yaml

sed -i 's#.*etcd_key:.*#  etcd_key: "/calico-secrets/etcd-key"#g' calico.yaml

sed -i "s#__ETCD_KEY_FILE__#/etc/etcd/ssl/etcd-key.pem#g" calico.yaml

sed -i "s#__ETCD_CERT_FILE__#/etc/etcd/ssl/etcd.pem#g" calico.yaml

sed -i "s#__ETCD_CA_CERT_FILE__#/etc/etcd/ssl/etcd-root-ca.pem#g" calico.yaml

sed -i "s#__KUBECONFIG_FILEPATH__#/etc/cni/net.d/calico-kubeconfig#g" calico.yaml

5.配置calico bgp 并且修改ip cidr:10.10.0.0/16

sed -i '/CALICO_IPV4POOL_IPIP/{n;s/Always/off/g}' calico.yaml

sed -i '/CALICO_IPV4POOL_CIDR/{n;s/192.168.0.0/10.10.0.0/g}' calico.yaml

6.kubectl安装calico

kubectl apply -f calico.yaml

注意:因为calico-node需要获取操作系统的权限运行，所以要在apiserver、kubelet中加入--allow-privileged=true

查看一下状态:

root@k8s-master:/tmp# kubectl get pods -n kube-system -owide

NAME                                        READY     STATUS    RESTARTS   AGE       IP               NODE

calico-kube-controllers-98989846-b4n72      1/1       Running   0          18d       192.168.56.102   k8s-node1

calico-node-58pck                           2/2       Running   0          18d       192.168.56.103   k8s-node2

calico-node-s2txw                           2/2       Running   0          18d       192.168.56.101   k8s-master

calico-node-svmbp                           2/2       Running   0          18d       192.168.56.102   k8s-node1

...

7.kubelet配置calico

找到kubelet的配置文件（我的环境在/etc/kubernetes/kubelet），加入

--network-plugin=cni

重启kubelet

8.测试一个pod

cat << EOF | kubectl create -f -

apiVersion: v1

kind: Pod

metadata:

  name: network-test

  namespace: test

spec:

  containers:

  - image: busybox:latest

    command:

      - sleep

      - "3600"

    name: network-test

EOF

root@k8s-master:~# kubectl -n test get pods -owide

NAME           READY     STATUS    RESTARTS   AGE       IP              NODE

network-test   1/1       Running   0          41s       10.10.169.139   k8s-node2

至此：calico安装已经完成

四、calicoctl使用

1.下载calicoctl

https://github.com/projectcalico/calicoctl/releases/download/v3.1.3/calicoctl-linux-amd64

2.查看当前的calico-node

root@k8s-master:/tmp# calicoctl get node

NAME

k8s-master

k8s-node1

k8s-node2

calicoctl get node -o yaml 查看详细信息

3.查看当前的ippool

root@k8s-master:/tmp# calicoctl get ippool

NAME                  CIDR

default-ipv4-ippool   10.10.0.0/16

default-ipv6-ippool   fdc6:1a69:2b39::/48

4.查看当前模式

root@k8s-master:/tmp# calicoctl node status

Calico process is running.

IPv4 BGP status

+----------------+-------------------+-------+----------+-------------+

|  PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |

+----------------+-------------------+-------+----------+-------------+

| 192.168.56.102 | node-to-node mesh | up    | 07:39:02 | Established |

| 192.168.56.103 | node-to-node mesh | up    | 07:39:02 | Established |

+----------------+-------------------+-------+----------+-------------+

IPv6 BGP status

No IPv6 peers found.

root@k8s-master:/tmp# netstat -anp | grep ESTABLISH | grep bird

tcp        0      0 192.168.56.101:33029    192.168.56.102:179      ESTABLISHED 26558/bird

tcp        0      0 192.168.56.101:58055    192.168.56.103:179      ESTABLISHED 26558/bird

当前运行在BGP Speaker 全互联模式(node-to-node mesh)模式，calico集群中的节点之间都会相互建立连接，用于路由交换。适合规模不大的集群中运行，一旦集群节点增大，mesh模式将形成一个巨大服务网格，连接数暴增

5.修改BGP Speaker RR模式

禁止mesh模式，配置bgpPeer

cat << EOF | calicoctl create -f -

apiVersion: projectcalico.org/v3

kind: BGPConfiguration

metadata:

  name: default

spec:

  logSeverityScreen: Info

  nodeToNodeMeshEnabled: false

  asNumber: 61234

EOF

cat << EOF | calicoctl create -f -

apiVersion: projectcalico.org/v3

kind: BGPPeer

metadata:

  name: bgppeer-global

spec:

  peerIP: 192.168.56.103

  asNumber: 61234

EOF

查看RR模式配置：

root@k8s-master:~# calicoctl get bgpconfig

NAME      LOGSEVERITY   MESHENABLED   ASNUMBER

default   Info          false         61234

root@k8s-master:~# calicoctl get bgppeer

NAME             PEERIP           NODE       ASN

bgppeer-global   192.168.56.103   (global)   61234

安装routereflector

docker run --privileged --net=host -d                             \

           --name=calico-rr                                       \

           -e IP=192.168.56.104                                   \

           -e ETCD_ENDPOINTS=https://192.168.56.101:2379          \

           -v /etc/etcd/ssl:/etc/calico/ssl                     \

           -e ETCD_CA_CERT_FILE=/etc/calico/ssl/etcd-root-ca.pem  \

           -e ETCD_CERT_FILE=/etc/calico/ssl/etcd.pem             \

           -e ETCD_KEY_FILE=/etc/calico/ssl/etcd-key.pem          \

           calico/routereflector:v0.6.1

查看效果：

root@k8s-master:~# calicoctl node status

Calico process is running.

IPv4 BGP status

+----------------+-----------+-------+----------+-------------+

|  PEER ADDRESS  | PEER TYPE | STATE |  SINCE   |    INFO     |

+----------------+-----------+-------+----------+-------------+

| 192.168.56.103 | global    | up    | 09:13:23 | Established |

+----------------+-----------+-------+----------+-------------+

IPv6 BGP status

No IPv6 peers found.

root@k8s-master:~# netstat -anp | grep ESTABLISH | grep bird

tcp        0      0 192.168.56.101:179      192.168.56.103:54903    ESTABLISHED 26558/bird

每台机器都只会与rr建立一条连接，并且与rr通信即可拿到所有路由，大大减少了连接数量

至此，本文结束

在下才疏学浅，有撒汤漏水的，请各位不吝赐教...