k8s网络之calico学习

一、知识准备

1.calico主要通过ipip协议与bgp协议来实现通信。前者通过ipip隧道作为通信基础，后者则是纯三层的路由交换

2.bgp协议主要由两种方式：BGP Speaker 全互联模式(node-to-node mesh)与BGP Speaker RR模式

3.本文主要探索一下calico bgp的两种模式

二、环境准备

组件	版本
OS	Ubuntu 18.04.1 LTS
docker	18.06.0-ce
k8s	1.10.1
calico	3.1.3

ip	hostname
192.168.56.101	k8s-master
192.168.56.102	k8s-node1
192.168.56.103	k8s-node2
192.168.56.104	k8s-node3

三、安装

k8s安装

参考官网安装以及社区诸多大神的安装帖子，这里就不班门弄斧了

本文的k8s的环境打开了rbac，etcd加入了证书

calico安装

主要参考官方文档 https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/calico

1.calico rbac

kubectl apply -f \
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/rbac.yaml

2.下载calico.yaml

curl \
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/calico.yaml \
-O

3.填入etcd地址

ETCD_ENDPOINTS="https://192.168.56.101:2379"
sed -i "s#.*etcd_endpoints:.*#  etcd_endpoints: \"${ETCD_ENDPOINTS}\"#g" calico.yaml

sed -i "s#__ETCD_ENDPOINTS__#${ETCD_ENDPOINTS}#g" calico.yaml

4.将etcd证书信息填入。我的etcd证书在/etc/etcd/ssl下

ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n'`
ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n'`
ETCD_CA=`cat /etc/etcd/ssl/etcd-root-ca.pem | base64 | tr -d '\n'`

sed -i "s#.*etcd-cert:.*#  etcd-cert: ${ETCD_CERT}#g" calico.yaml
sed -i "s#.*etcd-key:.*#  etcd-key: ${ETCD_KEY}#g" calico.yaml
sed -i "s#.*etcd-ca:.*#  etcd-ca: ${ETCD_CA}#g" calico.yaml

sed -i 's#.*etcd_ca:.*#  etcd_ca: "/calico-secrets/etcd-ca"#g' calico.yaml
sed -i 's#.*etcd_cert:.*#  etcd_cert: "/calico-secrets/etcd-cert"#g' calico.yaml
sed -i 's#.*etcd_key:.*#  etcd_key: "/calico-secrets/etcd-key"#g' calico.yaml

sed -i "s#__ETCD_KEY_FILE__#/etc/etcd/ssl/etcd-key.pem#g" calico.yaml
sed -i "s#__ETCD_CERT_FILE__#/etc/etcd/ssl/etcd.pem#g" calico.yaml
sed -i "s#__ETCD_CA_CERT_FILE__#/etc/etcd/ssl/etcd-root-ca.pem#g" calico.yaml
sed -i "s#__KUBECONFIG_FILEPATH__#/etc/cni/net.d/calico-kubeconfig#g" calico.yaml

5.配置calico bgp 并且修改ip cidr:10.10.0.0/16

sed -i '/CALICO_IPV4POOL_IPIP/{n;s/Always/off/g}' calico.yaml
sed -i '/CALICO_IPV4POOL_CIDR/{n;s/192.168.0.0/10.10.0.0/g}' calico.yaml

6.kubectl安装calico

kubectl apply -f calico.yaml

注意:因为calico-node需要获取操作系统的权限运行，所以要在apiserver、kubelet中加入--allow-privileged=true

查看一下状态:

root@k8s-master:/tmp# kubectl get pods -n kube-system -owide
NAME                                        READY     STATUS    RESTARTS   AGE       IP               NODE
calico-kube-controllers-98989846-b4n72      1/1       Running   0          18d       192.168.56.102   k8s-node1
calico-node-58pck                           2/2       Running   0          18d       192.168.56.103   k8s-node2
calico-node-s2txw                           2/2       Running   0          18d       192.168.56.101   k8s-master
calico-node-svmbp                           2/2       Running   0          18d       192.168.56.102   k8s-node1
...

7.kubelet配置calico

找到kubelet的配置文件（我的环境在/etc/kubernetes/kubelet），加入
--network-plugin=cni

重启kubelet

8.测试一个pod

cat << EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: network-test
  namespace: test
spec:
  containers:
  - image: busybox:latest
    command:
      - sleep
      - "3600"
    name: network-test
EOF

root@k8s-master:~# kubectl -n test get pods -owide
NAME           READY     STATUS    RESTARTS   AGE       IP              NODE
network-test   1/1       Running   0          41s       10.10.169.139   k8s-node2

至此：calico安装已经完成

四、calicoctl使用

1.下载calicoctl

https://github.com/projectcalico/calicoctl/releases/download/v3.1.3/calicoctl-linux-amd64

2.查看当前的calico-node

root@k8s-master:/tmp# calicoctl get node
NAME
k8s-master
k8s-node1
k8s-node2

calicoctl get node -o yaml 查看详细信息

3.查看当前的ippool

root@k8s-master:/tmp# calicoctl get ippool
NAME                  CIDR
default-ipv4-ippool   10.10.0.0/16
default-ipv6-ippool   fdc6:1a69:2b39::/48

4.查看当前模式

root@k8s-master:/tmp# calicoctl node status
Calico process is running.

IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+----------------+-------------------+-------+----------+-------------+
| 192.168.56.102 | node-to-node mesh | up    | 07:39:02 | Established |
| 192.168.56.103 | node-to-node mesh | up    | 07:39:02 | Established |
+----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

root@k8s-master:/tmp# netstat -anp | grep ESTABLISH | grep bird
tcp        0      0 192.168.56.101:33029    192.168.56.102:179      ESTABLISHED 26558/bird
tcp        0      0 192.168.56.101:58055    192.168.56.103:179      ESTABLISHED 26558/bird

当前运行在BGP Speaker 全互联模式(node-to-node mesh)模式，calico集群中的节点之间都会相互建立连接，用于路由交换。适合规模不大的集群中运行，一旦集群节点增大，mesh模式将形成一个巨大服务网格，连接数暴增

5.修改BGP Speaker RR模式

禁止mesh模式，配置bgpPeer

cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  asNumber: 61234
EOF

cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-global
spec:
  peerIP: 192.168.56.103
  asNumber: 61234
EOF

查看RR模式配置：

root@k8s-master:~# calicoctl get bgpconfig
NAME      LOGSEVERITY   MESHENABLED   ASNUMBER
default   Info          false         61234

root@k8s-master:~# calicoctl get bgppeer
NAME             PEERIP           NODE       ASN
bgppeer-global   192.168.56.103   (global)   61234

安装routereflector

docker run --privileged --net=host -d                             \
           --name=calico-rr                                       \
           -e IP=192.168.56.104                                   \
           -e ETCD_ENDPOINTS=https://192.168.56.101:2379          \
           -v /etc/etcd/ssl:/etc/calico/ssl                     \
           -e ETCD_CA_CERT_FILE=/etc/calico/ssl/etcd-root-ca.pem  \
           -e ETCD_CERT_FILE=/etc/calico/ssl/etcd.pem             \
           -e ETCD_KEY_FILE=/etc/calico/ssl/etcd-key.pem          \
           calico/routereflector:v0.6.1

查看效果：

root@k8s-master:~# calicoctl node status
Calico process is running.

IPv4 BGP status
+----------------+-----------+-------+----------+-------------+
|  PEER ADDRESS  | PEER TYPE | STATE |  SINCE   |    INFO     |
+----------------+-----------+-------+----------+-------------+
| 192.168.56.103 | global    | up    | 09:13:23 | Established |
+----------------+-----------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

root@k8s-master:~# netstat -anp | grep ESTABLISH | grep bird
tcp        0      0 192.168.56.101:179      192.168.56.103:54903    ESTABLISHED 26558/bird

每台机器都只会与rr建立一条连接，并且与rr通信即可拿到所有路由，大大减少了连接数量

---

至此，本文结束

在下才疏学浅，有撒汤漏水的，请各位不吝赐教...