二进制搭建kubernetes多master集群【二、配置flannel网络】

上一篇我们已经搭建etcd高可用集群，参考：二进制搭建kubernetes多master集群【一、使用TLS证书搭建etcd集群】

此文将搭建flannel网络，目的使跨主机的docker能够互相通信，也是保障kubernetes集群的网络基础和保障，下面正式开始配置。

一、生成Flannel网络TLS证书

在所有集群节点都安装Flannel，下面的操作在k8s-master1上进行，其他节点重复执行即可。（证书生成一次就行）

1、创建证书签名请求

cat > flanneld-csr.json <<EOF
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size":
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

• 该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空；

生成证书和私钥：

cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
  -ca-key=/etc/kubernetes/cert/ca-key.pem \
  -config=/etc/kubernetes/cert/ca-config.json \
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

2、将证书分发到所有集群节点/etc/kubernetes/cert/目录下

scp flanneld*.pem k8s-master1:/etc/kubernetes/cert/
scp flanneld*.pem k8s-master2:/etc/kubernetes/cert/
scp flanneld*.pem k8s-master3:/etc/kubernetes/cert/
scp flanneld*.pem k8s-node1:/etc/kubernetes/cert/
scp flanneld*.pem k8s-node2:/etc/kubernetes/cert/
scp flanneld*.pem k8s-node3:/etc/kubernetes/cert/

二、部署 Flannel

1、下载安装Flannel

wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz
cp {flanneld,mk-docker-opts.sh} /usr/local/bin

2、向 etcd 写入网段信息

下面2条命令在etcd集群中任意一台执行一次即可，也是是创建一个flannel网段供docker分配使用

[root@etcd1 cert]# etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/kubernetes/cert/etcd.pem --key-file=/etc/kubernetes/cert/etcd-key.pem mkdir /kubernetes/network
[root@etcd1 cert]# etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/kubernetes/cert/etcd.pem --key-file=/etc/kubernetes/cert/etcd-key.pem mk /kubernetes/network/config '{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan"}}'

3、创建system unit文件

[root@k8s-master1 ssl]# cat > /etc/systemd/system/flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/usr/local/bin/flanneld \
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \
  -etcd-certfile=/etc/kubernetes/cert/flanneld.pem \
  -etcd-keyfile=/etc/kubernetes/cert/flanneld-key.pem \
  -etcd-endpoints=https://192.168.80.4:2379,https://192.168.80.5:2379,https://192.168.80.6:2379 \
  -etcd-prefix=/kubernetes/network
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入到 /run/flannel/docker 文件中，后续 docker 启动时使用这个文件中参数值设置 docker0 网桥。

flanneld 使用系统缺省路由所在的接口和其它节点通信，对于有多个网络接口的机器（如，内网和公网），可以用 -iface=enpxx 选项值指定通信接口。

4、启动flannel并且设置开机自启动

systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld

5、查看flannel分配的子网信息

[root@k8s-master1 ~]# cat /run/flannel/docker
DOCKER_OPT_BIP="--bip=172.30.94.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.30.94.1/24 --ip-masq=true --mtu=1450"

[root@k8s-master1 ~]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=172.30.0.0/16
FLANNEL_SUBNET=172.30.94.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

/run/flannel/docker是flannel分配给docker的子网信息，/run/flannel/subnet.env包含了flannel整个大网段以及在此节点上的子网段。

6、查看flannel网络是否生效

[root@k8s-master1 ~]# ifconfig
docker0: flags=<UP,BROADCAST,RUNNING,MULTICAST>  mtu
        inet 172.30.94.1  netmask 255.255.255.0  broadcast 172.30.94.255
        inet6 fe80:::1aff:fed2:a4b4  prefixlen   scopeid 0x20<link>
        ether ::1a:d2:a4:b4  txqueuelen   (Ethernet)
        RX packets   bytes  (0.0 B)
        RX errors   dropped   overruns   frame
        TX packets   bytes  (648.0 B)
        TX errors   dropped  overruns   carrier   collisions 

ens192: flags=<UP,BROADCAST,RUNNING,MULTICAST>  mtu
        inet 192.168.20.210  netmask 255.255.254.0  broadcast 192.168.21.255
        inet6 fe80::ff2:187b:66fc:621b  prefixlen   scopeid 0x20<link>
        ether :0c::c3:dc:a5  txqueuelen   (Ethernet)
        RX packets   bytes  (590.6 MiB)
        RX errors   dropped   overruns   frame
        TX packets   bytes  (372.0 MiB)
        TX errors   dropped  overruns   carrier   collisions 

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.30.94.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80:::fcff:fe4e:b5ff  prefixlen   scopeid 0x20<link>
        ether 7a::fc:4e:b5:ff  txqueuelen   (Ethernet)
        RX packets   bytes  (0.0 B)
        RX errors   dropped   overruns   frame
        TX packets   bytes  (0.0 B)
        TX errors   dropped  overruns   carrier   collisions 

可以明显看到flannel1.1的网络信息，说明flannel网络已经正常。

三、配置docker支持flannel网络

1、所有node安装docker

关于安装docker，请参考：安装指定版本的docker

2、配置docker支持flannel网络，所有docker节点都操作

[root@k8s-master1 ~]# vi /etc/systemd/system/multi-user.target.wants/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd  and above support this version.
#TasksMax=infinity
TimeoutStartSec=
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

如上红色即为新增加支持flannel网络的配置

3、重启docker，使配置生效

systemctl daemon-reload
systemctl restart docker

4、查看docker网络是否生效

i、启动一个容器（如有现有容器可以不run一个新的）
[root@k8s-master1 ~]# docker run -itd centos
d63ee9c72b5023fgfg36ld93j6723hd72jd1hsp2303kf7
ii、查看ip地址是否是flannel网络分配的网段
[root@k8s-master1 ~]# docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' d63
172.30.94.2

5、查看所有集群主机的网络情况

[root@etcd1 cert]# etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/kubernetes/cert/etcd.pem --key-file=/etc/kubernetes/cert/etcd-key.pem ls /kubernetes/network/subnets
/kubernetes/network/subnets/172.30.94.0-
/kubernetes/network/subnets/172.30.51.0-
/kubernetes/network/subnets/172.30.10.0-24
/kubernetes/network/subnets/172.30.92.0-24
/kubernetes/network/subnets/172.30.85.0-24
/kubernetes/network/subnets/172.30.89.0-24

发现容器使用了172.30.94.0/24网段。属于flannel，配置完成。

下一篇将部署二进制搭建kubernetes多master集群【三、配置k8s master及高可用】