etcd和flannel实现docker跨物理机通信

实验目标

跨物理机的容器之间能直接访问
docker通过Flannel可以实现各容器间的相互通信，即宿主机和容器，容器和容器之间都能相互通信

实验环境

192.168.3.50 //etcd、flannel、docker
192.168.3.51 //flannel、docker

00、系统统一设置

hosts\selinux\firewalld\iptalbes

systemctl stop firewalld.service

#放开iptables，让flannel通过iptables实现跨网通信

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -F

01、etcd部署

https://github.com/etcd-io/etcd/releases
wget https://github.com/coreos/etcd/releases/download/v2.3.2/etcd-v2.3.2-linux-amd64.tar.gz
tar xvf etcd-v2.3.2-linux-amd64.tar.gz
cd etcd-v2.3.2-linux-amd64
mv etcd* /usr/local/bin/

加入systemd管理，并且为了以后扩展，我还设置了集群，大家可以不必添加集群设置
cat > /usr/lib/systemd/system/etcd.service <<-'EOF'
[Unit]
Description=etcd
[Service]
Environment=ETCD_NAME=kubernetes
Environment=ETCD_DATA_DIR=/var/lib/etcd
Environment=ETCD_LISTEN_CLIENT_URLS=http://192.168.3.50:4001,http://localhost:4001
Environment=ETCD_LISTEN_PEER_URLS=http://192.168.3.50:7001,http://localhost:7001
Environment=ETCD_INITIAL_ADVERTISE_PEER_URLS=http://192.168.3.50:7001,http://localhost:7001
Environment=ETCD_ADVERTISE_CLIENT_URLS=http://192.168.3.50:4001,http://localhost:4001
Environment=ETCD_INITIAL_CLUSTER_STATE=new
Environment=ETCD_INITIAL_CLUSTER_TOKEN=Kubernetes
Environment=ETCD_INITIAL_CLUSTER=kubernetes=http://192.168.3.50:7001,kubernetes=http://localhost:7001
ExecStart=/usr/local/bin/etcd
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl start etcd

etcd添加网段设置分配给docker网络的网段
etcdctl mk /coreos.com/network/config '{"Network":"172.20.0.0/16", "SubnetMin": "172.20.1.0", "SubnetMax": "172.20.254.0"}'
etcdctl get /coreos.com/network/config

02、docker-ce部署

参考如下文章：

https://www.cnblogs.com/xiaochina/p/10469715.html

https://www.cnblogs.com/xiaochina/p/11518007.html

03、flannel部署

注意：所有主机均需要安装flannel
二进制安装flannel
wget https://github.com/coreos/flannel/releases/download/v0.5.5/flannel-0.5.5-linux-amd64.tar.gz
[root@c_3 ~]# tar zxf flannel-0.5.5-linux-amd64.tar.gz
[root@c_3 ~]# mv flannel-0.5.5 /opt/flannel
[root@c_3 ~]#
[root@c_3 ~]# ll /opt/flannel/
total 16212
-rwxr-xr-x. 1 1000 1000 16581152 Nov 13 2015 flanneld
-rwxrwxr-x. 1 1000 1000 2008 Nov 13 2015 mk-docker-opts.sh
-rw-rw-r--. 1 1000 1000 11843 Nov 13 2015 README.md

添加一个flannel服务的System单元，简单的就可以
cat > /usr/lib/systemd/system/flanneld.service <<-'EOF'
[Unit]
Description=flannel
[Service]
ExecStart=/opt/flannel/flanneld \
　　-etcd-endpoints=http://192.168.3.50:4001
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl start flanneld && systemctl status flanneld

[root@c_3 ~]# ip a |grep flannel
4: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN group default qlen 500
inet 172.20.56.0/16 scope global flannel0

ot@c_3 opt]# etcdctl ls /coreos.com/network/subnets //etcd查看已经分配的网段
/coreos.com/network/subnets/172.20.6.0-24
/coreos.com/network/subnets/172.20.56.0-24

04、docker-ce修改配置

cd /opt/flannel

[root@c_3 flannel]# ./mk-docker-opts.sh --help
./mk-docker-opts.sh: illegal option -- -
./mk-docker-opts.sh [-f FLANNEL-ENV-FILE] [-d DOCKER-ENV-FILE] [-i] [-c] [-m] [-k COMBINED-KEY]

Generate Docker daemon options based on flannel env file
OPTIONS:
-f Path to flannel env file. Defaults to /run/flannel/subnet.env //flannel0网卡的配置信息
-d Path to Docker env file to write to. Defaults to /run/docker_opts.env
-i Output each Docker option as individual var. e.g. DOCKER_OPT_MTU=1500
-c Output combined Docker options into DOCKER_OPTS var //网络信息写入 /run/docker_opts.env
-k Set the combined options key to this value (default DOCKER_OPTS=)
-m Do not output --ip-masq (useful for older Docker version)

[root@c_3 flannel]# ./mk-docker-opts.sh -c    //创建并写入 /run/docker_opts.env
[root@c_3 flannel]# cat /run/docker_opts.env //保存环境键值对
DOCKER_OPTS=" --bip=172.20.56.1/24 --ip-masq=true --mtu=1472 "

docker使用flannel的网络传递数据修改docker启动参数
[root@c_3 ~]# systemctl cat docker | head -1 //查看docker读取的配置文件
# /usr/lib/systemd/system/docker.service

/usr/lib/systemd/system/docker.service
EnvironmentFile=-/run/docker_opts.env
ExecStart=/usr/bin/dockerd $DOCKER_OPTS

systemctl daemon-reload && systemctl restart docker

[root@c_3 flannel]# ip a|grep docker0 //docker0网络信息已经变化
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
inet 172.20.56.1/24 brd 172.20.56.255 scope global docker0

[root@c_3 flannel]# docker network ls
NETWORK ID NAME DRIVER SCOPE
3c6b97123f83 bridge bridge local
4501ec9545f6 host host local
ccd8eef9f50b none null local
[root@c_3 flannel]# docker network inspect 3c6b97123f83 |grep -i subnet
"Subnet": "172.20.56.1/24",

05、测试跨主机，容器通信

docker run --rm -it busybox sh   //各个host上运行

参考：

　　https://www.cnblogs.com/linjiqin/p/11549329.html