使用weave管理docker网络

weave简介

---

Weave creates a virtual network that connects Docker containers deployed across multiple hosts.

Applications use the network just as if the containers were all plugged into the same network switch, with no need to configure port mappings, links, etc. Services provided by application containers on the weave network can be made accessible to the outside world, regardless of where those containers are running. Similarly, existing internal systems can be exposed to application containers irrespective of their location.

Weave can traverse firewalls and operate in partially connected networks. Traffic can be encrypted, allowing hosts to be connected across an untrusted network.

With weave you can easily construct applications consisting of multiple containers, running anywhere.

Weave works alongside Docker's existing (single host) networking capabilities, so these can continue to be used by containers.

weave简单使用

---

sudo wget -O /usr/local/bin/weave https://raw.githubusercontent.com/zettio/weave/master/weave
sudo chmod a+x /usr/local/bin/weave

启动weave路由器，这个路由器其实也是在docker中启动的：

[root@h-46mow360 ~]# weave launch
Unable to find image 'zettio/weave' locally
3b3a3db2c186fccb5203dcc269b3febbbbf126591a7ebd8117a8a5250683749f

[root@h-46mow360 ~]# brctl show
bridge name bridge id       STP enabled interfaces
docker0     8000.56847afe9799   no      veth801050a
weave       8000.7afc2a03325e   no      vethwepl2146
[root@h-46mow360 ~]# docker ps
CONTAINER ID        IMAGE                           COMMAND                CREATED              STATUS              PORTS                                            NAMES
3b3a3db2c186        zettio/weave:git-a34e214201cb   "/home/weave/weaver    About a minute ago   Up About a minute   0.0.0.0:6783->6783/tcp, 0.0.0.0:6783->6783/udp   weave

在两台物理机上分别启动一个容器：

c1=$(weave run 10.0.3.3/24 -t -i -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /tmp/$(mktemp -d):/run systemd:systemd /usr/lib/systemd/systemd)

c2=$(weave run 10.0.3.5/24 -t -i -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /tmp/$(mktemp -d):/run systemd:systemd /usr/lib/systemd/systemd)

这个时候，两个容器之间是不通的，需要在两台weave的路由器之间建立连接：( if there is a firewall between $HOST1 and $HOST2, you must open port 6783 for TCP and UDP)

weave connect 10.33.0.9

这样，两台容器之间通了：

# nsenter --mount --uts --ipc --net --pid --target $(docker inspect --format "{{.State.Pid}}" "$c2")

-bash-4.2# ping -c 3 10.0.3.3

PING 10.0.3.3 (10.0.3.3) 56(84) bytes of data.

64 bytes from 10.0.3.3: icmp_seq=1 ttl=64 time=2.34 ms

64 bytes from 10.0.3.3: icmp_seq=2 ttl=64 time=1.52 ms

64 bytes from 10.0.3.3: icmp_seq=3 ttl=64 time=1.13 ms

--- 10.0.3.3 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2003ms

rtt min/avg/max/mdev = 1.131/1.667/2.345/0.505 ms

但容器内部还是无法与宿主机之间通信。如果想让容器与宿主机网络联通，需要将weave路由器与宿主机网络联通，即执行：  weave expose 10.0.3.102/24

 

weave其他特性

 

• 应用隔离：不同子网容器之间默认隔离的，即便它们位于同一台物理机上也相互不通；不同物理机之间的容器默认也是隔离的
• 物理机之间容器互通：weave connect $OTHER_HOST
• 动态添加网络：对于不是通过weave启动的容器，可以通过weave attach 10.0.1.1/24 $id来添加网络（detach删除网络）
• 安全性：可以通过weave launch -password wEaVe设置一个密码用于weave peers之间加密通信
• 与宿主机网络通信：weave expose 10.0.1.102/24，这个IP会配在weave网桥上
• 查看weave路由状态：weave ps
• 通过NAT实现外网访问docker容器

 

实现原理

 

Weave creates a network bridge on the host. Each container is connected to that bridge via a veth pair, the container side of which is given the IP address & netmask supplied in ‘weave run’. Also connected to the bridge is the weave router container.

A weave router captures Ethernet packets from its bridge-connected interface in promiscuous mode, using ‘pcap’. This typically excludes traffic between local containers, and between the host and local containers, all of which is routed straight over the bridge by the kernel. Captured packets are forwarded over UDP to weave router peers running on other hosts. On receipt of such a packet, a router injects the packet on its bridge interface using ‘pcap’ and/or forwards the packet to peers.

Weave routers learn which peer host a particular MAC address resides on. They combine this knowledge with topology information in order to make routing decisions and thus avoid forwarding every packet to every peer. The topology information captures which peers are connected to which other peers; weave can route packets in partially connected networks with changing topology.

Weave routers establish TCP connections to each other, over which they perform a protocol handshake and subsequently exchange topology information. These connections are encrypted if so configured. Peers also establish UDP “connections”, possibly encrypted, for the aforementioned packet forwarding. These “connections” are duplex and can traverse firewalls.

 

 

其他需要注意的问题

 

1. MTU considerations

 

MTU is the payload size of an ethernet frame. The ethernet headers are

not included in this and add a further 14 bytes (an ethernet header is

really at least 22 bytes. However only 14 bytes actually carry data -

the two mac fields and a type/length field. The other parts of the

header are there for signalling at the electrical layer and thus don't

get captured by pcap, nor need forwarding). We are then passing this

over UDP which adds a further 8 bytes, and over IP, which adds at

least another 20 bytes of headers (IP can have variable header

length. Yay). We also include the namehash as a 32 byte prefix.

 

Therefore, we have 20(ip) + 8(udp) + 32(namehash) + 2(length prefix) +

14(ethernet) = 76 bytes of overhead. Thus we should make sure the

internal MTU is 76 bytes lower than the external MTU. So on a normal

1500 byte ethernet MTU, we should ensure the other containers use

nothing greater than 1424.

 

2. 重启容器

 

如果使用weave，则就不能再使用docker自带的auto-restart feature（如docker run --restart=always redis），因为weave是在docker之外为容器配置的网络，容器重启的时候docker本身不会做这些事情。因而，还需额外的工具来管理容器的状态（比如systemd, upstart等），这些工具要调用weave命令（weave run/start/attach）来启动容器。

3. 为了便于开机自动启动weave，可以创建一个weave.service：

 

[Unit]

Description=Weave Network

Documentation=http://zettio.github.io/weave/After=docker.service

 

[Service]
ExecStartPre=/usr/local/bin/weave launch
ExecStart=/usr/bin/docker logs -f weave
SuccessExitStatus=2
ExecStop=/usr/local/bin/weave stop

 

[Install]
WantedBy=multi-user.target

 

参考文档 https://zettio.github.io/weave/