openstack(liberty): 简单网络连接图

openstack起初的网络部分是和计算核心nova合在一起的，后来被拆分出来，独立成为一个模块, 现在名为Neutron.

本博文是学习记录，记录的是基于GRE tunnel技术的neutron和计算主机之间的连接和通信流程。下面直接上图，结合图做一些简单的介绍（这些介绍的内容，主要来自RDO。）

在介绍上面这个图之前，需要简单的介绍一下，什么是GRE tunnel。

• GRE（Generic Routing Encapsulation）：通用路由封装协议。
• GRE是VPN的第三层隧道协议，采用了隧道（Tunnel）技术。
• Tunnel是虚拟的点对点连接，即点对点连接的虚拟接口。

至于更详细的内容，可以google。

下面就对neutron网络做简单的介绍，了解这个后，对openstack的深入学习，会有很大的帮组。因为我个人觉得，云计算里面，网络的虚拟化和管理，是相对比较难的一块，比起存储和计算的虚拟化要难。

网络连接图的Compute Host中，qbr是Linux bridge，虚拟出来的。这个主要是处理openstack的security group的，虽然openstack的网络核心部分采用的是open vswitch，但是这个和iptables规则不兼容，所以，安全策略就落在了虚拟出来的Linux bridge上了。后续，这个linux bridge可以看作firewall bridge。

在具体介绍之前，我先查看下我的测试环境devstack中的网络设备之间的连接关系：

 [root@ip---- ~]# ovs-vsctl show
 56f3ae0a-87b4--bc79-9a92a76590e3
     Bridge br-ex
         Port "qg-5d13ee78-0a"
             Interface "qg-5d13ee78-0a"
                 type: internal
         Port br-ex
             Interface br-ex
                 type: internal
     Bridge br-tun
         fail_mode: secure
         Port patch-int
             Interface patch-int
                 type: patch
                 options: {peer=patch-tun}
         Port br-tun
             Interface br-tun
                 type: internal
     Bridge br-int
         fail_mode: secure
         Port "qr-c36567ef-4d"
             tag:
             Interface "qr-c36567ef-4d"
                 type: internal
         Port patch-tun
             Interface patch-tun
                 type: patch
                 options: {peer=patch-int}
         Port "qr-2e4c72e5-5f"
             tag:
             Interface "qr-2e4c72e5-5f"
                 type: internal
         Port br-int
             Interface br-int
                 type: internal
         Port "tapf2d4361c-39"
             tag:
             Interface "tapf2d4361c-39"
                 type: internal
     ovs_version: "2.4.0"

Compute host: instance networking (A,B,C)

An outbound packet starts on eth0 of the virtual instance, which is connected to a tap device on the host, tap7c7ae61e-05. This tap device is attached to a Linux bridge device, qbr7c7ae61e-05.

Ideally, the TAP device vnet0 would be connected directly to the integration bridge, br-int. Unfortunately, this isn't possible because of how OpenStack security groups are currently implemented. OpenStack uses iptables rules on the TAP devices such as vnet0 to implement security groups, and Open vSwitch is not compatible with iptables rules that are applied directly on TAP devices that are connected to an Open vSwitch port.

Compute Host图中，A， Q都是vnet0，也就是虚拟机的网络接口，B/C 以及 R/S都是Linux bridge的tap设备，防火墙的规则就是设置在linux bridge （qbr）的对应tap上的。

Compute host: integration bridge (D,E)

The integration bridge, br-int, performs VLAN tagging and un-tagging for traffic coming from and to your instances.

这个过程，br-int相当于一个VLAN的处理机，将从没有加tag的来自VM1的数据从入口D接收后，进行加tag，然后发向E口，相反，会将带有tag的vlan上的数据从E口接收后，再去tag，根据VLAN的端口映射规则将数据转发到D(或者是去向其他的VM，例如VM2，则是转发到T口)

而对于T/E的数据处理，和D/E的处理流程类似。区别主要在D和T对应的VLAN ID会不同而已。

Compute host: tunnel bridge (F,G)

The tunnel bridge translates VLAN-tagged traffic from the integration bridge into GRE tunnels. The translation between VLAN IDs and tunnel IDs is performed by OpenFlow rules installed on br-tun.

在这里就体现出Neutron网络中的GRE tunnel机制了。就是G和Network Host的H口之间，将会建立GRE Tunnel。

Network host: tunnel bridge (H,I)

Traffic arrives on the network host via the GRE tunnel attached to br-tun. This bridge has a flow table very similar to br-tun on the compute host。

这个过程，非常类似上一个环节，只是数据处理的方向是反的，是从tunnel数据转向vlan的。

Network host: integration bridge

The integration bridge on the network controller serves to connect instances to network services, such as routers and DHCP servers.

Network host: DHCP server (O,P)

Each network for which DHCP is enabled has a DHCP server running on the network controller. The DHCP server is an instance of dnsmasq running inside a network namespace. A network namespace is a Linux kernel facility that allows groups of processes to have a network stack (interfaces, routing tables, iptables rules) distinct from that of the host.

这个namespace，在openstack环境下，可以通过ip netns查看，我的devstack测试平台上，执行这个命令得到下面的内容：

 [root@ip---- ~]# ip netns
 qrouter-a7bf0605--4f5c-bcbd-98a11bd175d4
 qdhcp-77e45b4b--467e-b38f-33dd86a5f83b

上面的qdhcp namespace的server主要是服务于private网络的IP地址分配的。而qrouter则主要负责数据对外的路由的。

在dhcp的namespace里面执行查看地址的命令，如下，看看都是什么信息呢？

 [root@ip---- ~]# ip netns exec qdhcp-77e45b4b--467e-b38f-33dd86a5f83b ip addr
 : lo: <LOOPBACK,UP,LOWER_UP> mtu  qdisc noqueue state UNKNOWN
     link/loopback ::::: brd :::::
     inet 127.0.0.1/ scope host lo
        valid_lft forever preferred_lft forever
     inet6 ::/ scope host
        valid_lft forever preferred_lft forever
 : tapf2d4361c-: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc noqueue state UNKNOWN
     link/ether fa::3e::fa:a0 brd ff:ff:ff:ff:ff:ff
     inet 10.0.0.2/ brd 10.0.0.255 scope global tapf2d4361c-
        valid_lft forever preferred_lft forever
     inet6 fdf8:e668:72e6::f816:3eff:fe85:faa0/ scope global mngtmpaddr dynamic
        valid_lft 86354sec preferred_lft 14354sec
     inet6 fe80::f816:3eff:fe85:faa0/ scope link
        valid_lft forever preferred_lft forever

You can find the dnsmasq process associated with this namespace by search the output of ps for the id (the number after qdhcp- in the namespace name)

 [root@ip---- ~]# ps -fe | grep 77e45b4b--467e-b38f-33dd86a5f83b
 nobody            Jan12 ?        :: dnsmasq --no-hosts --no-resolv --strict-order --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/77e45b4b--467e-b38f-33dd86a5f83b/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/77e45b4b--467e-b38f-33dd86a5f83b/host --addn-hosts=/opt/stack/data/neutron/dhcp/77e45b4b--467e-b38f-33dd86a5f83b/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/77e45b4b--467e-b38f-33dd86a5f83b/opts --dhcp-leasefile=/opt/stack/data/neutron/dhcp/77e45b4b--467e-b38f-33dd86a5f83b/leases --dhcp-match=set:ipxe, --bind-interfaces --interface=tapf2d4361c- --dhcp-range=set:tag0,10.0.0.0,static,86400s --dhcp-lease-max= --conf-file= --domain=openstacklocal

Network host: Router (M,N)

A Neutron router is a network namespace with a set of routing tables and iptables rules that performs the routing between subnets.

下面，通过ip netns来看看router namespace下的地址信息（router的namespace，前面已经列举出来了的）：

 [root@ip---- ~]# ip netns exec qrouter-a7bf0605--4f5c-bcbd-98a11bd175d4 ip addr
 : lo: <LOOPBACK,UP,LOWER_UP> mtu  qdisc noqueue state UNKNOWN
     link/loopback ::::: brd :::::
     inet 127.0.0.1/ scope host lo
        valid_lft forever preferred_lft forever
     inet6 ::/ scope host
        valid_lft forever preferred_lft forever
 : qr-c36567ef-4d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc noqueue state UNKNOWN
     link/ether fa::3e::fa: brd ff:ff:ff:ff:ff:ff
     inet 10.0.0.1/ brd 10.0.0.255 scope global qr-c36567ef-4d
        valid_lft forever preferred_lft forever
     inet6 fe80::f816:3eff:fe54:fa80/ scope link
        valid_lft forever preferred_lft forever
 : qg-5d13ee78-0a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc noqueue state UNKNOWN
     link/ether fa::3e::7c: brd ff:ff:ff:ff:ff:ff
     inet 172.24.4.2/ brd 172.24.4.255 scope global qg-5d13ee78-0a
        valid_lft forever preferred_lft forever
     inet6 :db8::/ scope global
        valid_lft forever preferred_lft forever
     inet6 fe80::f816:3eff:fe37:7c95/ scope link
        valid_lft forever preferred_lft forever
 : qr-2e4c72e5-5f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc noqueue state UNKNOWN
     link/ether fa::3e:cd::ad brd ff:ff:ff:ff:ff:ff
     inet6 fdf8:e668:72e6::/ scope global
        valid_lft forever preferred_lft forever
     inet6 fe80::f816:3eff:fecd:92ad/ scope link
        valid_lft forever preferred_lft forever

这个当中，可以看看其路由信息：

 [root@ip---- ~]# ip netns exec qrouter-a7bf0605--4f5c-bcbd-98a11bd175d4 ip route
 default via 172.24.4.1 dev qg-5d13ee78-0a
 10.0.0.0/ dev qr-c36567ef-4d  proto kernel  scope link  src 10.0.0.1
 172.24.4.0/ dev qg-5d13ee78-0a  proto kernel  scope link  src 172.24.4.2 

nat表的信息如下：

 [root@ip---- ~]# ip netns exec qrouter-a7bf0605--4f5c-bcbd-98a11bd175d4 iptables -t nat -S
 -P PREROUTING ACCEPT
 -P INPUT ACCEPT
 -P OUTPUT ACCEPT
 -P POSTROUTING ACCEPT
 -N neutron-l3-agent-OUTPUT
 -N neutron-l3-agent-POSTROUTING
 -N neutron-l3-agent-PREROUTING
 -N neutron-l3-agent-float-snat
 -N neutron-l3-agent-snat
 -N neutron-postrouting-bottom
 -A PREROUTING -j neutron-l3-agent-PREROUTING
 -A OUTPUT -j neutron-l3-agent-OUTPUT
 -A POSTROUTING -j neutron-l3-agent-POSTROUTING
 -A POSTROUTING -j neutron-postrouting-bottom
 -A neutron-l3-agent-POSTROUTING ! -i qg-5d13ee78-0a ! -o qg-5d13ee78-0a -m conntrack ! --ctstate DNAT -j ACCEPT
 -A neutron-l3-agent-PREROUTING -d 169.254.169.254/ -i qr-+ -p tcp -m tcp --dport  -j REDIRECT --to-ports
 -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
 -A neutron-l3-agent-snat -o qg-5d13ee78-0a -j SNAT --to-source 172.24.4.2
 -A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 172.24.4.2
 -A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat

Network host: External traffic (K,L)

"External" traffic flows through br-ex via the qg-5d13ee78-0a interface in the router name space, which connects to br-ex。