学习OpenStack之（6）：Neutron 深入学习之 OVS + GRE 之 Compute node 篇

0.环境

硬件环境见上一篇博客：学习OpenStack之（5）：在Mac上部署Juno版本OpenStack 四节点环境

OpenStack网络配置：一个tenant， 2个虚机

Type driver： GRE， Mechanism driver： OVS
一个public network： ext-net 和相应的subnet ext-subnet
一个VM network：demo-net 和相应的subnet：demo-subnet
一个router连接ext-subnet和demo-subnet

1、Compute 节点上networking组件

下面会用到OVS的两个重要命令：

ovs-vsctl：查询和更新ovs-vswitchd的配置
ovs-ofctl：查询和控制OpenFlow交换机和控制器

首先查询Compute节点上ovs-vswitchd的配置的配置：

root@compute1:/var/lib/nova# ovs-vsctl show

205a13a2-1ad6-4ae0-8c84-abed97444aa9

    Bridge br-int //OVS integration 桥 br-int

        fail_mode: secure

        Port "qvo37b25c08-e8" //端口，用来连接一个虚机网卡的TAP设备所连接的linux bridge

            tag:

            Interface "qvo37b25c08-e8"

        Port patch-tun //端口，用来连接桥br-tun

            Interface patch-tun

                type: patch

                options: {peer=patch-int} //和桥 br-tun上的patch-int是对等端口

        Port br-int

            Interface br-int

                type: internal

        Port "qvo155845ae-5e" //端口，用来连接另一个虚机网卡的TAP设备所连接的linux bridge

            tag:

            Interface "qvo155845ae-5e"

    Bridge br-tun //OVS Tunnel 桥br-tun

        Port br-tun

            Interface br-tun

                type: internal

        Port patch-int //端口patch-int，用来连接桥br-int

            Interface patch-int

                type: patch

                options: {peer=patch-tun}

        Port "gre-0a000115" //端口，连接GRE Tunnel

            Interface "gre-0a000115"

                type: gre

                options: {df_default="true", in_key=flow, local_ip="10.0.1.31", out_key=flow, remote_ip="10.0.1.21"}

    ovs_version: "2.0.2" //GRE Tunnel是点到点之间建立的，这头的IP为10.0.1.31，那头的IP地址为 10.0.1.21

继续看桥 br-tun：

root@compute1:/var/lib/nova# ovs-ofctl show br-tun

OFPT_FEATURES_REPLY (xid=0x2): dpid:0000f6b428614747

n_tables:, n_buffers:

capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP

actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE

 (patch-int): addr:3e:7b:d5:fa::8d //端口 patch-int的ID 是 1

     config:

     state:

     speed:  Mbps now,  Mbps max

 (gre-0a000115): addr:2a::b2::f3:5a //端口 gre-0a000115的ID 是 2

     config:

     state:

     speed:  Mbps now,  Mbps max

 LOCAL(br-tun): addr:f6:b4::::

     config:

     state:

     speed:  Mbps now,  Mbps max

OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=

每个虚机有个虚机网卡 eth0，eth0和host上的一个TAP设备连接，该TAP设备直接挂载在一个Linux Bridge上，该Linux Bridge和OVS integration bridge br-int相连。其实理想情况下，TAP设备能和OVS Integration Bridge 直接相连就好了，但是，因为OpenStack实现Security Group的需要，这里要多加一层Linux bridge。OpenStack使用Linux TAP设备上的iptables来实现Security Group规则，而OVS不支持直接和br-int桥相连的TAP设备上的iptables。通过查看虚机的libvirt XML定义文件 /var/lib/nova/instances/<instance-id>/libvirt.xml可以看出来虚机所连接的TAP设备：

<interface type="bridge">

      <mac address="fa:16:3e:fe:c7:87"/> //

      <source bridge="qbr37b25c08-e8"/> //虚机TAP设备所挂接的linux bridge

      <target dev="tap37b25c08-e8” /> //虚机所连接的interface 
</interface>

通过以上信息，我们可以画出compute 节点上的网络组建图：

2. Neutron使用TAP设备的iptables来实现Security groups

查看第一个虚机的TAP设备上的iptables：

root@compute1:/var/lib/nova# iptables -S | grep tap37b25c08-e8

-A neutron-openvswi-FORWARD -m physdev --physdev-out tap37b25c08-e8 --physdev-is-bridged -j neutron-openvswi-sg-chain

-A neutron-openvswi-FORWARD -m physdev --physdev-in tap37b25c08-e8 --physdev-is-bridged -j neutron-openvswi-sg-chain

-A neutron-openvswi-INPUT -m physdev --physdev-in tap37b25c08-e8 --physdev-is-bridged -j neutron-openvswi-o37b25c08-e

-A neutron-openvswi-sg-chain -m physdev --physdev-out tap37b25c08-e8 --physdev-is-bridged -j neutron-openvswi-i37b25c08-e

-A neutron-openvswi-sg-chain -m physdev --physdev-in tap37b25c08-e8 --physdev-is-bridged -j neutron-openvswi-o37b25c08-e

OpenStack Neutron在neutron-openvswi-sg-chain上实现security groups。在使用默认security group的情况下：

neutron-openvswi-o37b25c08-e 控制从虚机出去的traffic

-A neutron-openvswi-o37b25c08-e -p udp -m udp --sport 68 --dport 67 -j RETURN
-A neutron-openvswi-o37b25c08-e -j neutron-openvswi-s37b25c08-e
-A neutron-openvswi-o37b25c08-e -p udp -m udp --sport 67 --dport 68 -j DROP
-A neutron-openvswi-o37b25c08-e -m state --state INVALID -j DROP
-A neutron-openvswi-o37b25c08-e -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-o37b25c08-e -j RETURN
-A neutron-openvswi-o37b25c08-e -j neutron-openvswi-sg-fallback

neutron-openvswi-i37b25c08-e 控制进入虚机的traffic

-A neutron-openvswi-i37b25c08-e -m state --state INVALID -j DROP

-A neutron-openvswi-i37b25c08-e -m state --state RELATED,ESTABLISHED -j RETURN

-A neutron-openvswi-i37b25c08-e -s 10.0.0.116/ -p udp -m udp --sport  --dport  -j RETURN

-A neutron-openvswi-i37b25c08-e -p tcp -m tcp --dport  -j RETURN

-A neutron-openvswi-i37b25c08-e -p icmp -j RETURN

-A neutron-openvswi-i37b25c08-e -m set --match-set IPv48c0dc337-0a6d-4ad7- src -j RETURN

-A neutron-openvswi-i37b25c08-e -j neutron-openvswi-sg-fallback

使用下面的命令来添加一条secrutiy group 规则来允许使用TCP 22端口：

neutron security-group-rule-create --protocol tcp --port-range-min  --port-range-max  --direction ingress default

那么该TAP设备的iptables会出现下面的变化：

root@compute1:/var/lib/nova# iptables -S | grep

-A FORWARD -d 192.168.122.0/ -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

-A FORWARD -s 192.168.122.0/ -i virbr0 -j ACCEPT

-A neutron-openvswi-i155845ae- -p tcp -m tcp --dport  -j RETURN

-A neutron-openvswi-i155845ae- -p tcp -m tcp --dport  -j RETURN

-A neutron-openvswi-i37b25c08-e -p tcp -m tcp --dport  -j RETURN

-A neutron-openvswi-i37b25c08-e -p tcp -m tcp --dport  -j RETURN

3. OVS integration 桥 br-int添加和删除traffic的VLAN ID

每一个使用 neutron net-create 命令创建的network都有一个新的 VLAN ID.本例中因为只有一个network，所以VLAN ID是1，见ovsctl-vsctl show命令中的port tag值。

4. OVS Tunnel 桥 br-tun 处理 VLAN ID 和 Tunnel ID的转化

从以下OpenFlow rule tables可见两种ID的处理过程：

root@compute1:/var/lib/nova# ovs-ofctl dump-flows br-tun

NXST_FLOW reply (xid=0x4):

 cookie=0x0, duration=.036s, table=, n_packets=, n_bytes=, idle_age=, priority=,in_port= actions=resubmit(,) //从端口1及patch-int进来的traffic会被重新执行table 2的rule

 cookie=0x0, duration=.491s, table=, n_packets=, n_bytes=, idle_age=, priority=,in_port= actions=resubmit(,) //从端口2 即 gre 端口进来的traffic重新执行table 3

 cookie=0x0, duration=.939s, table=, n_packets=, n_bytes=, idle_age=, priority= actions=drop

 cookie=0x0, duration=.84s, table=, n_packets=, n_bytes=, idle_age=, priority=,dl_dst=:::::/::::: actions=resubmit(,) //重新执行table 20的rule

 cookie=0x0, duration=.745s, table=, n_packets=, n_bytes=, idle_age=, priority=,dl_dst=:::::/::::: actions=resubmit(,)

 cookie=0x0, duration=.307s, table=, n_packets=, n_bytes=, idle_age=, priority=,tun_id=0x1 actions=mod_vlan_vid:,resubmit(,) //从neutron node来的traffic，打上VLAN ID 1，重新执行table 10的 rule

 cookie=0x0, duration=.646s, table=, n_packets=, n_bytes=, idle_age=, priority= actions=drop

 cookie=0x0, duration=.495s, table=, n_packets=, n_bytes=, idle_age=, priority= actions=drop

 cookie=0x0, duration=.293s, table=, n_packets=, n_bytes=, idle_age=, priority= actions=learn(table=,hard_timeout=,priority=,NXM_OF_VLAN_TCI[..],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1 //学习规则 table 20，从port 1 即 patch-int发出

 cookie=0x0, duration=.093s, table=, n_packets=, n_bytes=, idle_age=, priority= actions=resubmit(,) //重新执行table 22的rule

 cookie=0x0, duration=.372s, table=, n_packets=, n_bytes=, idle_age=, hard_age=, dl_vlan= actions=strip_vlan,set_tunnel:0x1,output:,output:2 //去掉VLAN ID，打上TUNNEL ID 1 即 neutron 节点的TUNNEL ID，从端口2 即 gre 端口发出

 cookie=0x0, duration=.901s, table=, n_packets=, n_bytes=, idle_age=, priority= actions=drop

下一节将neutron节点。