QEMU Networking

QEMU Networking

QEMU has a number of really nice ways to set up networking for its guests. It can be a little bewildering to figure out how each of the options work, so I thought I'd write up what I found. Excuse the 'orrid ascii art :-)

VLANs and NICs

A VLAN is a network switch running in the context of a qemu process. There can be multiple vlans in each qemu process. Multiple interfaces can be connected to each vlan. Anything sent via one of the interfaces on a vlan is received by all the other interfaces on that vlan.

The most obvious type of interface which you can create is an interface which looks just like a specific network card to the guest OS. Any frames sent by the guest OS over that nic will appear on the vlan which the nic is a member of. Any frames sent to the vlan from other interfaces will be received by the guest OS via the nic.

         +-----------+
         |   Guest   |
         |    OS     |
         |   +---+   |
         |   |NIC|   |
         +---+-+-+---+
               ^           +--------+
               |           |        +<------> Other IF
               +---------->+  VLAN  |
                           |        +<------> Other IF
                           +--------+

So, for example, if you do:

 $> qemu -net nic,model=rtl8139,vlan=1,macaddr=52:54:00:12:34:56 ...

It will create a rtl8139 nic in the guest, with the given MAC address, and connect to vlan numbered 1. You can then use further -net options to connect other interfaces to that vlan.

Connecting VLANs Together

One possibility is to connect together vlans from multiple qemu processes. The way this works is that one qemu process connects to a socket in another qemu process. When a frame appears on the vlan in the first qemu process, it is forwarded to the corresponding vlan in the other qemu process and vice-versa.

 +-----------+                                      +-----------+
 |   Guest   |                                      |   Guest   |
 |     A     |                                      |     B     |
 |   +---+   |                                      |   +---+   |
 |   |NIC|   |                                      |   |NIC|   |
 +---+-+-+---+                                      +---+-+-+---+
       ^       +--------+                +--------+       ^
       |       |        |                |        |       |
       +------>+ VLAN 0 +<--> Socket <-->+ VLAN 2 +<------+
               |        |                |        |
               +--------+                +--------+

For example, you might start Guest A with:

 $> qemu -net nic -net socket,listen=:8010 ...

This qemu process is hosting a guest with a nic connected to VLAN 0, which in turn has a socket interface listening for connections on port 8010.

You could then start Guest B with:

 $> qemu -net nic,vlan=2 -net socket,vlan=2,connect=127.0.0.1:8010 ...

This qemu process would then have a guest with a nic connected to VLAN 2, which in turn has a socket interface connected to VLAN 0 in the first qemu process.

Thus, any frames transmitted by Guest A is received by Guest B, and vice versa.

(Note, the two vlans do not need to be numbered differently. It is only done here for illustrative purposes)

An extension of this concept is to connect vlans together using a multicast socket. For example, with:

 $> qemu -net nic -net socket,mcast=230.0.0.1:1234 ...
 $> qemu -net nic -net socket,mcast=230.0.0.1:1234 ...

you have two guests with their vlan 0 connected together over a multicase bus. Any number of guests can connect to the same multicast address and receive the frames sent by any guest to that vlan.

It's interesting to note that any of the above can be done by unpriviledged users.

Connecting VLANs To TAP Devices

Another option is to make a vlan available through a device in the host OS. Any frames transmitted via this device will appear on a vlan in the qemu process (and thus received by another other interfaces on the vlan) and frames sent to the vlan will be received by the device.

 +-----------+                            +-------+
 |   Guest   |                            |  TAP  |
 |    OS     |                            | Device|
 |   +---+   |                            |(qtap0)|
 |   |NIC|   |                            +---+---+
 +---+-+-+---+                                |
       ^       +------+                   +---+-----+
       |       |      |                   | Kernel  |
       +------>+ VLAN +<-->   File    <-->+ TUN/TAP |
               |      |     Descriptor    | Driver  |
               +------+                   +---------+

  $> qemu -net nic -net tap,ifname=qtap0 ...

This works using the kernel's TUN/TAP device driver. This driver basically allows a user-space application to obtain a file descriptor which is connected to a network device. Any frames sent to the kernel over the file descriptor will be received by the device and any frames transmitted via the device will be received by the application.

If you e.g. assign an IP address to the TAP device, applications in the guest will be able to connect to applications in the host listening for connections on that IP address. And if you enable port forwarding in the host, packets sent from the guest can be forwarded by the host kernel to the Internet.

Essentially, the TAP device looks just like a network device connected to a physical network to which the guest is also connected.

TAP devices are obtained by opening /dev/net/tun and invoking the TUNSETIFF ioctl(). This is not usually allowed for unpriviledged users so, in general, only root can use this method.

Connecting VLANs Using VDE

An extension of this idea is possible using VDE (Virtual Distributed Ethernet). This is a user-space program which can obtain a TAP device and allow a number of other programs to connect to it and bridge those connections to the TAP device. It would be quite similar in effect to connecting multiple qemu vlans together and connecting one of those vlans to a TAP device.

    +-----------+                           +-----------+
    |   Guest   |                           |   Guest   |
    |     A     |                           |     B     |
    |   +---+   |                           |   +---+   |
    |   |NIC|   |                           |   |NIC|   |
    +---+-+-+---+                           +---+-+-+---+
          ^       +------+         +------+       ^
          |       |      | +-----+ |      |       |
          +------>+ VLAN +-+ VDE +-+ VLAN +<------+
                  |      | +--+--+ |      |
                  +------+    |    +------+
                              |
                          +---+-----+  +--------+
                          | Kernel  |  |  TAP   |
                          | TUN/TAP +--+ Device |
                          | Driver  |  | (qtap) |
                          +---------+  +--------+

 $> vde_switch -hub -tap qtap -sock /var/run/qtap-ctl
 $> vqeq -vdesock /var/run/qtap-ctl qemu -net nic ...
 $> vqeq -vdesock /var/run/qtap-ctl qemu -net nic ...

Here we end up with the vde_switch receiving packets from the vlans in both qemu processes and forwarding to the qtap network interface in the host, and likewise, forwarding packets from the qtap device to both vlans. Since both guests have nics on these vlans, they can transmit/receive frames to/from the host OS and each other.

Now, if you think something as hairy as that hasn't been since the woolly mammoth, wait until you see the last option ...

Usermode Network Stack

QEMU's final, and most bizarre, networking option is also its default option. What this does is connect a "usermode network stack" to a vlan. This network stack is a standalone implementation of the ip, tcp, udp, dhcp and tftp (etc.) protocols. It can handle frames from the vlan by e.g. responding to dhcp requests with a valid address, responding to tftp requests with a file from the host filesystem or by creating udp/tcp sockets over which packet data can be forwarded.

  $> qemu -net nic,vlan=1 -net user,vlan=1 ...

Note that this network stack is running within the qemu process itself. So, for example there is no separate dhcp or tftp process handling those requests. Also, the stack is effectively acting as a proxy by unpacking application data from udp/tcp packets and forwarding them over a socket connecting the qemu process and the destination process.

Although bizarre, this option does provide a very useful default in that the guest OS will have largely transparent network access almost like any other application running on the host.