Linux Virtualization with Xen

Xen is the new virtualization kid on the block. It's gaining visibility and importance at a speed only projects such as Linux and Apache have seen before. Xen has been around for a couple of years: it was originally part of the Xenoserver platform, which aimed to build a public infrastructure for wide-area distributed computing. Ian Pratt, the principal investigator of the Xenoserver project at the University of Cambridge Computer Laboratory, still leads the development team.

Xen ended up being much more than a part of this project. Now many Linux distributions and some hardware vendors are picking it up. As with many important open source projects these days, it even has a company--Xensource--backing commercial versions and providing support for corporate customers. Xensource also employs several industry veterans. In short, Xen(source) has everything a good open source platform need to becomes an extremely important player in the industry.

At X-Tend, one of our main problems was that we didn't have enough machines to test all new distributions and applications. Basically there was no financially realistic way to provide our users with a quick test environment. My guess is that half of the planet has similar problems.

Ages ago, we used UserModeLinux, but most new users found it too complex to use. Later, we bootstrapped Qemu instances from our central imaging server. That worked. If we wanted to do some tests on an isolated environment, we quickly started Qemu with the distribution we needed. The only annoyance was that we actually wanted an environment that was constantly online and, in the event of a power outage (this is a test environment, not production), we didn't have to spend too much time getting it back. It had to be scriptable and automatable, and preferably would not require X.

With the arrival of Xen last year, all of that changed. This article describes how we tackled our problem and how we actually now have a stable and performant environment to test everything we want. It's so stable, we now use Xen for production environments!

What's Xen?

Xen is a virtual machine monitor for x86 that supports the execution of multiple guest operating systems with unprecedented levels of performance and resource isolation. Xen is open source software, released under the terms of the GNU General Public License.

Xen has become one of the most popular virtualization platforms during the last six months. Although it's not such a young project, it is now gaining acceptance in the corporate world as a valuable alternative to VMWare.

Adding Xen to your machine changes it from an ordinary x86 machine to a totally new platform. It's not an x86 anymore. It's a Xen machine. All the operating systems that you want to run on your machine won't work anymore if they know only aboutx86; they need to know about Xen. Of course, the Xen and x86 architecture are really similar, so for the end user and the applications that run on a platform ported to Xen, there is almost no difference.

When Xen is activated, it will also need to boot its first virtual machine, called Domain0. Domain0 has more privileges than the other virtual machines and typically is used only for managing the other (less privileged) virtual machines. Domain0 is also responsible for managing the hardware. Porting a platform to Xen changes almost nothing to the drivers, which means that most drivers supported in traditional Linux kernels also work in Xen.

Within Domain0, the xend daemon handles the management of the virtual machines. Control it via the xm command-line utility.

From there, you can create other virtual machines, or domains.

Xen and Different Distributions

We've been running Xen on different platforms ranging from an "antique" Suse 8.2 with a 2.4 series kernel, a Debian box, and Fedora Core 4 with a fresh 2.6 kernel. Unlike some other projects, Xen currently doesn't care whether you use 2.4 or 2.6, so people who are comfortable with a 2.4 kernel can still benefit from the Xen features. However, future releases probably won't have 2.4 support. People claim that installing Xen is difficult, but it's not, certainly if you compare it with other similar tools. GetXen.org is the place to start; it contains a tarball with most of the required binaries and tools, a demo CD, and pointers to the source code. Some distributions such as Fedora include prebuilt packages. As of this writing, the official stable Xen release is 2.0.7, but most people are already working with the 3.0 betas. 3.0 might be out by the time you actually read this.

It's really easy to start. Here's how we deployed a Debian virtual machine on a Fedora Core 4 install. We opted for a minimal FC4 install. After the installation, we updated, upgraded, and installed Xen with a couple of small commands:

$ yum update
$ yum install xen
$ yum install kernel-xen0
$ yum install kernel-xenU

Can it get easier? You should now carefully inspect your grub.conf file and find a part similar to:

title Xen 2.0 / XenLinux 2.6.9
  kernel /boot/xen.gz dom0_mem=131072
  module /boot/vmlinuz-2.6.9-xen0 root=/dev/hda1 ro console=tty0

Your version numbers may vary. If that's there, then it's time to reboot into that new entry. Voilá, you now have your first virtual machine up and running. Yes, at first sight the regular Linux version you have just booted into isn't running on a regular x86 anymore but is running on a Xen.

If you already started xend at boot time, run xm list to see output similar to:

HOSTA:/etc/xen/scripts # xm list
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----     41.2

Building a Virtual Host

Your next step is to create another virtual machine. The easiest way to do this is either to download an existing chroot image of the distribution you like or to build one yourself. Xen can use file-backed virtual block devices (dd if=/dev/zero of=vmdisk bs=1k seek 2048k count=1), physical devices (the actual /dev/hda9), LVM volumes (phy:VolumeGroup/root_volume), or an NFS root for your virtual machines. I prefer to use logical volumes on my machines, as they are really flexible to work with. With an existing disk /dev/sda5 available, I created logical volumes to use in my virtual machine:

$ pvcreate /dev/sda5
$ vgcreate vm_volumes /dev/sda5
$ vgchange -a y vm_volumes
$ lvcreate -L4096 -nroot.dokeos vm_volumes
$ lvcreate -L2048 -nvar.dokeos vm_volumes
$ lvcreate -L256 -nswap.dokeos vm_volumes
$ lvcreate -L1024 -nwww.dokeos vm_volumes

I usually create a directory /vhosts on my dom0 host where I mount my partitions. From there, I install the first FC4 base packages in a chroot on the actual future root device.

$ yum --installroot=/vhosts/root.dokeos/ -y groupinstall Base

You need to make a couple of quick fixes to make sure that you can open your initial console and so forth:

$ MAKEDEV -d /path/dev -x console
$ MAKEDEV -d /path/dev -x null
$ MAKEDEV -d /path/dev -x zer

It's almost ready. Now you need the configuration file for this virtual machine. Most of Xen's config files live in /etc/xen. You need a separate config file for each virtual machine you want to deploy on your host. They look like:

[root@xen xen]# cat dokeos.x-tend.be
kernel = "/boot/vmlinuz-2.6.11-1.1366_FC4xenU"
memory = 128
name   = "dokeos.x-tend.be"
nics   = 1
extra  = "selinux=0  3"

vif    = ['ip = "10.0.11.13", bridge=xen-br0']

disk   = ['phy:vm_volumes/root.dokeos,sda1,w'
         ,'phy:vm_volumes/var.dokeos,sda3,w'
         ,'phy:vm_volumes/www.dokeos,sda4,w'
         ,'phy:vm_volumes/swap.dokeos,sda2,w'
         ]

root   = "/dev/sda1 ro"

The config file is rather straightforward, and the Xen packages include examples. Now start your virtual machine with the command xm create config file. Add a -c to that command to see the machine booting. You should get a login prompt within seconds. That's how fast a physical machine should also boot, but I'll keep on dreaming for a couple of years.

If you create a symlink to the /etc/xen/auto directory, your virtual machines will start at boot time, if you enable thexendomains script at boot time