QEMU KVM Libvirt(12): Live Migration

由于KVM的架构为

Libvirt –> qemu –> KVM

所以对于live migration有两种方式，一种是qemu + KVM自己的方式，一种是libvirt的方式，当然libvirt也是基于qemu+kvm的方式

qemu + KVM自己的方式为使用monitor

KVM Migration

KVM currently supports savevm/loadvm and offline or live migration Migration commands are given when in qemu-monitor (Alt-Ctrl-2). Upon successful completion, the migrated VM continues to run on the destination host.

Requirements

• The VM image is accessible on both source and destination hosts (located on a shared storage, e.g. using nfs).
• It is recommended an images-directory would be found on the same path on both hosts (for migrations of a copy-on-write image -- an image created on top of a base-image using "qemu-image create -b ...")
• The src and dst hosts must be on the same subnet (keeping guest's network when tap is used).
• Do not use -snapshot qemu command line option.
• For tcp: migration protocol

the guest on the destination must be started the same way it was started on the source.

The live migration process has the following steps:

1. The virtual machine instance is running on the source host.

2. The virtual machine is started on the destination host in the frozen listening mode. The parameters used are the same as on the source host plus the -incoming tcp:ip:port parameter, where ip specifies the IP address and port specifies the port for listening to the incoming migration. If 0 is set as IP address, the virtual machine listens on all interfaces.

3. On the source host, switch to the monitor console and use the migrate -d tcp:destination_ip:port command to initiate the migration.

4. To determine the state of the migration, use the info migrate command in the monitor console on the source host.

5. To cancel the migration, use the migrate_cancel command in the monitor console on the source host.

6. To set the maximum tolerable downtime for migration in seconds, use the migrate_set_downtime number_of_seconds command.

7. To set the maximum speed for migration in bytes per second, use the migrate_set_speed bytes_per_second command.

要进行live migration首先要存储共享，我们这里用nfs

在一台nfs server上安装

apt-get install nfs-kernel-server

export如下的文件夹

# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#               to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/home/cliu8/nfs    *(rw,sync,no_root_squash)

在source机器和destination机器上，都mount这个文件夹

mount 16.158.166.150:/home/cliu8/nfs /home/cliu8/migrate

在文件夹里面放入两个image

root@escto-bj-hp-z620:/home/cliu8/migrate# ls -l
total 3842908
-rwxr-xr-x 1 root root 1717567488 Jul 18 00:01 ubuntu-14.04.img
-rwxr-xr-x 1 root root 2217869312 Jul 17 22:13 ubuntutest.img

在source机器上启动

qemu-system-x86_64 -enable-kvm -name ubuntutest  -m 2048 -hda /home/cliu8/migrate/ubuntu-14.04.img -vnc :19 -net nic -net user

在dest机器上同样启动，但是多了listen的参数

qemu-system-x86_64 -enable-kvm -name ubuntutest  -m 2048 -hda /home/cliu8/migrate/ubuntu-14.04.img -vnc :19 -net nic -net user -incoming tcp:0:4444

打开source机器的monitor

运行migrate -d tcp:16.158.166.150:4444

这个时候info migrate，显示Migration status: active

等变成complete，则migration结束

这个时候，另一面的机器已经起来了。

Libvirt Migration

Network data transports

migration的时候的数据传输有两种方式：

Hypervisor native transport

所谓native的transport，就是依赖于hypervisor，也即KVM的自有的机制去做网络传输，不支持加密，可能对于某些hypervisor来讲，网络还需要特殊的配置。

libvirt tunnelled transport

依赖于libvirt的RPC的网络通路进行网络传输，支持加密。

只要下面的命令能够执行，就能够传输

virsh -c qemu+ssh://cliu8@16.158.166.150/system list --all
virsh -c qemu+tcp://popsuper1982/system list --all
virsh -c qemu+tls://popsuper1982/system list --all

缺点是有很多额外的性能损耗

Communication control paths/flows

看完了数据通路，我们再来看控制通路

Migration一般涉及三个较色，admin, source, destination

Managed direct migration

admin节点控制整个migration的全过程，admin既控制source, 也控制destination，然而source和destination之间并不交互，因而migration过程中，如果admin挂了，就失败了。

Managed peer to peer migration

admin节点仅仅和source交互，告诉source，你要migrate到destination，然后source控制整个过程，如果admin在这个过程中挂了，不影响接着进行migration

注意的是，admin登录source的credential信息和source登录destination的credential信息是不一样的。

Unmanaged direct migration

admin和source都不控制migration的过程，而是admin的libvirt直接调用hypervisor的控制器，让hypervisor自己进行migration。

Configuration file handling

A transient guest only exists while it is running, and has no configuration file stored on disk.

A persistent guest maintains a configuration file on disk even when it is not running.

The virsh command has two flags to influence this behaviour.

The --undefine-source flag will cause the configuration file to be removed on the source host after a successful migration.

The --persist flag will cause a configuration file to be created on the destination host after a successful migration.

我们首先需要使得source和destination之间的libvirt是相互通的。

在source机器上，16.158.166.197

root@escto-bj-hp-z620:/home/cliu8/certtool# ls -l
total 60
-r--r--r-- 1 root root 1204 Jul 17 20:17 certificate_authority_certificate.pem
-r--r--r-- 1 root root 1972 Jul 17 20:17 certificate_authority_key.pem
-r--r--r-- 1 root root   37 Jul 17 20:17 certificate_authority_template.info
-r--r--r-- 1 root root 1379 Jul 17 20:17 escto-bj-hp-z620_client_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:17 escto-bj-hp-z620_client_key.pem
-r--r--r-- 1 root root  139 Jul 17 20:17 escto-bj-hp-z620_client_template.info
-r--r--r-- 1 root root 1310 Jul 17 20:17 escto-bj-hp-z620_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:17 escto-bj-hp-z620_server_key.pem
-r--r--r-- 1 root root   91 Jul 17 20:17 escto-bj-hp-z620_server_template.info
-r--r--r-- 1 root root 1371 Jul 17 20:17 popsuper1982_client_certificate.pem
-r--r--r-- 1 root root 1972 Jul 17 20:17 popsuper1982_client_key.pem
-r--r--r-- 1 root root  135 Jul 17 20:17 popsuper1982_client_template.info
-r--r--r-- 1 root root 1306 Jul 17 20:17 popsuper1982_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:17 popsuper1982_server_key.pem
-r--r--r-- 1 root root   87 Jul 17 20:17 popsuper1982_server_template.info

root@escto-bj-hp-z620:/home/cliu8/certtool# tree --charset ASCII /etc/pki/
/etc/pki/
|-- CA
|   `-- cacert.pem -> /home/cliu8/certtool/certificate_authority_certificate.pem
|-- libvirt
|   |-- clientcert.pem -> /home/cliu8/certtool/escto-bj-hp-z620_client_certificate.pem
|   |-- private
|   |   |-- clientkey.pem -> /home/cliu8/certtool/escto-bj-hp-z620_client_key.pem
|   |   `-- serverkey.pem -> /home/cliu8/certtool/escto-bj-hp-z620_server_key.pem
|   `-- servercert.pem -> /home/cliu8/certtool/escto-bj-hp-z620_server_certificate.pem
`-- nssdb -> /var/lib/nssdb

配置/etc/libvirt/libvirtd.conf

为了方便测试，我们对tcp, tls不进行密码设置

listen_tls = 1

listen_tcp = 1

tls_port = "16514"

tcp_port = "16509"

unix_sock_group = "libvirtd"

unix_sock_ro_perms = "0777"

unix_sock_rw_perms = "0770"

auth_unix_ro = "none"

auth_unix_rw = "none"

auth_tcp = "none"

auth_tls = "none"

测试下面的命令都能通过

virsh -c qemu+ssh://cliu8@16.158.166.150/system list --all
virsh -c qemu+tcp://popsuper1982/system list --all
virsh -c qemu+tls://popsuper1982/system list --all

在destination机器上，16.158.166.150

root@popsuper1982:/home/cliu8/certtool# ls -l
total 60
-r--r--r-- 1 root root 1204 Jul 15 22:31 certificate_authority_certificate.pem
-r--r--r-- 1 root root 1972 Jul 15 22:28 certificate_authority_key.pem
-r--r--r-- 1 root root   37 Jul 15 22:26 certificate_authority_template.info
-r--r--r-- 1 root root 1379 Jul 16 00:27 escto-bj-hp-z620_client_certificate.pem
-r--r--r-- 1 root root 1968 Jul 16 00:25 escto-bj-hp-z620_client_key.pem
-r--r--r-- 1 root root  139 Jul 16 00:24 escto-bj-hp-z620_client_template.info
-r--r--r-- 1 root root 1310 Jul 17 20:09 escto-bj-hp-z620_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:07 escto-bj-hp-z620_server_key.pem
-r--r--r-- 1 root root   91 Jul 17 20:06 escto-bj-hp-z620_server_template.info
-r--r--r-- 1 root root 1371 Jul 17 20:14 popsuper1982_client_certificate.pem
-r--r--r-- 1 root root 1972 Jul 17 20:13 popsuper1982_client_key.pem
-r--r--r-- 1 root root  135 Jul 17 20:12 popsuper1982_client_template.info
-r--r--r-- 1 root root 1306 Jul 16 00:09 popsuper1982_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 16 00:06 popsuper1982_server_key.pem
-r--r--r-- 1 root root   87 Jul 16 00:05 popsuper1982_server_template.info
root@popsuper1982:/home/cliu8/certtool# tree --charset ASCII /etc/pki/
/etc/pki/
|-- CA
|   `-- cacert.pem -> /home/cliu8/certtool/certificate_authority_certificate.pem
|-- libvirt
|   |-- clientcert.pem -> /home/cliu8/certtool/popsuper1982_client_certificate.pem
|   |-- private
|   |   |-- clientkey.pem -> /home/cliu8/certtool/popsuper1982_client_key.pem
|   |   `-- serverkey.pem -> /home/cliu8/certtool/popsuper1982_server_key.pem
|   `-- servercert.pem -> /home/cliu8/certtool/popsuper1982_server_certificate.pem
`-- nssdb -> /var/lib/nssdb

virsh -c qemu+ssh://cliu8@16.158.166.197/system list --all
virsh -c qemu+tcp://escto-bj-hp-z620/system list --all
virsh -c qemu+tls://escto-bj-hp-z620/system list –all

在source机器上，启动一个虚拟机virsh start ubuntu-14.04

# virsh dumpxml ubuntu-14.04
<domain type='kvm' id='55'>
  <name>ubuntu-14.04</name>
  <uuid>0f0806ab-531d-6134-5def-c5b495529284</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/home/cliu8/migrate/ubuntu-14.04.img'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:11:9b:d5:11'/>
      <source bridge='ubuntu_br'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='18a45d7e-d96b-4b9e-9d92-dc9ff3ea77e0'/>
      </virtualport>
      <target dev='vnet8'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/16'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/16'>
      <source path='/dev/pts/16'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5908' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
</domain>

其中cdrom和usb的都应该去掉，才能migration成功。

# virsh migrate --verbose --live --persistent ubuntu-14.04 qemu+tcp://popsuper1982/system     
Migration: [100 %]