OpenStack nova VM migration (live and cold) call flow

OpenStack nova compute supports two flavors of Virtual Machine (VM) migration:

• Cold migration -- migration of a VM which requires the VM to be powered off during the migrate operation during which time the VM is inaccessible.
• Hot or live migration -- zero down-time migration whereupon the VM is not powered off during the migration and thus remains accessible.

Understanding these VM migration operations from an OpenStack internals perspective can be a daunting task. I had the pleasure of digging into these flows in the latter part of 2013 and as part of that effort created a rough outline of the internal flows. Other's I've worked with found these flow outlines useful and thus they're provided below.

Note -- The outlines below were created based on the OpenStack source in late 2013 and thus reflect the state of OpenStack at that point in time.

Live Migration Flow:

• nova.api.openstack.compute.contrib.admin_actions._migrate_live()
• nova.compute.api.live_migrate()
  • update instance state to MIGRATING state
  • call into scheduler to live migrate (scheduler hint will be set to the host select (which may be none)).
• nova.scheduler.manager.live_migration()
• nova.scheduler.manager._schedule_live_migration()
• nova.conductor.tasks.live_migrate.LiveMigrationTask.execute()
  • check that the instance is running
  • check that the instance's host is up
  • if destination host provided, check that it..
    1. is different than the instance's host
    2. is up
    3. has enough memory
    4. is compatible with the instance's host (i.e. hypervisor type and version)
    5. passes live migration checks (call using amqp rpc into nova manager check_can_live_migrate_destination)
  • else destination host not provided, find a candidate destination host and check that it...
    1. is compatible with the instance's host (i.e. hypervisor type and version)
    2. passes live migration checks (call using amqp rpc into nova manager check_can_live_migrate_destination)
  • call using amqp rpc into nova manager live_migration
    Note: Migration data is initially set by check_can_live_migrate_destination and can be used for implementation specific parameters from this point.
• nova.compute.manager.check_can_live_migrate_destination()
  • driver.check_can_live_migrate_destination()
  • call using amqp rpc into nova manager check_can_live_migrate_source
  • driver.check_can_live_migrate_destination_cleanup()
• nova.compute.manager.check_can_live_migrate_source()
  • determine if the instance is volume backed and add result to the migration data
  • driver.check_can_live_migrate_source()
• nova.compute.manager.live_migration()
  • if block migration request then driver.get_instance_disk_info()
  • call using amqp rpc into nova manager pre_live_migration
    • Error handler: _rollback_live_migration
  • driver.live_migration()
• nova.compute.manager.pre_live_migration()
  • get the block device information for the instance
  • get the network information for the instance
  • driver.pre_live_migration()
  • setup networks on destination host by calling the network API setup_networks_on_host
  • driver.ensure_filtering_rules_for_instance()
• nova.compute.manager._rollback_live_migration()
  • update instance state to ACTIVE state
  • re-setup networks on source host by calling the network API setup_networks_on_host
  • for each instance volume connection call using amqp rpc into nova manager remove_volume_connection
  • if block migration or volume backed migration without shared storage
    • call using amqp rpc into nova manager rollback_live_migration_at_destination
• nova.compute.manager._post_live_migration()
  • driver.get_volume_connector()
  • for each instance volume connection call the volume API terminate_connection
  • driver.unfilter_instance()
  • call into conductor to network_migrate_instance_start which will eventually call the network API migrate_instance_start
  • call using amqp rpc into nova manager post_live_migration_at_destination
  • if block migration or not shared storage driver.destroy()
  • else driver.unplug_vifs()
  • tear down networks on source host by calling the network API setup_networks_on_host
• nova.compute.manager.post_live_migration_at_destination()
  • setup networks on destination host by calling the network API setup_networks_on_host
  • call into conductor to network_migrate_instance_finish which will eventually call the network API migrate_instance_finish
  • driver.post_live_migration_at_destination()
  • update instance to ACTIVE state
  • setup networks on destination host by calling the network API setup_networks_on_host
• nova.compute.manager.rollback_live_migration_at_destination()
  • tear down networks on destination host by calling the network API setup_networks_on_host
  • driver.destroy()
• nova.compute.manager.remove_volume_connection()
  • call _detach_volume
  • driver.get_volume_connector()
  • remove the volume connection by calling the volume API terminate_connection
• nova.compute.manager._detach_volume()
  • driver.detach_volume()
    • Since the live migration failed the VM should not be on the destination host.  So this should be a no-op.
  • If there is an exception detaching the volume then rollback the detach by calling the volume API roll_detaching

 

Cold Migration Flow:

• nova.api.openstack.compute.servers._resize()
• nova.api.openstack.compute.contrib.admin_actions._migrate()
• nova.compute.api.resize()
  • if flavor_id is not passed, migrate host only and keep the original flavor
  • else flavor_id is given, migrate host and resize to new flavor
  • lookup the image for the instance by calling the image API show
  • check quota headroom and reserve
  • update instance to RESIZE_PREP state
  • determine if the instance's current host should be ignored as a migration target and update filter properties for the scheduler accordingly
  • call into scheduler to prep_resize
• nova.scheduler.manager.prep_resize()
  • call scheduler driver to schedule_prep_resize
  • if no valid host was found then update instance to ACTIVE state and rollback quota reservation
  • if error occurred then update instance to ERROR state and rollback quota reservation
• nova.scheduler.filter_scheduler.schedule_prep_resize()
  • run through scheduler filters to select host
  • call using amqp rpc into nova manager prep_resize
• nova.compute.manager.prep_resize()
  • if no node specified call driver.get_available_nodes()
  • call _prep_resize
    • if an exception occurs then call into scheduler to prep_resize again if possible
• nova.compute.manager._prep_resize()
  • if same host is used then ensure that the same host is allowed (as per configuration)
  • call using amqp rpc into nova manager resize_instance
• nova.compute.manager.resize_instance()
  • get network and instance information
  • update instance to RESIZE_MIGRATING state
  • get block device information
  • call driver.migrate_disk_and_power_off()
  • call _terminate_volume_connections
  • call into conductor to network_migrate_instance_start which will eventually call the network API migrate_instance_start
  • update instance to RESIZE_MIGRATED state
  • call using amqp rpc into nova manager finish_resize
• nova.compute.manager._terminate_volume_connections()
  • if there is a volume connection to terminate
    • driver.get_volume_connector()
    • for each volume connection remove the connection by calling the volume API terminate_connection
• nova.compute.manager.finish_resize()
  • call _finish_resize
  • if successful commit the quota reservation
  • else rollback the quota reservation and update instance to ERROR state
• nova.compute.manager._finish_resize()
  • if the flavor is changing then update the instance with the new flavor
  • setup networks on destination host by calling the network API setup_networks_on_host
  • call into conductor to network_migrate_instance_finish which will eventually call the network API migrate_instance_finish
  • update instance to RESIZE_FINISHED state
  • refresh and get block device information
  • driver.finish_migration()
  • update instance to RESIZED state

Cold migration confirm flow:

• nova.api.openstack.compute.servers._action_confirm_resize()
• nova.compute.api.confirm_resize()
  • reserve quota for decrease in resource usage
  • call amqp rpc into nova manager confirm_resize
• nova.compute.manager.confirm_resize()
  • tear down networks on source host by calling the network API setup_networks_on_host
  • driver.confirm_migration()
  • update instance to ACTIVE (or possibly STOPPED) state
  • commit the quota reservation

Cold migration revert flow:

• nova.api.openstack.compute.servers._action_revert_resize()
• nova.compute.api.revert_resize()
  • reserve quota for increase in resource usage
  • update instance task state to RESIZE_REVERTING
  • call amqp rpc into nova manager revert_resize
• nova.compute.manager.revert_resize()
  • tear down networks on destination host by calling the network API setup_networks_on_host
  • call into conductor to network_migrate_instance_start which will eventually call the network API migrate_instance_start
  • get block device information
  • driver.destroy()
  • call _terminate_volume_connections
  • drop resize resources claimed on destination
  • call amqp rpc into nova manager finish_revert_resize
• nova.compute.manager.finish_revert_resize()
  • update instance back to pre-resize values
  • re-setup networks on source host by calling the network API setup_networks_on_host
  • refresh and get block device information
  • driver.finish_revert_migration()
  • update instance to RESIZE_REVERTING state
  • call into conductor to network_migrate_instance_finish which will eventually call the network API migrate_instance_finish
  • update instance to ACTIVE (or possibly STOPPED) state
  • commit the quota usage

Source: http://bodenr.blogspot.com/2014/03/openstack-nova-vm-migration-live-and.html