openstack vm_lifecycle

nova instance状态:power_state, vm_state, task_state

2015-09-22 Openstack 185

nova instance有3种状态：power_state, vm_state, task_state，分别对应horizon界面上的Power State，Status，Task

Openstack wiki上有介绍：

power_state is the hypervisor state, loaded “bottom-up” from compute worker;
vm_state reflects the stable state based on API calls, matching user expectation, revised “top-down” within API implementation.
task_state reflects the transition state introduced by in-progress API calls.
power_state and vm_state may conflict with each other, which needs to be resolved case-by-case.

Power_state

power_state反映的是hypervisor的状态。目前都是用libvirt作为hypervisor。
例如，nova会调用libvirt api去查询instance的state

[root@node-2 ~]# virsh list --all

 Id    Name                           State

----------------------------------------------------

 229   instance-00000139              running

 230   instance-00000138              running

 231   instance-00000137              running

 232   instance-00000155              running

 -     instance-00000136              shut off

 -     instance-0000013a              shut off

[root@node-2 ~]# virsh domstate instance-00000136

shut off

在libvirt中，共有几种状态：

undefined
defined(也叫stopped)
running
paused
saved

具体解释在这里
下图表示在几种状态的转换关系：

Notes：

有的转换依赖于domain（instance）是transient domain还是persistent domain，如running ---(shutdown)---> defined/undefined。两者的区别在这里。思考：Openstack创建的domain是否都是persistent domain？
virsh create创建transient domain；virsh define;virsh start创建、启动persistent domain

How is it updated？

调hypervisor api取得的状态总是对的；如果nova数据库中的状态与其不一致，那么更新nova数据库。
nova有定时任务来检查、更新power_state
如果对instance做了操作（task），并且该操作可能影响power_state，那么在该task结束之前需要更新power_state

power_state list

目前（2015/9/21 master）有6种状态：

# nova/compute/power_state.py

# NOTE(maoy): These are *not* virDomainState values from libvirt.

# The hex value happens to match virDomainState for backward-compatibility

# reasons.

NOSTATE = 0x00

RUNNING = 0x01

PAUSED = 0x03

SHUTDOWN = 0x04  # the VM is powered off

CRASHED = 0x06

SUSPENDED = 0x07

# TODO(justinsb): Power state really needs to be a proper class,

# so that we're not locked into the libvirt status codes and can put mapping

# logic here rather than spread throughout the code

STATE_MAP = {

    NOSTATE: 'pending',

    RUNNING: 'running',

    PAUSED: 'paused',

    SHUTDOWN: 'shutdown',

    CRASHED: 'crashed',

    SUSPENDED: 'suspended',

}

Vm_state

vm_state反映基于API调用的一种稳定状态，而task_state反映的是一种瞬时状态。

How is it updated？

只有当一个task（一个compute api call）结束时，才能更新vm_state。
如果成功则同时设置task_state=None，如果失败，vm_state可能不变（task支持rollback），也可能设置为ERROR。

vm_state list

# nova/compute/vm_states.py

ACTIVE = 'active'        # VM is running

BUILDING = 'building'    # VM only exists in DB

PAUSED = 'paused'

SUSPENDED = 'suspended'  # VM is suspended to disk.

STOPPED = 'stopped'      # VM is powered off, the disk image is still there.

RESCUED = 'rescued'      # A rescue image is running with the original VM image attached.

RESIZED = 'resized'      # a VM with the new size is active. The user is expected to manually confirm or revert.

SOFT_DELETED = 'soft-delete'  # VM is marked as deleted but the disk images are still available to restore.

DELETED = 'deleted'      # VM is permanently deleted.

ERROR = 'error'

SHELVED = 'shelved'      # VM is powered off, resources still on hypervisor

SHELVED_OFFLOADED = 'shelved_offloaded'  # VM and associated resources are not on hypervisor

ALLOW_SOFT_REBOOT = [ACTIVE]  # states we can soft reboot from

ALLOW_HARD_REBOOT = ALLOW_SOFT_REBOOT + [STOPPED, PAUSED, SUSPENDED, ERROR]  # states we allow hard reboot from

Task_state

task_state反映了基于API调用的瞬时状态。
它表示instance当前处于某个API调用中的某个阶段，比vm_state更细化，让用户了解某个action/task执行到哪个步骤了，比如创建instance时会有scheduling -> block_device_mapping -> networking -> spawning。但调用完之后设置为None。

task_state list

# nova/compute/task_states.py

"""Possible task states for instances.

urrent moment. These tasks can be generic, such as 'spawning', or specific,

such as 'block_device_mapping'. These task states allow for a better view into

what an instance is doing and should be displayed to users/administrators as

necessary.

"""

# possible task states during create()

SCHEDULING = 'scheduling'

BLOCK_DEVICE_MAPPING = 'block_device_mapping'

NETWORKING = 'networking'

SPAWNING = 'spawning'

# possible task states during snapshot()

IMAGE_SNAPSHOT = 'image_snapshot'

IMAGE_SNAPSHOT_PENDING = 'image_snapshot_pending'

IMAGE_PENDING_UPLOAD = 'image_pending_upload'

IMAGE_UPLOADING = 'image_uploading'

# possible task states during backup()

IMAGE_BACKUP = 'image_backup'

# possible task states during set_admin_password()

UPDATING_PASSWORD = 'updating_password'

# possible task states during resize()

RESIZE_PREP = 'resize_prep'

RESIZE_MIGRATING = 'resize_migrating'

RESIZE_MIGRATED = 'resize_migrated'

RESIZE_FINISH = 'resize_finish'

# possible task states during revert_resize()

RESIZE_REVERTING = 'resize_reverting'

# possible task states during confirm_resize()

RESIZE_CONFIRMING = 'resize_confirming'

# possible task states during reboot()

REBOOTING = 'rebooting'

REBOOT_PENDING = 'reboot_pending'

REBOOT_STARTED = 'reboot_started'

REBOOTING_HARD = 'rebooting_hard'

REBOOT_PENDING_HARD = 'reboot_pending_hard'

REBOOT_STARTED_HARD = 'reboot_started_hard'

# possible task states during pause()

PAUSING = 'pausing'

# possible task states during unpause()

UNPAUSING = 'unpausing'

# possible task states during suspend()

SUSPENDING = 'suspending'

# possible task states during resume()

RESUMING = 'resuming'

# possible task states during power_off()

POWERING_OFF = 'powering-off'

# possible task states during power_on()

POWERING_ON = 'powering-on'

# possible task states during rescue()

RESCUING = 'rescuing'

# possible task states during unrescue()

UNRESCUING = 'unrescuing'

# possible task states during rebuild()

REBUILDING = 'rebuilding'

REBUILD_BLOCK_DEVICE_MAPPING = "rebuild_block_device_mapping"

REBUILD_SPAWNING = 'rebuild_spawning'

# possible task states during live_migrate()

MIGRATING = "migrating"

# possible task states during delete()

DELETING = 'deleting'

# possible task states during soft_delete()

SOFT_DELETING = 'soft-deleting'

# possible task states during restore()

RESTORING = 'restoring'

# possible task states during shelve()

SHELVING = 'shelving'

SHELVING_IMAGE_PENDING_UPLOAD = 'shelving_image_pending_upload'

SHELVING_IMAGE_UPLOADING = 'shelving_image_uploading'

# possible task states during shelve_offload()

SHELVING_OFFLOADING = 'shelving_offloading'

# possible task states during unshelve()

UNSHELVING = 'unshelving'

task_state state machine diagram (in-progress):
https://docs.google.com/spreadsheets/d/1uvrFI_L86_tBcZGlE2ck3RnMgtsgjIqYVtWTIT9eD8I/edit#gid=3

http://docs.openstack.org/developer/nova/devref/vmstates.html
http://docs.openstack.org/developer/nova/vmstates.html

Code walk-through

nova-compute定义了如下文件：

nova/compute/power_state.py
nova/compute/vm_states.py
nova/compute/task_states.py

nova-api从自身角度定义了vm_state和task_state的对应关系

nova/api/openstack/common.py

from nova.compute import task_states

from nova.compute import vm_states

_STATE_MAP = {

    vm_states.ACTIVE: {

        'default': 'ACTIVE',

        task_states.REBOOTING: 'REBOOT',      # 前面是task_state, 后面是显示在cli/horizon上的'Status'

        task_states.REBOOT_PENDING: 'REBOOT',

        task_states.REBOOT_STARTED: 'REBOOT',

        task_states.REBOOTING_HARD: 'HARD_REBOOT',

        task_states.REBOOT_PENDING_HARD: 'HARD_REBOOT',

        task_states.REBOOT_STARTED_HARD: 'HARD_REBOOT',

        task_states.UPDATING_PASSWORD: 'PASSWORD',

        task_states.REBUILDING: 'REBUILD',

        task_states.REBUILD_BLOCK_DEVICE_MAPPING: 'REBUILD',

        task_states.REBUILD_SPAWNING: 'REBUILD',

        task_states.MIGRATING: 'MIGRATING',

        task_states.RESIZE_PREP: 'RESIZE',

        task_states.RESIZE_MIGRATING: 'RESIZE',

        task_states.RESIZE_MIGRATED: 'RESIZE',

        task_states.RESIZE_FINISH: 'RESIZE',

    },

    vm_states.BUILDING: {

        'default': 'BUILD',

    },

    vm_states.STOPPED: {

        'default': 'SHUTOFF',

        task_states.RESIZE_PREP: 'RESIZE',

        task_states.RESIZE_MIGRATING: 'RESIZE',

        task_states.RESIZE_MIGRATED: 'RESIZE',

        task_states.RESIZE_FINISH: 'RESIZE',

        task_states.REBUILDING: 'REBUILD',

        task_states.REBUILD_BLOCK_DEVICE_MAPPING: 'REBUILD',

        task_states.REBUILD_SPAWNING: 'REBUILD',

    },

    vm_states.RESIZED: {

        'default': 'VERIFY_RESIZE',

        # Note(maoy): the OS API spec 1.1 doesn't have CONFIRMING_RESIZE

        # state so we comment that out for future reference only.

        #task_states.RESIZE_CONFIRMING: 'CONFIRMING_RESIZE',

        task_states.RESIZE_REVERTING: 'REVERT_RESIZE',

    },

    vm_states.PAUSED: {

        'default': 'PAUSED',

        task_states.MIGRATING: 'MIGRATING',

    },

    vm_states.SUSPENDED: {

        'default': 'SUSPENDED',

    },

    vm_states.RESCUED: {

        'default': 'RESCUE',

    },

    vm_states.ERROR: {

        'default': 'ERROR',

        task_states.REBUILDING: 'REBUILD',

        task_states.REBUILD_BLOCK_DEVICE_MAPPING: 'REBUILD',

        task_states.REBUILD_SPAWNING: 'REBUILD',

    },

    vm_states.DELETED: {

        'default': 'DELETED',

    },

    vm_states.SOFT_DELETED: {

        'default': 'SOFT_DELETED',

    },

    vm_states.SHELVED: {

        'default': 'SHELVED',

    },

    vm_states.SHELVED_OFFLOADED: {

        'default': 'SHELVED_OFFLOADED',

    },

}

def status_from_state(vm_state, task_state='default'):

    """Given vm_state and task_state, return a status string."""

    task_map = _STATE_MAP.get(vm_state, dict(default='UNKNOWN'))

    status = task_map.get(task_state, task_map['default'])

    if status == "UNKNOWN":

        LOG.error(_LE("status is UNKNOWN from vm_state=%(vm_state)s "

                      "task_state=%(task_state)s. Bad upgrade or db "

                      "corrupted?"),

                  {'vm_state': vm_state, 'task_state': task_state})

    return status

从status_from_state方法看，_STATE_MAP表示vm_state处于某个状态时，所有可能的task_state有哪些，都穷举出来了。这里注意，某些task_state的default值，在nova/compute/task_states.py中没有定义，如：

    vm_states.ACTIVE: {

        'default': 'ACTIVE',

        ...}

    vm_states.STOPPED: {

        'default': 'SHUTOFF',

        ...}

    vm_states.RESIZED: {

        'default': 'VERIFY_RESIZE',

        ...}

    vm_states.ERROR: {

        'default': 'ERROR',

        ...}

给定一对vm_state/task_state，status_from_state能返回一个status字符串，这个status就是cli/horizon上的'Status'：

[root@node-129 ~]# nova list | cut -d"|" -f1-6

+--------------------------------------+-------------------------------------------------------+---------+------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| ID                                   | Name                                                  | Status  | Task State | Power State

+--------------------------------------+-------------------------------------------------------+---------+------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| f6d4e67b-885c-4a10-891f-d91675e52bae | 123                                                   | SHUTOFF | -          | Shutdown

| 05be07c9-6b2a-4db9-b03b-8322c035816c | 652                                                   | SHUTOFF | -          | Shutdown

| 19d1308c-f19b-4860-979f-220fd376ef58 | ITMGMT                                                | SHUTOFF | -          | Shutdown

| c496c710-29f8-46c3-90f8-6c2606d4b252 | Winboo-vm                                             | ACTIVE  | -          | Running

| a27f4030-ece5-4aeb-a84e-70bf0389f81f | abc                                                   | ACTIVE  | -          | Running

比较一下，nova-api列出的"Status"其实比nova-compute的vm_state（14个）多，共20个：

ACTIVE
REBOOT
HARD_REBOOT
PASSWORD
REBUILD
MIGRATING
RESIZE
BUILD
SHUTOFF
VERIFY_RESIZE
REVERT_RESIZE
PAUSED
SUSPENDED
RESCUE**
ERROR
DELETED
SOFT_DELETED**
SHELVED
SHELVED_OFFLOADED**
UNKNOWN

状态转移

参考:

不同的操作，只有当instance处于特定的vm_state和task_state时才能进行。这个控制不是由nova-api来完成的，而是由nova-compute来决定的：
nova/compute/api.py定义了nova-compute暴露给外部的接口。大多数方法名称就是操作名，如create,restore, shelve, soft_delete。

基本上所有方法都用check_instance_state修饰了。这个修饰器限定了该操作在某些vm_state/task_state下才可以进行。比如：

@wrap_check_policy

@check_instance_lock

@check_instance_state(vm_state=[vm_states.SOFT_DELETED])

def restore(self, context, instance):

    """Restore a previously deleted (but not reclaimed) instance."""

@wrap_check_policy

@check_instance_cell

@check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.STOPPED,

                                vm_states.PAUSED, vm_states.SUSPENDED])

def snapshot(self, context, instance, name, extra_properties=None):

    """Snapshot the given instance."""

@wrap_check_policy

@check_instance_lock

@check_instance_state(vm_state=set(

                vm_states.ALLOW_SOFT_REBOOT + vm_states.ALLOW_HARD_REBOOT),

                      task_state=[None, task_states.REBOOTING,

                                  task_states.REBOOT_PENDING,

                                  task_states.REBOOT_STARTED,

                                  task_states.REBOOTING_HARD,

                                  task_states.RESUMING,

                                  task_states.UNPAUSING,

                                  task_states.PAUSING,

                                  task_states.SUSPENDING])

def reboot(self, context, instance, reboot_type):

    """Reboot the given instance."""

所以只要查看check_instance_state的参数就能确定操作可行与否。