cinder create volume的流程(1)

前提：代码的跟踪，使用的是ocata版本

零、执行cinder create 命令，创建数据卷，打开debug开关

[root@osnode241001 ~]# cinder --debug create --display-name=chenwei 1
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://11.27.241.251:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG:keystoneclient.session:RESP: [200] CaseInsensitiveDict({'date': 'Tue, 22 Jan 2019 06:25:22 GMT', 
'vary': 'X-Auth-Token', 'content-length': '423', 'content-type': 'application/json', 'connection': 'close'})
RESP BODY: {"version": 
{"status": "stable", 
"updated": "2014-04-17T00:00:00Z", 
"media-types": [{"base": "application/json", 
"type": "application/vnd.openstack.identity-v2.0+json"}, 
{"base": "application/xml", 
"type": "application/vnd.openstack.identity-v2.0+xml"}], 
"id": "v2.0", 
"links": [{"href": "http://10.27.241.251:5000/v2.0/", "rel": "self"}, 
{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://11.27.241.251:5000/v2.0/tokens
DEBUG:keystoneclient.session:REQ: curl -i -X POST http://11.27.241.251:8776/v1/3333f85b0b784f7cbbb21755e932229b/volumes 
-H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: TOKEN_REDACTED"
-d '{"volume": {
"status": "creating",
"availability_zone": null,
"source_volid": null,
"display_description": null,
"snapshot_id": null,
"user_id": null,
"size": 1,
"display_name": "chenwei",
"imageRef": null,
"attach_status": "detached",
"volume_type": null,
"project_id": null,
"metadata": {}}
}'
DEBUG:keystoneclient.session:RESP: [200] CaseInsensitiveDict({'content-length': '362', 
'x-compute-request-id': 'req-931befc0-e7e7-4e0e-bd57-225e3320eb35', 
'connection': 'close', 'date': 'Tue, 22 Jan 2019 06:25:23 GMT', 
'content-type': 'application/json', 'x-openstack-request-id': 'req-931befc0-e7e7-4e0e-bd57-225e3320eb35'})
RESP BODY: {"volume": {"status": "creating", 
"display_name": "chenwei", "attachments": [], 
"availability_zone": "nova", "bootable": "false",
 "encrypted": false, "created_at": "2019-01-22T06:25:23.454840", 
"display_description": null, "volume_type": "None", 
"snapshot_id": null, "source_volid": null, "metadata": {}, "id": "dd9dab6b-3b09-4b28-b782-77fdeefd099d", "size": 1}}

+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2019-01-22T06:25:23.454840      |
| display_description |                 None                 |
|     display_name    |               chenwei                |
|      encrypted      |                False                 |
|          id         | dd9dab6b-3b09-4b28-b782-77fdeefd099d |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

一、cinder-api流程部分　

cinder-api接收到以http协议发送的POST请求，经过http body分析：创建一个size为1GB的volume
进入文件cinder/api/v1/volumes.py调用class VolumeController(wsgi.Controller):类的create方法

def create(self, req, body)
    前半部分是对cinder request请求体里面的内容进行获取和验证
	#最主要代码
	new_volume = self.volume_api.create(context,
            size,
            volume.get('display_name'),
            volume.get('display_description'),
            **kwargs)

进入cinder/volume/api.py文件的class API(base.Base)类create方法　　

class API(base.Base):
    def create(self, context, size, name, description, snapshot=None,
               image_id=None, volume_type=None, metadata=None,
               availability_zone=None, source_volume=None,
               scheduler_hints=None,
               source_replica=None, consistencygroup=None,
               cgsnapshot=None, multiattach=False, source_cg=None,
               group=None, group_snapshot=None, source_group=None):
    #检查从镜像创建卷还是新建
    check_policy(context, 'create_from_image' if image_id else 'create')
    #当从镜像或者卷创建新卷时，新卷的大小如果未明确指定，那么默认取原来卷的值
    #cinderclient下发的volume的大小是字符串形式，而有些cloud调用是直接调用api，所以需要对size的类型进行检查
    前半部分主要逻辑是对要创建的卷所需的参数进一步验证，例如：卷可用域，卷的类型检查等
    #最主要逻辑代码
            sched_rpcapi = (self.scheduler_rpcapi if (
                            not cgsnapshot and not source_cg and
                            not group_snapshot and not source_group)
                            else None)
            volume_rpcapi = (self.volume_rpcapi if (
                             not cgsnapshot and not source_cg and
                             not group_snapshot and not source_group)
                             else None)
    #进入创建卷的工作流
    #cinder-api启动volume_create_api工作流（flow），flow的状态从pending->running->success，
    #该工作流包含若干个任务（TASK），TASK的转换状态与工作流一样，都是到success结束，每个TASK完成特定的任务
            flow_engine = create_volume.get_flow(self.db,
                                                 self.image_service,
                                                 availability_zones,
                                                 create_what,
                                                 sched_rpcapi,
                                                 volume_rpcapi)    

进入cinder/volume/flows/api/create_volume.py文件中，调用get_flow方法

def get_flow(db_api, image_service_api, availability_zones, create_what,
             scheduler_rpcapi=None, volume_rpcapi=None):
    """Constructs and returns the api entrypoint flow.
     构造并返回api entrypoint 工作流
    This flow will do the following:

    1. Inject keys & values for dependent tasks.
    为依赖任务注入的键和值。
    2. Extracts and validates the input keys & values.
    提取并验证输入的键和值
    3. Reserves the quota (reverts quota on any failures).
    保留配额(在发生故障时恢复配额)
    4. Creates the database entry.
    创建数据库条目
    5. Commits the quota.
    提交配额
    6. Casts to volume manager or scheduler for further processing.
    将数据类型转换到volume manager or scheduler以进行进一步处理
    """
    创建api线性流
    api_flow = linear_flow.Flow(flow_name)

    #依次添加ExtractVolumeRequestTask、QuotaReserveTask、EntryCreateTask、QuotaCommitTask
    api_flow.add(ExtractVolumeRequestTask(
        image_service_api,
        availability_zones,
        rebind={'size': 'raw_size',
                'availability_zone': 'raw_availability_zone',
                'volume_type': 'raw_volume_type'}))
    api_flow.add(QuotaReserveTask(),
                 EntryCreateTask(),
                 QuotaCommitTask())
    if scheduler_rpcapi and volume_rpcapi:
        # This will cast it out to either the scheduler or volume manager via
        # the rpc apis provided.
        api_flow.add(VolumeCastTask(scheduler_rpcapi, volume_rpcapi, db_api))

    # Now load (but do not run) the flow using the provided initial data.
    return taskflow.engines.load(api_flow, store=create_what)

volume_create_api工作流包含的TASK：
1)ExtractVolumeRequestTask获取request信息
2)QuotaReserverTask预留配额
3)EntryCreateTask在数据库中创建volume条目
4)QuotaCommitTask确认配额
1-4都是创建volume的准备工作
5)VolumeCastTask向cinder-scheduler发送消息，实际就是把消息发给rabbitmq，
最后，volume_create_api flow达到success状态，完成了pending->running->success整个流程。
这里的Volume created successfully只代表cinder-api处理完成，不代表后端存储节点上的volume创建成功

VolumeCastTask类的功能
class VolumeCastTask(flow_utils.CinderTask):
    """Performs a volume create cast to the scheduler or to the volume manager.
    向 scheduler or to the volume manager发送一个创建卷的cast
    This will signal a transition of the api workflow to another child and/or
    related workflow on another component.
    将api工作流转换为另一个子工作流和/或另一个组件上的相关工作流
    Reversion strategy: rollback source volume status and error out newly
    created volume.
    """
    def __init__(self, scheduler_rpcapi, volume_rpcapi, db):
        requires = ['image_id', 'scheduler_hints', 'snapshot_id',
                    'source_volid', 'volume_id', 'volume', 'volume_type',
                    'volume_properties', 'source_replicaid',
                    'consistencygroup_id', 'cgsnapshot_id', 'group_id', ]
        super(VolumeCastTask, self).__init__(addons=[ACTION],
                                             requires=requires)
        self.volume_rpcapi = volume_rpcapi #会进入cinder/volume/rpcapi.py文件中create_volume方法
        self.scheduler_rpcapi = scheduler_rpcapi #会进入cinder/scheduler/rpcapi.py 文件中create_volume方法
        self.db = db

二、cinder-scheduler流程部分

进入cinder/scheduler/rpcapi.py文件中class SchedulerAPI类的create_volume方法

class SchedulerAPI(rpc.RPCAPI)类的
    def create_volume(self, ctxt, volume, snapshot_id=None, image_id=None,
                      request_spec=None, filter_properties=None):
        volume.create_worker()
        cctxt = self._get_cctxt()
        msg_args = {'snapshot_id': snapshot_id, 'image_id': image_id,
                    'request_spec': request_spec,
                    'filter_properties': filter_properties, 'volume': volume}
        return cctxt.cast(ctxt, 'create_volume', **msg_args)

自动映射到cinder/scheduler/manager.py中的 create_volume 方法

class SchedulerManager类
    def create_volume(self, context, volume, snapshot_id=None, image_id=None,
                      request_spec=None, filter_properties=None):
        self._wait_for_scheduler()

        try:
        #cinder-scheduler开启一个volume_create_scheduler flow工作流
            flow_engine = create_volume.get_flow(context,
                                                 self.driver,
                                                 request_spec,
                                                 filter_properties,
                                                 volume,
                                                 snapshot_id,
                                                 image_id)
        except Exception:
            msg = _("Failed to create scheduler manager volume flow")
            LOG.exception(msg)
            raise exception.CinderException(msg)

        with flow_utils.DynamicLogListener(flow_engine, logger=LOG):

进入到cinder/scheduler/flows/create_volume.py 文件中调用get_flow函数

def get_flow(context, driver_api, request_spec=None,
             filter_properties=None,
             volume=None, snapshot_id=None, image_id=None):

    """Constructs and returns the scheduler entrypoint flow.
        构造并返回scheduler entrypoint 工作流
    This flow will do the following:
    这个流主要做如下内容：
    1. Inject keys & values for dependent tasks.
    为依赖任务注入键和值
    2. Extract a scheduler specification from the provided inputs.
    从提供的输入中提取scheduler规格参数
    3. Use provided scheduler driver to select host and pass volume creation request further.
    使用提供的调度程序驱动程序选择主机并传递卷创建请求
    创建volume_create_scheduler 线性工作流
    flow_name = ACTION.replace(":", "_") + "_scheduler"
    scheduler_flow = linear_flow.Flow(flow_name)    

    # This will extract and clean the spec from the starting values.
    这将从初始值中提取和清除规范
    scheduler_flow.add(ExtractSchedulerSpecTask(
        rebind={'request_spec': 'raw_request_spec'}))

    # This will activate the desired scheduler driver (and handle any
    # driver related failures appropriately).
    这将激活期望的调度驱动程序
    scheduler_flow.add(ScheduleCreateVolumeTask(driver_api))

    # Now load (but do not run) the flow using the provided initial data.
    return taskflow.engines.load(scheduler_flow, store=create_what)

volume_create_scheduler flow 工作流,包含两个任务
ExtractSchedulerSpecTask
ScheduleCreateVolumeTask 完成filter和weight的工作
filter依次经过：
AvailabilityZoneFilter，可以给存储节点划分az，这样就可以通过该过滤器过滤到用户指定的az中创建卷；
CapacityFilter过滤掉存储空间不满足用户需求的存储节点；
CapabilitiesFilter，不同的后端存储，比如lvm、ceph、sheepdog等等这些产品有不同的特性，
cinder用户允许创建volume时指定需要的Capabilities（特性）,通过volume Type来指定这个特性，volume Type可以根据不同的存储产品特性自定义

最后volume_create_scheduler工作流到达success状态

class ScheduleCreateVolumeTask(flow_utils.CinderTask):
    def __init__(self, driver_api, **kwargs):
        super(ScheduleCreateVolumeTask, self).__init__(addons=[ACTION],
                                                       **kwargs)
        self.driver_api = driver_api
        self.message_api = message_api.API()

进入到cinder/scheduler/driver.py
class Scheduler(object):
所有调度类都需要继承的基类
    """The base class that all Scheduler classes should inherit from."""

    def __init__(self):
        self.host_manager = importutils.import_object(
            CONF.scheduler_host_manager)
        self.volume_rpcapi = volume_rpcapi.VolumeAPI() #会调用cinder/volume/rpcapi.py的class VolumeAPI(rpc.RPCAPI)
        

cinder-scheduler发送消息给cinder-volume,让其创建volume.

进入 cinder/scheduler/filter_scheduler.py,方法为class FilterScheduler(driver.Scheduler):

 def schedule_create_volume(self, context, request_spec, filter_properties):
  #主要实现是
        self.volume_rpcapi.create_volume(context, updated_volume, request_spec,
                                         filter_properties,
                                         allow_reschedule=True) 

三、cinder-volume流程部分

进入cinder/volume/rpcapi.py文件中调用create_volume方法 rpc调用

    def create_volume(self, ctxt, volume, request_spec, filter_properties,
                      allow_reschedule=True):
        cctxt = self._get_cctxt(volume.service_topic_queue)
        cctxt.cast(ctxt, 'create_volume',
                   request_spec=request_spec,
                   filter_properties=filter_properties,
                   allow_reschedule=allow_reschedule,
                   volume=volume)

自动映射到cinder/volume/manager.py 文件中类VolumeManager中的create_volume方法

   @objects.Volume.set_workers
    def create_volume(self, context, volume, request_spec=None,
                      filter_properties=None, allow_reschedule=True):
        """Creates the volume."""
        #最主要的是创建卷工作流程
        cinder-volume开启volume_create_manager flow工作流
            flow_engine = create_volume.get_flow(
                context_elevated,
                self,
                self.db,
                self.driver,
                self.scheduler_rpcapi,
                self.host,
                volume,
                allow_reschedule,
                context,
                request_spec,
                filter_properties,
                image_volume_cache=self.image_volume_cache,
            )    

进入cinder/volume/flows\manager/create_volume.py文件中调用get_flow方法

def get_flow(context, manager, db, driver, scheduler_rpcapi, host, volume,
             allow_reschedule, reschedule_context, request_spec,
             filter_properties, image_volume_cache=None):

    """Constructs and returns the manager entrypoint flow.
    构造并返回manager entrypoint工作流
    This flow will do the following:
    1. Determines if rescheduling is enabled (ahead of time).
    确定是否启用了重新调度(提前)
    2. Inject keys & values for dependent tasks.
    为依赖任务注入键和值
    3. Selects 1 of 2 activated only on *failure* tasks (one to update the db
       status & notify or one to update the db status & notify & *reschedule*).
       在仅“失败”任务上，选择激活的2个选项中的1个，一个是更新数据库状态和通知
       另一个是更新数据库状态，通知和重新调度
    4. Extracts a volume specification from the provided inputs.
    从提供的输入中提取卷规格参数
    5. Notifies that the volume has started to be created.
    通知卷已开始创建
    6. Creates a volume from the extracted volume specification.
    从提取的卷规范中创建卷
    7. Attaches a on-success *only* task that notifies that the volume creation
       has ended and performs further database status updates.
    附加一个on-success“only”任务，该任务通知卷的创建已结束并执行进一步的数据库状态更新    

    #volume_create_manager flow线性工作流
    flow_name = ACTION.replace(":", "_") + "_manager"
    volume_flow = linear_flow.Flow(flow_name)
    volume_flow.add(ExtractVolumeRefTask(db, host, set_error=False))

    retry = filter_properties.get('retry', None)

    # Always add OnFailureRescheduleTask and we handle the change of volume's
    # status when reverting the flow. Meanwhile, no need to revert process of
    # ExtractVolumeRefTask.
    do_reschedule = allow_reschedule and request_spec and retry
    volume_flow.add(OnFailureRescheduleTask(reschedule_context, db, driver,
                                            scheduler_rpcapi, do_reschedule))

    LOG.debug("Volume reschedule parameters: %(allow)s "
              "retry: %(retry)s", {'allow': allow_reschedule, 'retry': retry})

    volume_flow.add(ExtractVolumeSpecTask(db),
                    NotifyVolumeActionTask(db, "create.start"),
                    CreateVolumeFromSpecTask(manager,
                                             db,
                                             driver,
                                             image_volume_cache),
                    CreateVolumeOnFinishTask(db, "create.end"))

    # Now load (but do not run) the flow using the provided initial data.
    return taskflow.engines.load(volume_flow, store=create_what)

该volume_create_manager flow的Task包括：
1)ExtractVolumeRefTask
2)OnFailureRescheduleTask
3)ExtractVolumeSpecTask
4)NotifyVolumeActionTask
这四个Task为volume创建做准备
5)CreateVolumeFromSpecTask
真正的执行volume创建任务，openstack默认的是lvm，可以在cinder.conf里指定后端存储类型，当然就用lvcreate这样的命令创建卷了，
实际使用sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate 命令行，

6)CreateVolumeOnFinishTask
CreatevolumeOnFinishTask，完成扫尾工作
最后volume_create_manager flow达到success状态

参考文章：

https://blog.csdn.net/u010855924/article/details/60881917