OpenStack 使用Ceph 配置指导

概述

Ceph 作为分布式文件系统，不但具有高可靠性、高扩展性、高性能。 也是统一存储系统。支持对象存储、块存储、文件存储，本文介绍怎样使用Ceph 块存储作为OpenStack的Glance、Nova、Cinder组件的后端存储

前面已经分享过ceph 集群的部署，以下怎样让openstack在glance、cinder、nova组件后端配置使用ceph来进行存储

參考文档

http://docs.ceph.com/docs/master/rbd/rbd-openstack/

http://docs.openfans.org/ceph/ceph4e2d658765876863/ceph-1/copy_of_ceph-block-device3010ceph57578bbe59073011/openstack301057578bbe59077684openstack3011#u

环境信息

192.168.3.8     ceph node1  （部署mon,mds）

192.168.3.9     ceph node2  （部署osd)

192.168.3.10   ceph node3   (部署osd)

192.168.3.4     openstack    （devstack 部署，包含 controller, network,compute)

说明

ceph 不支持qcow2格式的镜像。所以假设配置nova部分假设要启动虚拟机在ceph ，镜像必须是raw

配置glance

配置认证

1. 将node1节点/etc/ceph文件夹下的两个文件复制到openstack 的 controller节点和compute节点（我这里都在一个节点）

cd /etc/ceph/

scp ceph.conf ceph.client.admin.keyring 192.168.3.4:/etc/ceph/

2. 改动ceph.client.admin.keyring的权限

chmod +r /etc/ceph/ceph.client.admin.keyring

3. 在ceph148上创建glance的存储池

rados mkpool glance

4. 编辑openstack上glance的配置文件/etc/glance/glance-api.conf中例如以下配置项

rbd_store_ceph_conf = /etc/ceph/ceph.conf

stores = rbd    (L 曾经版本号不须要这个)

default_store = rbd

rbd_store_user = admin  (这里的username是上面cephx认证的用户)

rbd_store_pool = glance

rbd_store_chunk_size = 8

6,重新启动glance-api 服务

7,上传一个镜像

stack@map-VirtualBox:~/devstack$ glance image-create --name "cirros-0.3.2-x86_64-10" --disk-format qcow2 --container-format bare <./files/images/cirros-0.3.4-x86_64-uec/cirros-0.3.4-x86_64-blank.img
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | eb9139e4942121f22bbc2afc0400b2a4     |
| container_format | bare                                 |
| created_at       | 2016-02-02T06:45:09Z                 |
| disk_format      | qcow2                                |
| id               | 5d57ba01-a905-42b6-bc9b-39a10b8c7fcb |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | cirros-0.3.2-x86_64-10               |
| owner            | bffb0c753d004509b0ef6ae9bd4777ba     |
| protected        | False                                |
| size             | 25165824                             |
| status           | active                               |
| tags             | []                                   |
| updated_at       | 2016-02-02T06:45:19Z                 |
| virtual_size     | None                                 |
| visibility       | private                              |
+------------------+--------------------------------------+

8，查看ceph 池中的对象。刚上传的镜像已经到ceph池

root@map-VirtualBox:/etc/glance# rbd ls images
5d57ba01-a905-42b6-bc9b-39a10b8c7fcb
root@map-VirtualBox:/etc/glance# 

9,查看 image 池中占用情况

root@map-VirtualBox:~# rados df
pool name       category                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB
cephfs_data     -                          0            0            0            0           0            0            0            0            0
cephfs_metadata -                          0            0            0            0           0            0            0            0            0
data            -                          0            0            0            0           0            0            0            0            0
<span style="color:#ff0000;">images          -                      24577            6            0            0           0          113           87           20        24577</span>
metadata        -                          2           20            0            0           0            6            5           31            8
nova            -                          0            0            0            0           0            0            0            0            0
rbd             -                          0            0            0            0           0            0            0            0            0
test            -                     137633           46            0            0           0          287         1110          642       132406
vms             -                          0            1            0            0           0          308          598         1572        43931
volumes         -                          0            0            0            0           0            0            0            0            0
  total used        10891912           73
  total avail        5085560
  total space       15977472

配置nova

1,创建nova pool

rados mkpool vms

root@map-VirtualBox:/etc/nova# rados mkpool nova
successfully created pool nova
root@map-VirtualBox:/etc/nova# rados lspools
data
metadata
rbd
cephfs_data
cephfs_metadata
test
volumes
images
vms
nova

2,生成uuid

root@map-VirtualBox:/etc/nova# uuidgen
ebdba075-59bc-4408-9a2c-d44b16d56bd3

3,创建screen

root@map-VirtualBox:/etc/nova# cat > secret.xml <<EOF
> <secret ephemeral='no' private='no'>
>   <uuid>ebdba075-59bc-4408-9a2c-d44b16d56bd3</uuid>
>   <usage type='ceph'>
>     <name>client.admin secret</name>
>   </usage>
> </secret>
> EOF

4,运行virsh secret-define --file secret.xml

root@map-VirtualBox:/etc/nova# virsh secret-define --file secret.xml
Secret ebdba075-59bc-4408-9a2c-d44b16d56bd3 created

5,运行例如以下命令，设置

root@map-VirtualBox:/etc/nova# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
        key = AQDPoZRWcIQCDhAAwGapOCdp2uql2HJN2HpD9w==
root@map-VirtualBox:/etc/nova# echo "AQDPoZRWcIQCDhAAwGapOCdp2uql2HJN2HpD9w==">key
root@map-VirtualBox:/etc/nova# virsh secret-set-value --secret ebdba075-59bc-4408-9a2c-d44b16d56bd3 --base64 $(cat key)
Secret value set

查看secret验证配置是否正确

root@map-VirtualBox:~# virsh secret-define
error: command 'secret-define' requires <file> option
root@map-VirtualBox:~# virsh secret-list
 UUID                                  Usage
--------------------------------------------------------------------------------
 ebdba075-59bc-4408-9a2c-d44b16d56bd3  ceph client.admin secret

root@map-VirtualBox:~# virsh secret-dumpxml ebdba075-59bc-4408-9a2c-d44b16d56bd3
<secret ephemeral='no' private='no'>
  <uuid>ebdba075-59bc-4408-9a2c-d44b16d56bd3</uuid>
  <usage type='ceph'>
    <name>client.admin secret</name>
  </usage>
</secret>

root@map-VirtualBox:~# 

6,配置nova.conf

先备份nova.conf

root@map-VirtualBox:/etc/nova# ls
api-paste.ini  key  nova.conf  policy.json  rootwrap.conf  rootwrap.d  secret.xml
root@map-VirtualBox:/etc/nova# cp nova.conf nova.conf.bak

编辑nova.conf 中例如以下内容

images_type=rbd

images_rbd_pool=vms

images_rbd_ceph_conf=/etc/ceph/ceph.conf

rbd_user=admin

rbd_secret_uuid=ebdba075-59bc-4408-9a2c-d44b16d56bd3

cpu_mode=none

7,重新启动nova-compute

8,创建一个虚拟机

watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQv/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" width="807" height="266">

9，查看ceph 池中的对象，刚创建的虚拟机已经到ceph池

root@map-VirtualBox:/etc/cinder# rados df
pool name       category                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB
cephfs_data     -                          0            0            0            0           0            0            0            0            0
cephfs_metadata -                          0            0            0            0           0            0            0            0            0
data            -                          0            0            0            0           0            0            0            0            0
images          -                      49153           11            0            0           0          269        24785           40        49154
metadata        -                          2           20            0            0           0            6            5           31            8
nova            -                          0            0            0            0           0            0            0            0            0
rbd             -                          0            0            0            0           0            0            0            0            0
test            -                     137633           46            0            0           0          287         1110          642       132406
<span style="color:#ff0000;">vms             -                      37455           20            0            0           0          491          919         2400        78635</span>
volumes         -                          0            0            0            0           0            0            0            0            0
  total used        10990976           97
  total avail        4986496
  total space       15977472
root@map-VirtualBox:/etc/cinder# rbd ls vms
06aedb93-087f-4110-9d18-8428aa9ede29_disk
06aedb93-087f-4110-9d18-8428aa9ede29_disk.config
2a1ef5e0-97ee-4c41-8ad9-f253d019c5d2_disk
2a1ef5e0-97ee-4c41-8ad9-f253d019c5d2_disk.config
root@map-VirtualBox:/etc/cinder#

stack@map-VirtualBox:~/devstack$ nova list
+--------------------------------------+------+---------+------------+-------------+------------------+
| ID                                   | Name | Status  | Task State | Power State | Networks         |
+--------------------------------------+------+---------+------------+-------------+------------------+
| 06aedb93-087f-4110-9d18-8428aa9ede29 | sdf  | ACTIVE  | -          | Running     | private=10.0.0.3 |
| 2a1ef5e0-97ee-4c41-8ad9-f253d019c5d2 | we   | SHUTOFF | -          | Shutdown    | private=10.0.0.2 |
+--------------------------------------+------+---------+------------+-------------+------------------+
stack@map-VirtualBox:~/devstack$