Ceph 块存储

任何普通的linux主机都可以充当ceph客户机，客户机通过网络与ceph存储集群交互以存储或检索用户数据。Ceph RBD支持已经添加到linux主线内核中，从2.6.34以及以后版本开始。

===================在管理端192.168.20.181操作================================

su - ceph-admin

cd my-cluster

创建ceph块客户端用户名和认证密钥

ceph auth get-or-create client.rbd mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=rbd' |tee ./ceph.client.rbd.keyring

===================在客户端192.168.20.184操作================================

修改hosts文件：

cat /etc/hosts

192.168.20.181 c720181

192.168.20.182 c720182

192.168.20.183 c720183

192.168.20.184 c720184

创建存放密钥和配置文件目录：

mkdir -p /etc/ceph

===================在管理端192.168.20.181操作================================

把密钥和配置文件拷贝到客户端

scp ceph.client.rbd.keyring /etc/ceph/ceph.conf root@192.168.20.184:/etc/ceph/

===================在客户端192.168.20.184操作================================

检查客户端是否符合块设备环境要求

[root@c720184 ~]# uname -r

3.10.0-957.el7.x86_64                 #高于2.6.34就可以

[root@c720184 ~]# modprobe rbd

[root@c720184 ~]# echo $?

0                                            #说明执行成功

安装ceph客户端

yum -y install ceph   (如果没有配置ceph.repo 和 epel.repo 要先配置，否则安装不成功)

连接ceph集群：

ceph -s --name client.rbd

（1）集群创建存储池

默认创建块设备，会直接创建在rbd池中，但使用ceph-deploy安装（L版ceph）后，该rbd池并没有创建

#创建池

===================在管理端192.168.20.181操作================================

[ceph-admin@c720181 my-cluster]$ ceph osd lspools   #查看集群池存储池

[ceph-admin@c720181 my-cluster]$ ceph osd pool create rbd 128   #128为place group 数量

pool 'rbd' created

[ceph-admin@c720181 my-cluster]$ ceph osd lspools

1 rbd,

（2）客户端创建块设备

[root@c720184 ~]# rbd create rbd1 --size 10240 --name client.rbd

[root@c720184 ~]# rbd ls -p rbd --name client.rbd

rbd1

[root@c720184 ~]# rbd list --name client.rbd

rbd1

[root@c720184 ~]# rbd --image rbd1 info --name client.rbd

rbd image 'rbd1':

size 10GiB in 2560 objects

order 22 (4MiB objects)

block_name_prefix: rbd_data.106b6b8b4567

format: 2

features: layering, exclusive-lock, object-map, fast-diff, deep-flatten

flags:

create_timestamp: Sun Aug 18 18:56:02 2019

（3）客户端映射块设备

[root@c720184 ~]# rbd map --image rbd1 --name client.rbd

直接映射会报如下错误，是因为创建的块设备属性对于该内核版本不支持。

features: layering, exclusive-lock, object-map, fast-diff, deep-flatten

解决方式有如下三种：

a.动态禁用 （建议）

rbd feature disable rbd1 exclusive-lock object-map deep-flatten fast-diff --name client.rbd

b.创建RBD块设备时，只启用分层特性

rbd create rbd1 --size 10240 --image-feature layering --name client.rbd

c.ceph配置文件中禁用

rbd_default_features = 1

比如这里使用动态禁用：（截图里面少了--name client.rbd，不然会报错）

rbd feature disable rbd1 exclusive-lock object-map deep-flatten fast-diff --name client.rbd

重新映射块设备：

[root@c720184 ~]# rbd map --image rbd1 --name client.rbd

（4）创建文件系统并挂载

[root@c720184 ~]# fdisk -l /dev/rbd0

Disk /dev/rbd0: 10.7 GB, 10737418240 bytes, 20971520 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes

[root@c720184 ~]# mkfs.xfs /dev/rbd0

meta-data=/dev/rbd0              isize=512    agcount=16, agsize=163840 blks

=                       sectsz=512   attr=2, projid32bit=1

=                       crc=1        finobt=0, sparse=0

data     =                       bsize=4096   blocks=2621440, imaxpct=25

=                       sunit=1024   swidth=1024 blks

naming   =version 2              bsize=4096   ascii-ci=0 ftype=1

log      =internal log           bsize=4096   blocks=2560, version=2

=                       sectsz=512   sunit=8 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@c720184 ~]# mkdir /mnt/ceph-disk1

[root@c720184 ~]# mount /dev/rbd0 /mnt/ceph-disk1

[root@c720184 ~]# df -h

Filesystem               Size  Used Avail Use% Mounted on

/dev/mapper/centos-root   17G  1.5G   16G   9% /

devtmpfs                 908M     0  908M   0% /dev

tmpfs                    920M     0  920M   0% /dev/shm

tmpfs                    920M  8.5M  911M   1% /run

tmpfs                    920M     0  920M   0% /sys/fs/cgroup

/dev/sda1               1014M  145M  870M  15% /boot

tmpfs                    184M     0  184M   0% /run/user/0

/dev/rbd0                 10G   33M   10G   1% /mnt/ceph-disk1

（5）写入数据测试

[root@c720184 ~]# dd if=/dev/zero of=/mnt/ceph-disk1/file1 count=100 bs=1M

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 1.04015 s, 101 MB/s

[root@c720184 ~]# ll -h /mnt/ceph-disk1/

total 100M

-rw-r--r--. 1 root root 100M Aug 18 19:11 file1

（6）做成服务，设置开机自动挂载

wget -O /usr/local/bin/rbd-mount https://raw.githubusercontent.com/aishangwei/ceph-demo/master/client/rbd-mount

#注意需要修改里面的配置参数

#!/bin/bash

# Pool name where block device image is stored
export poolname=rbd

# Disk image name
export rbdimage=rbd1

# Mounted Directory
export mountpoint=/mnt/ceph-disk1

# Image mount/unmount and pool are passed from the systemd service as arguments
# Are we are mounting or unmounting
if [ "$1" == "m" ]; then
   modprobe rbd
   rbd feature disable $rbdimage object-map fast-diff deep-flatten
   rbd map $rbdimage --id rbd --keyring /etc/ceph/ceph.client.rbd.keyring
   mkdir -p $mountpoint
   mount /dev/rbd/$poolname/$rbdimage $mountpoint
fi
if [ "$1" == "u" ]; then
   umount $mountpoint
   rbd unmap /dev/rbd/$poolname/$rbdimage
fi

#添加可执行权限

chmod +x /usr/local/bin/rbd-mount

wget -O /etc/systemd/system/rbd-mount.service https://raw.githubusercontent.com/aishangwei/ceph-demo/master/client/rbd-mount.service

[root@c720184 ~]# systemctl daemon-reload

[root@c720184 ~]# systemctl enable rbd-mount.service

测试重启是否自动挂载：

reboot -f

df -h

这里为了节省时间，没有重启，而是先取消挂载，启动挂载服务看是否挂载上。

[root@c720184 ~]# umount /mnt/ceph-disk1/

[root@c720184 ~]# systemctl start rbd-mount.service

[root@client ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/cl_-root   50G  1.8G   49G   4% /
devtmpfs              910M     0  910M   0% /dev
tmpfs                 920M     0  920M   0% /dev/shm
tmpfs                 920M   17M  904M   2% /run
tmpfs                 920M     0  920M   0% /sys/fs/cgroup
/dev/sda1            1014M  184M  831M  19% /boot
/dev/mapper/cl_-home  196G   33M  195G   1% /home
tmpfs                 184M     0  184M   0% /run/user/0
ceph-fuse              54G  1.0G   53G   2% /mnt/cephfs
c720182:/              54G  1.0G   53G   2% /mnt/cephnfs
/dev/rbd1              20G   33M   20G   1% /mnt/ceph-disk1