ceph-块存储客户端

ceph块存储

ceph块设备，以前称为RADOS块设备，为客户机提供可靠性、分布式和高性能的块存储磁盘。RADOS块设备利用librbd库并以顺序的形式在ceph集群的多个osd上存储数据块。RBD是由ceph的RADOS层支持，因此每个块设备都分布在多个ceph节点上，提供了性能和优异的可靠性。RBD有linux内核的本地支持，这意味着RBD驱动程序从过去几年就与linux内核集成的很好。除了可靠性和性能外，RBD还提供了企业特性，例如完整和增量快照、瘦配置、写时复制克隆、动态调整大小等，RBD还支持内存缓存，这大大提高了性能。

安装ceph块存储客户端

创建ceph块客户端用户名和认证密钥

[ceph-admin@ceph-node1 my-cluster]$ ceph auth get-or-create client.rbd mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=rbd' | tee ./ceph.client.rbd.keyring
[client.rbd]
    key = AQChG2Vcu552KRAAMf4/SdfSVa4sFDZPfsY8bg==

[ceph-admin@ceph-node1 my-cluster]$ ceph auth get client.rbd
exported keyring for client.rbd
[client.rbd]
    key = AQChG2Vcu552KRAAMf4/SdfSVa4sFDZPfsY8bg==
    caps mon = "allow r"
    caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=rbd"

将密钥文件和配置文件拷贝到客户端

[ceph-admin@ceph-node1 my-cluster]$ scp ceph.client.rbd.keyring /etc/ceph/ceph.conf root@192.168.0.123:/etc/ceph

检查客户端是否符合块设备环境要求

[root@localhost ~]# uname -r
3.10.0-862.el7.x86_64
[root@localhost ~]# modprobe rbd
[root@localhost ~]# echo $?
0

安装ceph客户端

[root@localhost ~]# wget -O /etc/yum.repos.d/ceph.repo https://raw.githubusercontent.com/aishangwei/ceph-demo/master/ceph-deploy/ceph.repo
[root@localhost ~]# yum install -y ceph

测试密钥连接集群

[root@localhost ~]# ceph -s --name client.rbd
  cluster:
    id:     cde2c9f7-009e-4bb4-a206-95afa4c43495
    health: HEALTH_OK

  services:
    mon:  daemons, quorum ceph-node1,ceph-node2,ceph-node3
    mgr: ceph-node1(active), standbys: ceph-node2, ceph-node3
    osd:  osds:  up,  in

  data:
    pools:    pools,  pgs
    objects:  objects, 0B
    usage:   .06GiB used, 171GiB / 180GiB avail
    pgs:

客户端创建块设备及映射

创建rbd池

[ceph-admin@ceph-node1 my-cluster]$ ceph osd lspools
[ceph-admin@ceph-node1 my-cluster]$ ceph osd pool create rbd 128
pool 'rbd' created
[ceph-admin@ceph-node1 my-cluster]$ ceph osd lspools
 rbd,

客户端创建块设备

[root@localhost ceph]# rbd create rbd1 --size  --name client.rbd

查看

[root@localhost ceph]# rbd ls -p rbd --name client.rbd
rbd1
[root@localhost ceph]# rbd list --name client.rbd
rbd1
[root@localhost ceph]# rbd --image rbd1 info --name client.rbd
rbd image 'rbd1':
    size 10GiB in  objects
    order  (4MiB objects)
    block_name_prefix: rbd_data.faa76b8b4567
    format:
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    flags:
    create_timestamp: Thu Feb  :: 

映射到客户端

[root@localhost ceph]# rbd map --image rbd1 --name client.rbd
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable rbd1 object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: () No such device or address

映射报错

layering：分层支持
exclusive-lock：排它锁定支持对
object-map：对象映射支持，需要排它所（exclusive-lock）。
deep-flatten：快照平支持（snapshot flatten support）
fast-diff：在client-node1上使用krbd（内核rbd）客户机进行快速diff计算（需要对象映射），我们将无法在centos内核3.10上映射块设备映像，因为该内核不支持对象映射（object-map）、深平（deep-flatten）和快速diff（fast-diff）（在内核4.9中引入了支持）。为了解决这个问题，我们将禁用不支持的特性，有几个选项可以做到这一点：

1）动态禁用

[root@localhost ceph]# rbd feature disable rbd1 exclusive-lock object-map fast-diff deep-flatten --name client.rbd

2）创建rbd镜像时之启用分层特性

[root@localhost ceph]# rbd create rbd2 --size  --image-feature layering --name client.rbd

3）ceph配置文件中禁用

rbd default features = 

再次映射到客户端

[root@localhost ceph]# rbd map --image rbd1 --name client.rbd
/dev/rbd0
[root@localhost ceph]# rbd showmapped --name client.rbd
id pool image snap device
  rbd  rbd1  -    /dev/rbd0

创建文件系统，并挂载

[root@localhost ceph]# fdisk -l /dev/rbd0

Disk /dev/rbd0: 10.7 GB,  bytes,  sectors
Units = sectors of  *  =  bytes
Sector size (logical/physical):  bytes /  bytes
I/O size (minimum/optimal):  bytes /  bytes

[root@localhost ceph]# mkfs.xfs /dev/rbd0
meta-data=/dev/rbd0              isize=    agcount=, agsize= blks
         =                       sectsz=   attr=, projid32bit=
         =                       crc=        finobt=, sparse=
data     =                       bsize=   blocks=, imaxpct=
         =                       sunit=   swidth= blks
naming   =version               bsize=   ascii-ci= ftype=
log      =internal log           bsize=   blocks=, version=
         =                       sectsz=   sunit= blks, lazy-count=
realtime =none                   extsz=   blocks=, rtextents=
[root@localhost ceph]# mkdir /mnt/ceph-disk1
[root@localhost ceph]# mount /dev/rbd0 /mnt/ceph-disk1
[root@localhost ceph]# df -h /mnt/ceph-disk1
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd0        10G   33M   10G   % /mnt/ceph-disk1

写入数据测试

[root@localhost ceph]# ll /mnt/ceph-disk1/
total
[root@localhost ceph]# dd if=/dev/zero of=/mnt/ceph-disk1/file1 count= bs=1M
+ records in
+ records out
 bytes ( MB) copied, 0.127818 s,  MB/s
[root@localhost ceph]# ll /mnt/ceph-disk1/
total
-rw-r--r--  root root  Feb  : file1
[root@localhost ceph]# df -h /mnt/ceph-disk1/
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd0        10G  133M  .9G   % /mnt/ceph-disk1

开机自动挂载

下载脚本

[root@localhost ceph]# wget -O /usr/local/bin/rbd-mount https://raw.githubusercontent.com/aishangwei/ceph-demo/master/client/rbd-mount
[root@localhost ceph]# chmod +x /usr/local/bin/rbd-mount
[root@localhost ceph]# cat /usr/local/bin/rbd-mount
#!/bin/bash

# Pool name where block device image is stored
export poolname=rbd

# Disk image name
export rbdimage=rbd1

# Mounted Directory
export mountpoint=/mnt/ceph-disk1

# Image mount/unmount and pool are passed from the systemd service as arguments
# Are we are mounting or unmounting
if [ "$1" == "m" ]; then
   modprobe rbd
   rbd feature disable $rbdimage object-map fast-diff deep-flatten
   rbd map $rbdimage --id rbd --keyring /etc/ceph/ceph.client.rbd.keyring
   mkdir -p $mountpoint
   mount /dev/rbd/$poolname/$rbdimage $mountpoint
fi
if [ "$1" == "u" ]; then
   umount $mountpoint
   rbd unmap /dev/rbd/$poolname/$rbdimage
fi

做成系统服务

[root@localhost ceph]# wget -O /etc/systemd/system/rbd-mount.service https://raw.githubusercontent.com/aishangwei/ceph-demo/master/client/rbd-mount.service
[root@localhost ceph]# systemctl daemon-reload
[root@localhost ceph]# systemctl enable rbd-mount.service

重启查看自动挂载

[root@localhost ceph]# reboot -f
[root@localhost ceph]# df -h /mnt/ceph-disk1/
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd1        10G  133M  .9G   % /mnt/ceph-disk1
[root@localhost ceph]# ll -h /mnt/ceph-disk1/
total 100M
-rw-r--r--  root root 100M Feb  : file1