ceph 安装ceph问题汇总

1、在不同节点安装ceph时，出现以下异常：
参考这里

ceph deploy RuntimeError: NoSectionError: No section: 'ceph'

解决方法：

在报错的机器上尝试：
yum remove ceph-release
rm  /etc/yum.repos.d/ceph.repo.rpmsave

执行以后再执行

ceph-deploy install admin  node1 node2 node3

2、ceph用户没有权限
**执行激活osd盘时出现 **

ceph-deploy osd activate node2:/var/local/osd0 node3:/var/local/osd1

[node3][WARNIN] ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '1', '--monmap', '/var/local/osd1/activate.monmap', '--osd-data', '/var/local/osd1', '--osd-journal', '/var/local/osd1/journal', '--osd-uuid', 'ee9c4a6c-e2b5-46cd-9e91-fd5d6f36fd57', '--keyring', '/var/local/osd1/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2016-08-31 16:38:58.909449 7f48adbb7800 -1 filestore(/var/local/osd1) mkfs: write_version_stamp() failed: (13) Permission denied
[node3][WARNIN] 2016-08-31 16:38:58.909475 7f48adbb7800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13
[node3][WARNIN] 2016-08-31 16:38:58.909529 7f48adbb7800 -1  ** ERROR: error creating empty object store in /var/local/osd1: (13) Permission denied
[node3][WARNIN]
[node3][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /var/local/osd1

解决方法

chown ceph:ceph  /var/local/osd0
chown ceph:ceph  /var/local/osd1

---

 

创建成功ceph健康状况良好

3、RuntimeError: bootstrap-rgw keyring not found; run 'gatherkeys'

root@admin ceph]# ceph-deploy rgw create node2
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.34): /usr/bin/ceph-deploy rgw create node2
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('node2', 'rgw.node2')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x24aed88>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0x241da28>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts node2:rgw.node2
[ceph_deploy][ERROR ] RuntimeError: bootstrap-rgw keyring not found; run 'gatherkeys'

执行如下命令解决

ceph-deploy gatherkeys node2

4 解决找不到/bootstrap-osd/ceph.keyring

ceph-deploy mon create-initial 命令执行后， 如果提示：

[ceph_deploy.mon][INFO  ] Running gatherkeys...
[ceph_deploy.gatherkeys][DEBUG ] Checking manila59 for /etc/ceph/ceph.client.admin.keyring
[manila59][DEBUG ] connected to host: manila59
[manila59][DEBUG ] detect platform information from remote host
[manila59][DEBUG ] detect machine type
[manila59][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on ['manila59']
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking manila59 for /var/lib/ceph/bootstrap-osd/ceph.keyring
[manila59][DEBUG ] connected to host: manila59
[manila59][DEBUG ] detect platform information from remote host
[manila59][DEBUG ] detect machine type
[manila59][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['manila59']
[ceph_deploy.gatherkeys][DEBUG ] Checking manila59 for /var/lib/ceph/bootstrap-mds/ceph.keyring
[manila59][DEBUG ] connected to host: manila59
[manila59][DEBUG ] detect platform information from remote host
[manila59][DEBUG ] detect machine type
[manila59][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['manila59']

可以执行： [root@manila59 deploy]# ceph-create-keys --id manila59

分析发现此处可能连接了外网，要确保DNS正常。

ceph-deploy osd prepare manila59:/dev/sdb命令后，如果提示：

[ceph_deploy][ERROR ] RuntimeError: bootstrap-osd keyring not found; run 'gatherkeys'

执行ceph-deploy gatherkeys manila59：

那么就会先找：/etc/ceph/ceph.client.admin.keyring，然后再找/var/lib/ceph/bootstrap-osd/ceph.keyring和/var/lib/ceph/bootstrap-mds/ceph.keyring ，最后发现ceph.bootstrap-mds.keyring key from manila59。

注意：disk zap 命令是不需要权限的，而osd prepare 需要bootstrap-mds.keyring。

在上面的 map 映射操作时，可能出现如下错误

RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable"

查看系统日志可以看到如下输出

~ dmesg | tail[-1127592253.530346] rbd: image data: image uses unsupported features: 0x38[-1127590337.563180] libceph: mon0 192.168.1.11:6789 session established[-1127590337.563741] libceph: client4200 fsid dd9fdfee-438a-47aa-be21-114372bc1f44

问题原因: 在 Ceph 高版本进行 map image 时，默认 Ceph 在创建 image(上文 data)时会增加许多 features，这些 features 需要内核支持，在 Centos7 的内核上支持有限，所以需要手动关掉一些 features

首先使用 rbd info data 命令列出创建的 image 的 features

rbd image 'data':
        size 10240 MB in 2560 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.37c6238e1f29
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        flags:

在 features 中我们可以看到默认开启了很多:

• layering: 支持分层

• striping: 支持条带化 v2

• exclusive-lock: 支持独占锁

• object-map: 支持对象映射(依赖 exclusive-lock)

• fast-diff: 快速计算差异(依赖 object-map)

• deep-flatten: 支持快照扁平化操作

• journaling: 支持记录 IO 操作(依赖独占锁）

而实际上 Centos 7 的 3.10 内核只支持 layering… 所以我们要手动关闭一些 features，然后重新 map；如果想要一劳永逸，可以在 ceph.conf 中加入 rbd_default_features = 1 来设置默认 features(数值仅是 layering 对应的 bit 码所对应的整数值)。

关闭不支持特性
rbd feature disable data exclusive-lock, object-map, fast-diff, deep-flatten
重新映射
rbd map data --name client.admin
成功后返回设备位置
/dev/rbd0