k8s使用ceph存储

目录

• ceph配置
• k8s 配置
  • 通过静态pv,pvc使用ceph
  • 测试多pod挂载静态pv数据不一致问题
  • StoragaClass 方式
• ceph 常用命令
• k8s 常用命令
• k8s各类端口及IP说明
  • 端口说明
• 问题
• PV访问模式
• 测试结果
• 参考

---

ceph配置

rbd create --size 100 rbd/nginx-image

[root@localhost my-cluster]# rbd list
nginx-image
[root@localhost my-cluster]# rbd info nginx-image
rbd image 'nginx-image':
        size 100MiB in 25 objects
        order 22 (4MiB objects)
        block_name_prefix: rbd_data.5e4d6b8b4567
        format: 2
        features: layering
        flags:
        create_timestamp: Tue Apr 30 18:10:05 2019
[root@localhost my-cluster]#
# 获取密码
[root@localhost my-cluster]# ceph auth get-key client.admin | base64
QVFDRTQ4ZGNLRFVIRFJBQTVGd2J5QzU0d3B0cGJuOTREcjM1VHc9PQ==

k8s 配置

通过静态pv,pvc使用ceph

每次重建需要先执行 rbd map 将镜像挂载到node节点，rbd卷允许将Rados块设备卷安装到Pod中。与删除Pod时擦除emptyDir不同，rbd卷的内容被保留，而卷仅被卸载。这意味着RBD卷可以预先填充数据，并且可以在pod之间“传递”数据。ceph RBD只能进行单节点读写或多节点读，不能进行多节点读写.但是有的业务可能需要多节点读写的功能，可用cephfs解决了这个问题。

• 安装ceph-common

  yum -y install ceph-common

• ceph配置拷至k8s node 节点

  ceph.conf ,ceph.client.admin.keyring 至 /etc/ceph/ 目录

• 格式化为 xfs

[root@localhost my-cluster]# rbd map nginx-image
/dev/rbd0
[root@localhost my-cluster]# mkfs.xfs /dev/rbd0
[root@localhost my-cluster]# rbd unmap nginx-image

• 创建ceph的secret

cat ceph-secret.yaml
**********************
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
type: "kubernetes.io/rbd"
data:
  key: QVFDTTlXOWFOMk9IR3hBQXZyUjFjdGJDSFpoZUtmckY0N2tZOUE9PQ==
kubectl create -f ceph-secret.yaml
[root@node1 work]# kubectl get secret
NAME                  TYPE                                  DATA      AGE
ceph-secret           Opaque                                1         8d
default-token-7s88r   kubernetes.io/service-account-token   3         11d

• 创建PV

[root@node1 work]# cat nginx-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nginx-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  rbd:
    monitors:
      - 192.168.6.156:6789,192.168.6.157:6789,192.168.6.158:6789
    pool: rbd
    image: nginx-image
    user: admin
    secretRef:
      name: ceph-secret
    fsType: xfs
    readOnly: false
  persistentVolumeReclaimPolicy: Recycle

• 创建PVC

[root@node1 work]# cat nginx-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

kubectl create -f nginx-pvc.yml

• 创建Deployment并挂载

[root@node1 work]# cat nginx-deploy.yml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: demo
        image: mritd/demo
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/data"
            name: data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: nginx-pvc

[root@node1 work]# rbd map nginx-image
/dev/rbd0
[root@node1 work]# kubectl create -f nginx-deploy.yml
deployment "demo" created

• 创建service

[root@node1 work]# kubectl expose deployment/demo
service "demo" exposed
[root@node1 work]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
demo         ClusterIP   10.254.170.53   <none>        80/TCP    6m
kubernetes   ClusterIP   10.254.0.1      <none>        443/TCP   11d

当前k8s1.8版本对于pv 的 RWO 似乎是有bug的，三个pod都挂载了pv，都可写数据，但相互看不到，正常应当是只有一个pod 创建成功。rbd使用前，需要在节点上执行 rbd map 命令，也不太方便。

https://github.com/kubernetes/kubernetes/issues/60903

测试多pod挂载静态pv数据不一致问题

[root@node1 work]# ansible k8s -a 'rbd map nginx-image'
192.168.6.161 | SUCCESS | rc=0 >>
/dev/rbd0

192.168.6.162 | SUCCESS | rc=0 >>
/dev/rbd0

192.168.6.163 | SUCCESS | rc=0 >>
/dev/rbd1
[root@node1 work]# kubectl create -f nginx-deploy.yml
deployment "demo" created
[root@node1 work]# kubectl get pods
NAME                    READY     STATUS    RESTARTS   AGE
ceph-mysql-pod          1/1       Running   0          8d
demo-579d6c87d6-5c59l   1/1       Running   0          10s
demo-579d6c87d6-dck74   1/1       Running   0          10s
demo-579d6c87d6-gg9jf   1/1       Running   0          10s

[root@node1 work]# for i in `kubectl get pods -l app=demo|grep demo|awk '{print $1}'`; do kubectl exec -it $i touch /data/$i.txt ;done;

[root@node1 work]# for i in `kubectl get pods -l app=demo|grep demo|awk '{print $1}'`; do kubectl exec -it $i ls  /data ;done;
3.txt                      demo-579d6c87d6-5c59l.txt
3.txt                      demo-579d6c87d6-dck74.txt
3.txt                      demo-579d6c87d6-gg9jf.txt

每个 pod 文件各不相同。

删除deploy后重建pod
[root@node1 work]# kubectl delete -f nginx-deploy.yml
deployment "demo" deleted
[root@node1 work]# ansible k8s -a 'rbd map nginx-image'
192.168.6.161 | SUCCESS | rc=0 >>
/dev/rbd0

192.168.6.162 | SUCCESS | rc=0 >>
/dev/rbd0

192.168.6.163 | SUCCESS | rc=0 >>
/dev/rbd1

[root@node1 work]# kubectl create -f nginx-deploy.yml
deployment "demo" created
[root@node1 work]# kubectl get pods
NAME                    READY     STATUS    RESTARTS   AGE
ceph-mysql-pod          1/1       Running   0          8d
demo-579d6c87d6-fbdc2   1/1       Running   0          4s
demo-579d6c87d6-hslhw   1/1       Running   0          4s
demo-579d6c87d6-p5dc5   1/1       Running   0          4s
[root@node1 work]# for i in `kubectl get pods -l app=demo|grep demo|awk '{print $1}'`; do kubectl exec -it $i ls  /data ;done;
3.txt                      demo-579d6c87d6-gg9jf.txt
3.txt                      demo-579d6c87d6-gg9jf.txt
3.txt                      demo-579d6c87d6-gg9jf.txt

rbd中只保留了最后一份。

---

StoragaClass 方式

在 1.4 以后，kubernetes 提供了一种更加方便的动态创建 PV 的方式；也就是说使用 StoragaClass 时无需预先创建固定大小的 PV，等待使用者创建 PVC 来使用；而是直接创建 PVC 即可分配使用。也无需到各Node节点上执行rbd map 镜像。

• 创建系统级 Secret

注意: 由于 StorageClass 要求 Ceph 的 Secret type必须为 kubernetes.io/rbd ，所以上一步创建的 ceph-secret 需要先被删除，然后使用如下命令重新创建；此时的 key 并没有经过 base64

[root@node1 work]# kubectl delete secret ceph-secret
secret "ceph-secret" deleted
[root@node1 work]# ceph auth get-key client.admin
AQCE48dcKDUHDRAA5FwbyC54wptpbn94Dr35Tw==
# 这个 secret type 必须为 kubernetes.io/rbd，否则会造成 PVC 无法使用
kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" --from-literal=key='AQCE48dcKDUHDRAA5FwbyC54wptpbn94Dr35Tw=='

• 创建 StorageClass

cat << EOF >> ceph.storageclass.yml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: test-storageclass
provisioner: kubernetes.io/rbd
parameters:
  monitors: 192.168.6.156:6789,192.168.6.157:6789,192.168.6.158:6789
  # Ceph 客户端用户 ID(非 k8s 的)
  adminId: admin
  adminSecretName: ceph-secret
  pool: rbd
  userId: admin
  userSecretName: ceph-secret
EOF

[root@node1 work]# kubectl create -f ceph.storageclass.yml
storageclass "ceph-storageclass" created
[root@node1 work]# kubectl get storageclass
NAME                PROVISIONER
ceph-storageclass   kubernetes.io/rbd

• 创建PVC

[root@node1 work]# vim nginx-pvc2.yaml               

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-pvc2
  annotations:
    volume.beta.kubernetes.io/storage-class: ceph-storageclass
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

[root@node1 work]# kubectl create -f nginx-pvc2.yaml
persistentvolumeclaim "nginx-pvc2" create

• 创建Deployment

[root@node1 work]# vim nginx-deploy2.yml 

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: demo2
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: demo2
    spec:
      containers:
      - name: nginx-demo2
        image: mritd/demo
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/data"
            name: data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: nginx-pvc2

[root@node1 work]# kubectl create -f nginx-deploy2.yml
deployment "demo2" created
[root@node1 work]# kubectl get pods -o wide
NAME                     READY     STATUS              RESTARTS   AGE       IP             NODE
ceph-mysql-pod           1/1       Running             0          8d        172.30.128.6   node3
demo2-66fd75bb8d-nwxmr   1/1       Running             0          15s       172.30.128.3   node3
demo2-66fd75bb8d-wmt7g   0/1       ContainerCreating   0          15s       <none>         node1
demo2-66fd75bb8d-xh47j   0/1       ContainerCreating   0          15s       <none>         node2

可以看出只创建成功一个pod，符合预期，只有一个节点进行读写，其余节点无法创建成功。

https://github.com/kubernetes/kubernetes/issues/67474

将配置修改为 ReadOnlyMany 后仍不能多个节点挂载，可能也是bug.

ceph 常用命令

• rbd 查看锁

rbd lock list nginx-image

• rbd 查看 map情况

rbd showmapped

k8s 常用命令

• 获取pod IP

[root@node1 work]# kubectl get pods -l app=demo  -o yaml|grep podIP
  podIP: 172.30.128.3
  podIP: 172.30.96.3
  podIP: 172.30.184.2

• 生成service创建配置文件

  kubectl get svc -l app=demo -o yaml

• 查看 service 描述

  kubectl describe svc demo

k8s各类端口及IP说明

端口说明

targetPort：容器接收流量的端口；port：抽象的 Service 端口，可以使任何其它 Pod 访问该 Service 的端口

问题

• rbd: failed to lock image nginx-image (maybe locked by other nodes), error exit status 1
  
  Error syncing pod
  
  May 7 10:50:19 node2 kubelet: E0507 10:50:19.891141 27177 kubelet.go:1633] Unable to mount volumes for pod "demo-67bf76f84c-z8kmx_defaul t(3eaf6dab-7072-11e9-b0eb-000c29bda28d)": timeout expired waiting for volumes to attach/mount for pod "default"/"demo-67bf76f84c-z8kmx". li st of unattached/unmounted volumes=[data]; skipping pod May 7 10:50:19 node2 kubelet: E0507 10:50:19.891219 27177 pod_workers.go:182] Error syncing pod 3eaf6dab-7072-11e9-b0eb-000c29bda28d ("d emo-67bf76f84c-z8kmx_default(3eaf6dab-7072-11e9-b0eb-000c29bda28d)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"demo-67bf76f84c-z8kmx". list of unattached/unmounted volumes=[data]
  
  解决
  
  每节点执行:

  rbd map nginx-image
  [root@node3 ~]# rbd showmapped
  id pool image       snap device
  0  rbd  db-image    -    /dev/rbd0
  1  rbd  nginx-image -    /dev/rbd1
  

PV访问模式

• 访问模式包括：
  • ReadWriteOnce —— 该volume只能被单个节点以读写的方式映射
  • ReadOnlyMany —— 该volume可以被多个节点以只读方式映射
  • ReadWriteMany —— 该volume只能被多个节点以读写的方式映射
• 状态
  • Available：空闲的资源，未绑定给PVC
  • Bound：绑定给了某个PVC
  • Released：PVC已经删除了，但是PV还没有被集群回收
  • Failed：PV在自动回收中失败了
• 当前的回收策略有：
  • Retain：手动回收
  • Recycle：需要擦出后才能再使用
  • Delete：相关联的存储资产，如AWS EBS，GCE PD，Azure Disk，or OpenStack Cinder卷都会被删除

当前，只有NFS和HostPath支持回收利用，AWS EBS，GCE PD，Azure Disk，or OpenStack Cinder卷支持删除操作。

测试结果

k8s RBD只能进行单节点读写或多节点读，不能进行多节点读写。测试ceph的rbd发现，多节点读好像有bug，生产使用还是最好能具备一定开发能力。

参考

https://mritd.me/2017/06/03/use-ceph-storage-on-kubernetes/

rbd相关介绍

PV支持模式