StorageClass 简单学习

StorageClass 简单学习

---

学习资料来源

https://www.jianshu.com/p/5e565a8049fc
https://zhuanlan.zhihu.com/p/289501984
https://blog.csdn.net/ag1942/article/details/115371793
https://stackoverflow.com/questions/65376314/kubernetes-nfs-provider-selflink-was-empty

等等..

第一部分: StorageClass 资源获取

• 搭建nfs, 以及提供服务, 我这边采用的是 * 指定ip地址, 避免k8s内部无法连接.
• k8s集群内每个节点都需要打开 nfs systemctl enable nfs && systemctl restart nfs

下载资料

git clone https://github.com/kubernetes-incubator/external-storage.git

• 公司网络太差, 也可以直接使用浏览器下载然后解压缩的方式来实现.

---

第二部分: 部署使用

2.1 部署deployment

注意下载好的文件内有多种外部存储, 为了简单起见使用nfs-client的方式进行

• 进入到nfs-client 文件夹中进行相关工作
• 注意我自己手动下载了一些文件.最开始并没有这么多文件夹的.

/k8ssc/external-storage-master/nfs-client/deploy
[root@k8s-master01 deploy]# ll
总用量 36
-rw-r--r-- 1 root root  226 10月 21 2020 class.yaml
-rw-r--r-- 1 root root 1030 10月 21 2020 deployment-arm.yaml
-rw-r--r-- 1 root root 1008 5月   4 09:58 deployment.yaml
-rw-r--r-- 1 root root  278 5月   4 10:09 nfs-clusterrolebinding.yaml
-rw-r--r-- 1 root root  836 5月   4 10:08 nfs-clusterrole.yaml
drwxr-xr-x 2 root root  214 10月 21 2020 objects
-rw-r--r-- 1 root root 1656 5月   4 09:44 rbac2.yaml
-rw-r--r-- 1 root root 1819 10月 21 2020 rbac.yaml
-rw-r--r-- 1 root root  241 5月   4 09:24 test-claim.yaml
-rw-r--r-- 1 root root  424 10月 21 2020 test-pod.yaml

• 需要注意的是 deployment 需要修改之后再进行部署
• 修改内容为将服务修改成自己本地的nfs-server以及目录

[root@k8s-master01 deploy]# exportfs
/nfs            <world>
# 注意 这个目录是我本地的环境信息.
[root@k8s-master01 deploy]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 10.110.83.201 # 需要修改为自己的nfs-server
            - name: NFS_PATH
              value: /nfs #需要修改成自己的服务目录
      volumes:
        - name: nfs-client-root
          nfs:
            server: 10.110.83.201 # 同样处理.
            path: /nfs #同样处理.

• 修改完deployment.yaml 之后就可以执行部署了.
  
  kubectl apply -f deployment.yaml
• 部署完成之后可以进行查看

[root@k8s-master01 deploy]# kubectl get pods
NAME                                      READY   STATUS             RESTARTS   AGE
mysql-7b9cf5df76-78n5d                    1/1     Running            0          12h
nfs-client-provisioner-7f547686c4-59294   1/1     Running            0          80m
redis-master-0                            1/1     Running            0          69m
redis-slave-0                             0/1     CrashLoopBackOff   18         69m
[root@k8s-master01 deploy]# kubectl get deployment
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
mysql                    1/1     1            1           12h
nfs-client-provisioner   1/1     1            1           80m

• 注意: 为了简单可以先在一个节点将这个images 拉取到 然后分发到各个node 节点中.
• 注意: 有问题需要kubectl describe pods podid 或者是 kubectl logs -f podname 查看相关错误信息.

2.2 部署 StorageClass

• 不需要修改相关文件, 直接部署即可.

cat class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"

• 直接部署即可 kubectl apply -f class.yaml
• 部署完之后可以进行查看:

[root@k8s-master01 deploy]# kubectl get sc
NAME                        PROVISIONER                            RECLAIMPOLICY   VOLUMEBINDINGMODE
managed-nfs-storage         fuseim.pri/ifs                         Delete          Immediate              false                  96m

• 著有需要记住这个name 后面会有用.

2.3 授权

• 注意: 这里面我遇到了一个很大的坑.
  
  按照原始文档创建之后一直报错
  
  leaderelection.go:234] error retrieving resource lock default/fuseim.pri-ifs: Unauthorized
  
  后来经过多发查找发现问题原因 如果创建到default 的表空间, 可能对backends的获取有问题,除了需要执行的kubectl apply -f rbac.yaml 之后还需要执行另外的权限设置.
  
  注意 default 的命令空间 设置.

nfs-clusterrole.yaml
新增内容: 

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    resourceNames: ["nfs-provisioner"]
    verbs: ["use"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

2.4 简要测试验证.

• 创建pvc
  
  kubectl apply -f test-claim.yaml

• 注意文件里面的 class 名字.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

• 可以查看相关信息:

[root@k8s-master01 deploy]# kubectl get pvc
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
mysql                       Bound    pv-volume-mysql                            8Gi        RWO                                  12h
pvc-test                    Bound    pv-test                                    1Gi        RWO                                  12h
redis-data-redis-master-0   Bound    pvc-34e2b8de-13af-4f60-9b45-3259c45e2338   8Gi        RWO            managed-nfs-storage   83m
redis-data-redis-slave-0    Bound    pvc-b651443d-ecff-46a4-9507-977180cde3be   8Gi        RWO            managed-nfs-storage   83m
test-claim                  Bound    pvc-cd79f7a1-2eab-443a-b183-e7730e38840a   1Mi        RWX            managed-nfs-storage   107m
[root@k8s-master01 deploy]# cd /nfs
[root@k8s-master01 nfs]# ll
总用量 0
-rw-r--r-- 1 root root  0 5月   4 09:49 1
drwxrwxrwx 2 root root 28 5月   4 10:34 default-redis-data-redis-master-0-pvc-34e2b8de-13af-4f60-9b45-3259c45e2338
drwxrwxrwx 2 root root 28 5月   4 10:34 default-redis-data-redis-slave-0-pvc-b651443d-ecff-46a4-9507-977180cde3be
drwxrwxrwx 2 root root  6 5月   4 10:11 default-test-claim-pvc-cd79f7a1-2eab-443a-b183-e7730e38840a

---

第三部分: 其他问题.

• 除了default 命令空间之后 还遇到一个 一直提示 avaible 但是不能 bound的错误 经过一番查看, 找到一个合理的解释为:

In v1.16, we will deprecate the SelfLink field in both ObjectMeta and ListMeta objects by: documenting in field definition that it is deprecated and is going to be removed adding a release-note about field deprecation We will also introduce a feature gate to allow disabling setting SelfLink fields and opaque the logic setting it behind this feature gate.

In v1.20 (12 months and 4 release from v1.16) we will switch off the feature gate which will automatically disable setting SelfLinks. However it will still be possible to revert the behavior by changing value of a feature gate.

In v1.21, we will get rid of the whole code propagating those fields and fields themselves. In the meantime, we will go over places referencing that field (see below) and get rid of those too.

• 解决方法为在kube-apiserver.yaml 里面添加内容为:

在文件: /etc/kubernetes/manifests/kube-apiserver.yaml
添加内容:
- --feature-gates=RemoveSelfLink=false
# 注意多主节点, 每个都需要修改.
# 注意kubeadm搭建的 修改了 次文件 apiserver 会自动重启.
修改之后为: 

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.110.83.201:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --feature-gates=RemoveSelfLink=false

• 但是怀疑跟这个没关系, 因为 我用的1.21 说已经完全废除了.

---

第四部分: 简单使用

• 设置storageclass 为默认 default 的存储类

kubectl patch storageclass managed-nfs-storage  -p '{"metadata": {"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"true"}}}'

执行 mysql的安装

helm install mysqltestsc stable/mysql
然后可以执行
kubectl exec -it mysqltestscxxxx /bin/bash 的方式来连接查看
查看密码的命令为:
kubectl get secret --namespace default mysqltestsc -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo