k8s之持久卷NFS

一、简介

NFS网络存储卷，Kubernetes原生支持NFS作为Kubernetes的持久存储卷之一。NFS可以实现Pod的跨界点的数据持久性。

• 首先需要创建一个nfs 服务器，作为存储服务器；
• 将nfs服务器上导出 (export)的文件系统用作存储卷；
• nfs是文件系统级共享服务，它支持多路挂载请求，可由多个Pod对象同时用作存储卷后端；

二、准备NFS服务器

1、安装NFS-server

root@k8s-node03:~# apt search ^nfs-ker
Sorting... Done
Full Text Search... Done
nfs-kernel-server/jammy-updates 1:2.6.1-1ubuntu1.2 amd64
  support for NFS kernel server

root@k8s-node03:~# apt install nfs-kernel-server
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  nfs-kernel-server
0 upgraded, 1 newly installed, 0 to remove and 65 not upgraded.
Need to get 140 kB of archives.
After this operation, 526 kB of additional disk space will be used.
Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu jammy-updates/main amd64 nfs-kernel-server amd64 1:2.6.1-1ubuntu1.2 [140 kB]
Fetched 140 kB in 2s (72.6 kB/s)
Selecting previously unselected package nfs-kernel-server.
(Reading database ... 94308 files and directories currently installed.)
Preparing to unpack .../nfs-kernel-server_1%3a2.6.1-1ubuntu1.2_amd64.deb ...
Unpacking nfs-kernel-server (1:2.6.1-1ubuntu1.2) ...
Setting up nfs-kernel-server (1:2.6.1-1ubuntu1.2) ...
Created symlink /etc/systemd/system/nfs-client.target.wants/nfs-blkmap.service → /lib/systemd/system/nfs-blkmap.service.
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-server.service → /lib/systemd/system/nfs-server.service.
nfs-mountd.service is a disabled or a static unit, not starting it.
nfsdcld.service is a disabled or a static unit, not starting it.

Creating config file /etc/exports with new version

Creating config file /etc/default/nfs-kernel-server with new version
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...
Scanning candidates...
Scanning linux images...                                                                                                                                                     

Restarting services...
Service restarts being deferred:
 /etc/needrestart/restart.d/dbus.service
 systemctl restart docker.service
 systemctl restart getty@tty1.service
 systemctl restart networkd-dispatcher.service
 systemctl restart systemd-logind.service
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

2、准备共享目录

root@k8s-node03:~# mkdir -pv /data/redis
mkdir: created directory '/data'
mkdir: created directory '/data/redis'
root@k8s-node03:~# vi /etc/exports
root@k8s-node03:~# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#               to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/data/redis  192.168.10.0/24(rw,no_root_squash,no_subtree_check)

3、启动NFS服务

root@k8s-node03:~# systemctl restart nfs-server
root@k8s-node03:~# systemctl status nfs-server
● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
     Active: active (exited) since Tue 2024-01-23 22:14:32 CST; 10s ago
    Process: 1595248 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
    Process: 1595250 ExecStart=/usr/sbin/rpc.nfsd (code=exited, status=0/SUCCESS)
   Main PID: 1595250 (code=exited, status=0/SUCCESS)
        CPU: 10ms

Jan 23 22:14:32 k8s-node03 systemd[1]: Starting NFS server and services...
Jan 23 22:14:32 k8s-node03 systemd[1]: Finished NFS server and services.

三、客户端安装NFS工具

1、安装NFS客户端工具

root@k8s-node02:~# apt install nfs-common
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
nfs-common is already the newest version (1:2.6.1-1ubuntu1.2).
0 upgraded, 0 newly installed, 0 to remove and 65 not upgraded.
root@k8s-node03:~#

2、验证挂载NFS共享目录

root@k8s-node02:~# mount -t nfs 192.168.10.13:/data/redis /mnt/
root@k8s-node02:~# cd /mnt/
root@k8s-node02:/mnt# ls
root@k8s-node02:/mnt# touch test
root@k8s-node02:/mnt# echo 666> test

3、NFS查看已经有数据

root@k8s-node03:~# cat /data/redis/test
666
root@k8s-node03:~#

四、Pod挂载NFS存储卷，验证数据存储的持久性

1、创建Pod并挂载NFS存储卷

root@k8s-master01:~/learning-k8s/examples/volumes# cat pod-with-nfs-vol.yaml
apiVersion: v1
kind: Pod
metadata:
  name: redis-nfs-002
spec:
  containers:
  - name: redis
    image: redis:7-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: redisdata
      mountPath: /data
  volumes:
  - name: redisdata
    nfs:
      server: 192.168.10.13
      path: /data/redis

root@k8s-master01:~/learning-k8s/examples/volumes# kubectl apply -f pod-with-nfs-vol.yaml
root@k8s-master01:~/learning-k8s/examples/volumes# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
redis-nfs-002            1/1     Running   0          19s

2、进入容器创建测试数据

root@k8s-master01:~/learning-k8s/examples/volumes# kubectl exec -it redis-nfs-002 /bin/sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/data # redis-c
redis-check-aof  redis-check-rdb  redis-cli
/data # redis-cli
127.0.0.1:6379> SET mykey test
OK
127.0.0.1:6379> BGSAVE
Background saving started
127.0.0.1:6379> EXIT
/data # ls
dump.rdb  test

3、删除Pod，验证数据持久性

• 第一次nfs Pod被调度在k8s-node02 节点上。

root@k8s-master01:~/learning-k8s/examples/volumes# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
redis-nfs-002            1/1     Running   0          7m49s   10.244.1.4    k8s-node02   <none>           <none>

• 删除Pod,并重新创建，人为调度到k8s-node01节点上

root@k8s-master01:~/learning-k8s/examples/volumes# kubectl delete -f pod-with-nfs-vol.yaml
pod "redis-nfs-002" deleted
root@k8s-master01:~/learning-k8s/examples/volumes# cat pod-with-nfs-vol.yaml
apiVersion: v1
kind: Pod
metadata:
  name: redis-nfs-002
spec:
  nodeName: k8s-node01
  containers:
  - name: redis
    image: redis:7-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: redisdata
      mountPath: /data
  volumes:
  - name: redisdata
    nfs:
      server: 192.168.10.13
      path: /data/redis

root@k8s-master01:~/learning-k8s/examples/volumes# kubectl apply -f pod-with-nfs-vol.yaml
pod/redis-nfs-002 created
root@k8s-master01:~/learning-k8s/examples/volumes# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
redis-nfs-002            1/1     Running   0          4s      10.244.2.31   k8s-node01   <none>           <none>

• 进入容器查看数据没有丢失，实现跨界点的数据持久性

root@k8s-master01:~/learning-k8s/examples/volumes# kubectl exec -it redis-nfs-002 /bin/sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/data # ls
dump.rdb  test
/data # redis-cli
127.0.0.1:6379> GET mykey
"test"
127.0.0.1:6379>