K8S集群etcd备份与恢复

参考链接：

K8S集群多master：Etcd v3备份与恢复

K8S集群单master：Kubernetes Etcd 数据备份与恢复

ETCD系列之一：简介：https://developer.aliyun.com/article/11035

集群节点详情

# kubectl get nodes -o wide
NAME      STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
master1   Ready    master   20h   v1.18.6   10.129.7.222   <none>        CentOS Linux 7 (Core)   3.10.0-1127.13.1.el7.x86_64   docker://19.3.9
master2   Ready    master   19h   v1.18.6   10.129.7.223   <none>        CentOS Linux 7 (Core)   3.10.0-1127.13.1.el7.x86_64   docker://19.3.9
master3   Ready    master   19h   v1.18.6   10.129.7.224   <none>        CentOS Linux 7 (Core)   3.10.0-1127.13.1.el7.x86_64   docker://19.3.9
node1     Ready    <none>   19h   v1.18.6   10.129.7.225   <none>        CentOS Linux 7 (Core)   3.10.0-1127.13.1.el7.x86_64   docker://19.3.9
node2     Ready    <none>   19h   v1.18.6   10.129.7.226   <none>        CentOS Linux 7 (Core)   3.10.0-1127.13.1.el7.x86_64   docker://19.3.9
node3     Ready    <none>   19h   v1.18.6   10.129.7.227   <none>        CentOS Linux 7 (Core)   3.10.0-1127.13.1.el7.x86_64   docker://19.3.9 

安装 etcdctl 命令

etcdctl的安装

etcdctl的二进制文件可以在 github.com/coreos/etcd/releases 选择对应的版本下载，例如可以执行以下install_etcdctl.sh的脚本，修改其中的版本信息。

#!/bin/bash
ETCD_VER=v3.3.4
ETCD_DIR=etcd-download
DOWNLOAD_URL=https://github.com/coreos/etcd/releases/download

# Download
mkdir ${ETCD_DIR}
cd ${ETCD_DIR}
wget ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar -xzvf etcd-${ETCD_VER}-linux-amd64.tar.gz

# install
cd etcd-${ETCD_VER}-linux-amd64
cp etcdctl /usr/local/bin/

export ETCDCTL_API=3

1、备份master1

mkdir -pv /backup        //创建目录

etcdctl --endpoints=https://10.129.7.222:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
--key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
snapshot save /backup/etcd-snapshot.db

2、拷贝 ETCD 备份快照( 从 master1 机器上拷贝备份 )

scp /backup/etcd-snapshot.db root@master2:/backup/
scp /backup/etcd-snapshot.db root@master3:/backup/

3、查看 kube-apiserver、etcd 镜像

docker ps|grep etcd && docker ps|grep kube-apiserver

4、移除且备份 /etc/kubernetes/manifests 目录

mv /etc/kubernetes/manifests /etc/kubernetes/manifests.bak

5、备份现有 Etcd 数据

mv /var/lib/etcd /var/lib/etcd.bak

重复第4步骤，查看 kube-apiserver、etcd 镜像是否停止

6、恢复备份

(1) 在master1上执行

etcdctl snapshot restore /backup/etcd-snapshot.db \
--name etcd-0 \
--initial-cluster "etcd-0=https://10.129.7.222:2380,etcd-1=https://10.129.7.223:2380,etcd-2=https://10.129.7.224:2380" \
--initial-cluster-token etcd-cluster \
--initial-advertise-peer-urls https://10.129.7.222:2380 \
--data-dir=/var/lib/etcd/

(2) 在master2上执行

etcdctl snapshot restore /backup/etcd-snapshot.db \
--name etcd-1 \
--initial-cluster "etcd-0=https://10.129.7.222:2380,etcd-1=https://10.129.7.223:2380,etcd-2=https://10.129.7.224:2380" \
--initial-cluster-token etcd-cluster \
--initial-advertise-peer-urls https://10.129.7.223:2380 \
--data-dir=/var/lib/etcd/

(3) 在master3上执行

etcdctl snapshot restore /backup/etcd-snapshot.db \
--name etcd-2 \
--initial-cluster "etcd-0=https://10.129.7.222:2380,etcd-1=https://10.129.7.223:2380,etcd-2=https://10.129.7.224:2380" \
--initial-cluster-token etcd-cluster \
--initial-advertise-peer-urls https://10.129.7.224:2380 \
--data-dir=/var/lib/etcd/

7、恢复 Kube-Apiserver 与 Etcd 镜像

mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests

8、检查集群

kubectl get nodes

etcdctl --endpoints=https://10.129.7.222:2379 \
--endpoints=https://10.129.7.223:2379 \
--endpoints=https://10.129.7.224:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
endpoint health