kubernetes-集群备份和恢复

一、备份

 

思路：

①集群运行中etcd数据备份到磁盘上

②kubeasz项目创建的集群，需要备份CA证书文件，以及ansible的hosts文件

 

【deploy节点操作】

1：创建存放备份文件目录

  [root@master ~]# mkdir -p /backup/k8s1 

 

2：etcd数据保存到备份目录下

 [root@master ~]# ETCDCTL_API= etcdctl snapshot save /backup/k8s1/snapshot.db
 Snapshot saved at /backup/k8s1/snapshot.db
 [root@master ~]# du -h /backup/k8s1/snapshot.db
 1.6M    /backup/k8s1/snapshot.db

3：拷贝kubernetes目录下ssl文件

 [root@master ~]# cp /etc/kubernetes/ssl/* /backup/k8s1/
 [root@master ~]# ll /backup/k8s1/
 总用量 1628
 -rw-r--r--. 1 root root 1675 12月 10 21:21 admin-key.pem
 -rw-r--r--. 1 root root 1391 12月 10 21:21 admin.pem
 -rw-r--r--. 1 root root 997 12月 10 21:21 aggregator-proxy.csr
 -rw-r--r--. 1 root root 219 12月 10 21:21 aggregator-proxy-csr.json
 -rw-------. 1 root root 1675 12月 10 21:21 aggregator-proxy-key.pem
 -rw-r--r--. 1 root root 1383 12月 10 21:21 aggregator-proxy.pem
 -rw-r--r--. 1 root root 294 12月 10 21:21 ca-config.json
 -rw-r--r--. 1 root root 1675 12月 10 21:21 ca-key.pem
 -rw-r--r--. 1 root root 1350 12月 10 21:21 ca.pem
 -rw-r--r--. 1 root root 1082 12月 10 21:21 kubelet.csr
 -rw-r--r--. 1 root root 283 12月 10 21:21 kubelet-csr.json
 -rw-------. 1 root root 1675 12月 10 21:21 kubelet-key.pem
 -rw-r--r--. 1 root root 1452 12月 10 21:21 kubelet.pem
 -rw-r--r--. 1 root root 1273 12月 10 21:21 kubernetes.csr
 -rw-r--r--. 1 root root 488 12月 10 21:21 kubernetes-csr.json
 -rw-------. 1 root root 1679 12月 10 21:21 kubernetes-key.pem
 -rw-r--r--. 1 root root 1639 12月 10 21:21 kubernetes.pem
 -rw-r--r--. 1 root root 1593376 12月 10 21:32 snapshot.db

4：模拟集群崩溃，执行clean.yml清除操作

[root@master ~]# cd /etc/ansible/

[root@master ansible]# ansible-playbook .clean.yml

二、恢复

 

【deploy节点操作】

1：恢复ca证书 

1 [root@master ansible]# mkdir -p /etc/kubernetes/ssl

2 [root@master ansible]# cp /backup/k8s1/ca* /etc/kubernetes/ssl/

 

2：开始执行重建集群操作

 [root@master ansible]# ansible-playbook .prepare.yml
 [root@master ansible]# ansible-playbook .etcd.yml
 [root@master ansible]# ansible-playbook .docker.yml
 [root@master ansible]# ansible-playbook .kube-master.yml
 [root@master ansible]# ansible-playbook .kube-node.yml

3：暂停etcd服务

[root@master ansible]# ansible etcd -m service -a 'name=etcd state=stopped'

4：清空数据

 [root@master ansible]# ansible etcd -m file -a 'name=/var/lib/etcd/member/ state=absent'
 [DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user
 configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

 192.168.1.203 | CHANGED => {
     "ansible_facts": {
         "discovered_interpreter_python": "/usr/bin/python"
     },
     "changed": true,
     "path": "/var/lib/etcd/member/",
     "state": "absent"
 }
 192.168.1.202 | CHANGED => {
     "ansible_facts": {
         "discovered_interpreter_python": "/usr/bin/python"
     },
     "changed": true,
     "path": "/var/lib/etcd/member/",
     "state": "absent"
 }
 192.168.1.200 | CHANGED => {
     "ansible_facts": {
         "discovered_interpreter_python": "/usr/bin/python"
     },
     "changed": true,
     "path": "/var/lib/etcd/member/",
     "state": "absent"
 }

4：将备份的etcd数据文件同步到每个etcd节点上

 [root@master ansible]# for i in  ; do rsync -av /backup/k8s1 192.168..$i:/backup/; done
 sending incremental file list
 created directory /backup
 k8s1/
 k8s1/admin-key.pem
 k8s1/admin.pem
 k8s1/aggregator-proxy-csr.json
 k8s1/aggregator-proxy-key.pem
 k8s1/aggregator-proxy.csr
 k8s1/aggregator-proxy.pem
 k8s1/ca-config.json
 k8s1/ca-key.pem
 k8s1/ca.pem
 k8s1/kubelet-csr.json
 k8s1/kubelet-key.pem
 k8s1/kubelet.csr
 k8s1/kubelet.pem
 k8s1/kubernetes-csr.json
 k8s1/kubernetes-key.pem
 k8s1/kubernetes.csr
 k8s1/kubernetes.pem
 k8s1/snapshot.db

 sent ,, bytes  received  bytes  ,239.60 bytes/sec
 total size is ,,  speedup is 1.00
 sending incremental file list
 created directory /backup
 k8s1/
 k8s1/admin-key.pem
 k8s1/admin.pem
 k8s1/aggregator-proxy-csr.json
 k8s1/aggregator-proxy-key.pem
 k8s1/aggregator-proxy.csr
 k8s1/aggregator-proxy.pem
 k8s1/ca-config.json
 k8s1/ca-key.pem
 k8s1/ca.pem
 k8s1/kubelet-csr.json
 k8s1/kubelet-key.pem
 k8s1/kubelet.csr
 k8s1/kubelet.pem
 k8s1/kubernetes-csr.json
 k8s1/kubernetes-key.pem
 k8s1/kubernetes.csr
 k8s1/kubernetes.pem
 k8s1/snapshot.db

 sent ,, bytes  received  bytes  ,,066.00 bytes/sec
 total size is ,,  speedup is 1.00

5：在每个etcd节点执行下面数据恢复操作，然后重启etcd

##说明：在/etc/systemd/system/etcd.service找到--inital-cluster etcd1=https://xxxx:2380,etcd2=https://xxxx:2380,etcd3=https://xxxx:2380替换恢复命令中的--initial-cluster{ }变量，--name=【当前etcd-node-name】,最后还需要填写当前节点的IP:2380

①【deploy操作】

 [root@master ansible]# cd /backup/k8s1/
 [root@master k8s1]# ETCDCTL_API= etcdctl snapshot restore snapshot.db  --name etcd1 --initial-cluster etcd1=https://192.168.1.200:2380,etcd2=https://192.168.1.202:2380,etcd3=https://192.168.1.203:2380  --initial-cluster-token etcd-cluster-0 --initial-advertise-peer-urls https://192.168.1.200:2380
 -- ::50.037127 I | mvcc: restore compact to
 -- ::50.052409 I | etcdserver/membership: added member 12229714d8728d0e [https://192.168.1.200:2380] to cluster b8ef796b710cde7d
 -- ::50.052451 I | etcdserver/membership: added member 552fb05951af50c9 [https://192.168.1.203:2380] to cluster b8ef796b710cde7d
 -- ::50.052474 I | etcdserver/membership: added member 8b4f4a6559bf7c2c [https://192.168.1.202:2380] to cluster b8ef796b710cde7d

执行上面步骤后，会在当前节点目录下，生成一个【node-name】.etcd目录文件

 [root@master k8s1]# tree etcd1.etcd/
 etcd1.etcd/
 └── member
     ├── snap
     │   ├── -.snap
     │   └── db
     └── wal
         └── -.wal
 [root@master k8s1]# cp -r etcd1.etcd/member /var/lib/etcd/
 [root@master k8s1]# systemctl restart etcd   

②【etcd2节点操作】

 [root@node1 ~]# cd /backup/k8s1/
 [root@node1 k8s1]# ETCDCTL_API= etcdctl snapshot restore snapshot.db  --name etcd2 --initial-cluster etcd1=https://192.168.1.200:2380,etcd2=https://192.168.1.202:2380,etcd3=https://192.168.1.203:2380  --initial-cluster-token etcd-cluster-0 --initial-advertise-peer-urls https://192.168.1.202:2380
 -- ::35.175032 I | mvcc: restore compact to
 -- ::35.232386 I | etcdserver/membership: added member 12229714d8728d0e [https://192.168.1.200:2380] to cluster b8ef796b710cde7d
 -- ::35.232507 I | etcdserver/membership: added member 552fb05951af50c9 [https://192.168.1.203:2380] to cluster b8ef796b710cde7d
 -- ::35.232541 I | etcdserver/membership: added member 8b4f4a6559bf7c2c [https://192.168.1.202:2380] to cluster b8ef796b710cde7d
 [root@node1 k8s1]# tree etcd2.etcd/
 etcd2.etcd/
 └── member
     ├── snap
     │   ├── -.snap
     │   └── db
     └── wal
         └── -.wal
 [root@node1 k8s1]# cp -r etcd1.etcd/member /var/lib/etcd/
 [root@node1 k8s1]# systemctl restart etcd

③【etcd3节点操作】

 [root@node2 ~]# cd /backup/k8s1/
 [root@node2 k8s1]# ETCDCTL_API= etcdctl snapshot restore snapshot.db  --name etcd3 --initial-cluster etcd1=https://192.168.1.200:2380,etcd2=https://192.168.1.202:2380,etcd3=https://192.168.1.203:2380  --initial-cluster-token etcd-cluster-0 --initial-advertise-peer-urls https://192.168.1.203:2380
 -- ::55.943364 I | mvcc: restore compact to
 -- ::55.988674 I | etcdserver/membership: added member 12229714d8728d0e [https://192.168.1.200:2380] to cluster b8ef796b710cde7d
 -- ::55.988726 I | etcdserver/membership: added member 552fb05951af50c9 [https://192.168.1.203:2380] to cluster b8ef796b710cde7d
 -- ::55.988754 I | etcdserver/membership: added member 8b4f4a6559bf7c2c [https://192.168.1.202:2380] to cluster b8ef796b710cde7d
 [root@node2 k8s1]# tree etcd3.etcd/
 etcd3.etcd/
 └── member
     ├── snap
     │   ├── -.snap
     │   └── db
     └── wal
         └── -.wa
 [root@node2 k8s1]# cp -r etcd1.etcd/member /var/lib/etcd/
 [root@node2 k8s1]# systemctl restart etcd

6：在deploy节点上操作重建网络

[root@master ansible]# cd /etc/ansible/

[root@master ansible]# ansible-playbook tools/change_k8s_network.yml

7：查看pod、svc恢复是否成功

 [root@master ansible]# kubectl get svc
 NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
 kubernetes   ClusterIP   10.68.0.1       <none>        /TCP    5d5h
 nginx        ClusterIP   10.68.241.175   <none>        /TCP     5d4h
 tomcat       ClusterIP   10.68.235.35    <none>        /TCP   76m

 [root@master ansible]# kubectl get pods
 NAME                     READY   STATUS              RESTARTS   AGE
 nginx-7c45b84548-4998z   /     Running                       5d4h
 tomcat-8fc9f5995-9kl5b   /     Running                       77m

三、自动备份、自动恢复

1：一键备份

[root@master ansible]# ansible-playbook /etc/ansible/.backup.yml

2：模拟故障

[root@master ansible]# ansible-playbook /etc/ansible/.clean.yml

修改文件/etc/ansible/roles/cluster-restore/defaults/main.yml，指定要恢复的etcd快照备份，如果不修改就是最新的一次

3：执行自动恢复操作

[root@master ansible]# ansible-playbook /etc/ansible/.restore.yml

[root@master ansible]# ansible-playbook /etc/ansible/tools/change_k8s_network.yml