一.系统环境

服务器版本	docker软件版本	CPU架构
CentOS Linux release 7.4.1708 (Core)	Docker version 20.10.12	x86_64

二.前言

etcd 是兼顾一致性与高可用性的键值对数据库，可以作为保存 Kubernetes 所有集群数据的后台数据库。保持 etcd 集群的稳定对 Kubernetes 集群的稳定性至关重要。

三.etcd数据库

3.1 概述

etcd是使用Go语言开发的一个开源的、高可用的分布式key-value存储系统，可以用于配置共享和服务的注册和发现。类似项目有zookeeper和consul。

etcd具有以下特点：

完全复制：集群中的每个节点都可以使用完整的存档
高可用性：Etcd可用于避免硬件的单点故障或网络问题
一致性：每次读取都会返回跨多主机的最新写入
简单：包括一个定义良好、面向用户的API（gRPC）
安全：实现了带有可选的客户端证书身份验证的自动化TLS
快速：每秒10000次写入的基准速度
可靠：使用Raft算法实现了强一致、高可用的服务存储目录

Kubernetes 集群对etcd集群有几点要求：

运行的 etcd 集群个数成员为奇数。
etcd 是一个 leader-based 分布式系统。确保主节点定期向所有从节点发送心跳，以保持集群稳定。
确保不发生资源不足：集群的性能和稳定性对网络和磁盘 I/O 非常敏感。任何资源匮乏都会导致心跳超时，从而导致集群的不稳定。不稳定的情况表明没有选出任何主节点。在这种情况下，集群不能对其当前状态进行任何更改，这意味着不能调度新的 Pod。
保持 etcd 集群的稳定对 Kubernetes 集群的稳定性至关重要。因此，请在专用机器或隔离环境上运行 etcd 集群，以满足所需资源需求。
在生产中运行的 etcd 的最低推荐版本是 3.2.10+。

四.安装部署etcd单节点

4.1 环境介绍

首先安装etcd单节点，etcd架构：etcd1机器作为etcd的服务端，etcd2机器作为客户端访问

服务器	操作系统版本	CPU架构	进程	功能描述
etcd1/192.168.110.133	CentOS Linux release 7.4.1708 (Core)	x86_64	etcd	etcd服务端
etcd2/192.168.110.131	CentOS Linux release 7.4.1708 (Core)	x86_64	etcd	etcd客户端

4.2 配置节点的基本环境

先配置节点的基本环境，所有节点都要同时设置，在此以etcd1作为示例

首先设置主机名

[root@localhost ~]# cat /etc/hostname

etcd1

配置IP地址（可选）

[root@localhost ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens32

[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens32

TYPE=Ethernet

BOOTPROTO=static

NAME=ens32

DEVICE=ens32

ONBOOT=yes

DNS1=114.114.114.114

IPADDR=192.168.110.133

NETMASK=255.255.255.0

GATEWAY=192.168.110.2

ZONE=trusted

#重启网卡

[root@localhost ~]# service network restart

Restarting network (via systemctl):                        [  确定  ]

[root@localhost ~]# systemctl restart NetworkManager

重启机器之后看是否能访问网络

[root@etcd1 ~]# ping www.baidu.com

PING www.a.shifen.com (14.215.177.38) 56(84) bytes of data.

64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=1 ttl=128 time=31.1 ms

64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=2 ttl=128 time=30.5 ms

64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=3 ttl=128 time=31.9 ms

64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=4 ttl=128 time=30.6 ms

^C

--- www.a.shifen.com ping statistics ---

4 packets transmitted, 4 received, 0% packet loss, time 3007ms

rtt min/avg/max/mdev = 30.593/31.075/31.926/0.533 ms

配置IP和主机名映射

[root@etcd1 ~]# vim /etc/hosts

[root@etcd1 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.110.133 etcd1

192.168.110.131 etcd2

192.168.110.132 etcd3

复制到另外两个主机

[root@etcd1 ~]# scp /etc/hosts etcd3:/etc/hosts

root@etcd3's password:

hosts                                                                                                                                                                          100%  224   281.3KB/s   00:00    

[root@etcd1 ~]# scp /etc/hosts etcd2:/etc/hosts

root@etcd2's password:

hosts                                                                                                                                                                          100%  224   218.1KB/s   00:00

能相互ping通则正常

[root@etcd1 ~]# ping etcd1

PING etcd1 (192.168.110.133) 56(84) bytes of data.

64 bytes from etcd1 (192.168.110.133): icmp_seq=1 ttl=64 time=0.029 ms

64 bytes from etcd1 (192.168.110.133): icmp_seq=2 ttl=64 time=0.033 ms

64 bytes from etcd1 (192.168.110.133): icmp_seq=3 ttl=64 time=0.043 ms

^C

--- etcd1 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2002ms

rtt min/avg/max/mdev = 0.029/0.035/0.043/0.005 ms

[root@etcd1 ~]# ping etcd2

PING etcd2 (192.168.110.131) 56(84) bytes of data.

64 bytes from etcd2 (192.168.110.131): icmp_seq=1 ttl=64 time=1.61 ms

64 bytes from etcd2 (192.168.110.131): icmp_seq=2 ttl=64 time=1.92 ms

^C

--- etcd2 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 1010ms

rtt min/avg/max/mdev = 1.612/1.769/1.926/0.157 ms

[root@etcd1 ~]# ping etcd3

PING etcd3 (192.168.110.132) 56(84) bytes of data.

64 bytes from etcd3 (192.168.110.132): icmp_seq=1 ttl=64 time=0.484 ms

64 bytes from etcd3 (192.168.110.132): icmp_seq=2 ttl=64 time=2.65 ms

64 bytes from etcd3 (192.168.110.132): icmp_seq=3 ttl=64 time=2.65 ms

^C

--- etcd3 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2021ms

rtt min/avg/max/mdev = 0.484/1.931/2.657/1.023 ms

关闭屏保（可选）

[root@etcd1 ~]# setterm -blank 0

配置yum源

[root@etcd1 ~]# rm -rf /etc/yum.repos.d/* ;wget ftp://ftp.rhce.cc/k8s/* -P /etc/yum.repos.d/

关闭selinux

[root@etcd1 ~]# cat /etc/selinux/config 

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

#     enforcing - SELinux security policy is enforced.

#     permissive - SELinux prints warnings instead of enforcing.

#     disabled - No SELinux policy is loaded.

SELINUX=disabled

# SELINUXTYPE= can take one of three two values:

#     targeted - Targeted processes are protected,

#     minimum - Modification of targeted policy. Only selected processes are protected.

#     mls - Multi Level Security protection.

SELINUXTYPE=targeted 

[root@etcd1 ~]# getenforce

Disabled

配置防火墙允许所有数据通过

[root@etcd1 ~]# firewall-cmd --set-default-zone=trusted

Warning: ZONE_ALREADY_SET: trusted

success

[root@etcd1 ~]# firewall-cmd --get-default-zone

trusted

4.3 安装部署etcd单节点

etcd1机器安装etcd

[root@etcd1 ~]# yum -y install etcd

etcd的配置文件为/etc/etcd/etcd.conf ,注释如下：

[root@etcd1 ~]# vim /etc/etcd/etcd.conf 

#配置文件简单注释：etcd节点间通信端口2380，客户端访问etcd的端口为2379

#数据目录：ETCD_DATA_DIR="/var/lib/etcd/default.etcd

[root@etcd1 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$"

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

#ETCD_LISTEN_PEER_URLS="http://localhost:2380"

ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"

#etcd名字

ETCD_NAME="default"

修改配置文件如下，添加etcd1的IP和端口

[root@etcd1 ~]# vim /etc/etcd/etcd.conf

[root@etcd1 ~]# cat /etc/etcd/etcd.conf

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="http://192.168.110.133:2380,http://localhost:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"

ETCD_NAME="default"

#[Clustering]

ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"

启动etcd

[root@etcd1 ~]# systemctl enable etcd --now

Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.

[root@etcd1 ~]# systemctl status etcd

● etcd.service - Etcd Server

   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)

   Active: active (running) since 一 2022-01-10 22:53:13 CST; 9s ago

 Main PID: 1148 (etcd)

   CGroup: /system.slice/etcd.service

           └─1148 /usr/bin/etcd --name=default --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://192.168.110.133:2379,http://localhost:2379

1月 10 22:53:13 etcd1 etcd[1148]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2

1月 10 22:53:13 etcd1 etcd[1148]: setting up the initial cluster version to 3.3

1月 10 22:53:13 etcd1 etcd[1148]: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32

1月 10 22:53:13 etcd1 etcd[1148]: set the initial cluster version to 3.3

1月 10 22:53:13 etcd1 etcd[1148]: enabled capabilities for version 3.3

1月 10 22:53:13 etcd1 etcd[1148]: ready to serve client requests

1月 10 22:53:13 etcd1 etcd[1148]: ready to serve client requests

1月 10 22:53:13 etcd1 etcd[1148]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

1月 10 22:53:13 etcd1 etcd[1148]: serving insecure client requests on 192.168.110.133:2379, this is strongly discouraged!

1月 10 22:53:13 etcd1 systemd[1]: Started Etcd Server.

查看etcd的成员，查看有几个节点

[root@etcd1 ~]# etcdctl member list

8e9e05c52164694d: name=default peerURLs=http://localhost:2380 clientURLs=http://localhost:2379 isLeader=true

查看集群健康状态

[root@etcd1 ~]# etcdctl cluster-health

member 8e9e05c52164694d is healthy: got healthy result from http://localhost:2379

cluster is healthy

4.4 使用客户端访问etcd服务

现在etcd2机器作为客户端访问etcd1机器上的etcd服务

首先etcd2机器安装etcd

[root@etcd2 ~]# yum -y install etcd

查看etcdctl的帮助

[root@etcd2 ~]# etcdctl --help

连接192.168.110.133上的etcd服务，发现拒绝

[root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 ls /

Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp [::1]:2379: connect: connection refused

error #0: dial tcp [::1]:2379: connect: connection refused

修改etcd1上的配置文件ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"，并重启etcd服务

[root@etcd1 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$"

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="http://192.168.110.133:2380,http://localhost:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"

ETCD_NAME="default"

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"

#重启

[root@etcd1 ~]# systemctl restart etcd

[root@etcd1 ~]# systemctl status etcd

此时客户端连接192.168.110.133成功，ls /表示查询根目录下内容

[root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 ls /

4.4.1 使用2版本API管理etcd

etcd写入数据的时候有两个版本：2版本和3版本，默认是2版本，Set environment variable ETCDCTL_API=3 to use v3 API or ETCDCTL_API=2 to use v2 API

现在还没有数据

[root@etcd1 ~]# etcdctl ls /

etcd1创建数据

[root@etcd1 ~]# etcdctl mkdir /comeon

[root@etcd1 ~]# etcdctl ls /

/comeon

客户端连接也可以看到数据

[root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 ls /

/comeon

删除数据

[root@etcd1 ~]# etcdctl rmdir /comeon

[root@etcd1 ~]# etcdctl ls /

4.4.2 使用3版本API管理etcd

现在使用3版本的API

[root@etcd1 ~]# export ETCDCTL_API=3

设置为3版本的API之后，help显示的内容也会不同

[root@etcd1 ~]# etcdctl --help

NAME:

        etcdctl - A simple command line client for etcd3.

USAGE:

        etcdctl

VERSION:

        3.3.11

API VERSION:

        3.3

COMMANDS:

        get			Gets the key or a range of keys

        put			Puts the given key into the store

        del			Removes the specified key or range of keys [key, range_end)

        txn			Txn processes all the requests in one transaction

        compaction		Compacts the event history in etcd

        alarm disarm		Disarms all alarms

        alarm list		Lists all alarms

        defrag			Defragments the storage of the etcd members with given endpoints

        endpoint health		Checks the healthiness of endpoints specified in `--endpoints` flag

        endpoint status		Prints out the status of endpoints specified in `--endpoints` flag

        endpoint hashkv		Prints the KV history hash for each endpoint in --endpoints

        move-leader		Transfers leadership to another etcd cluster member.

        watch			Watches events stream on keys or prefixes

        version			Prints the version of etcdctl

        lease grant		Creates leases

        lease revoke		Revokes leases

        lease timetolive	Get lease information

        lease list		List all active leases

        lease keep-alive	Keeps leases alive (renew)

        member add		Adds a member into the cluster

        member remove		Removes a member from the cluster

        member update		Updates a member in the cluster

        member list		Lists all members in the cluster

        snapshot save		Stores an etcd node backend snapshot to a given file

        snapshot restore	Restores an etcd member snapshot to an etcd directory

        snapshot status		Gets backend snapshot status of a given file

        make-mirror		Makes a mirror at the destination etcd cluster

        migrate			Migrates keys in a v2 store to a mvcc store

        lock			Acquires a named lock

        elect			Observes and participates in leader election

        auth enable		Enables authentication

        auth disable		Disables authentication

        user add		Adds a new user

        user delete		Deletes a user

        user get		Gets detailed information of a user

        user list		Lists all users

        user passwd		Changes password of user

        user grant-role		Grants a role to a user

        user revoke-role	Revokes a role from a user

        role add		Adds a new role

        role delete		Deletes a role

        role get		Gets detailed information of a role

        role list		Lists all roles

        role grant-permission	Grants a key to a role

        role revoke-permission	Revokes a key from a role

        check perf		Check the performance of the etcd cluster

        help			Help about any command

......

  -w, --write-out="simple"			set the output format (fields, json, protobuf, simple, table)

写数据

[root@etcd1 ~]# etcdctl put student1 99

OK

查数据

[root@etcd1 ~]# etcdctl get student1

student1

99

注意：2版本API和3版本API不可互用，从k8s1.5版本，etcd就开始使用3版本往etcd里写数据

[root@etcd2 ~]# export ETCDCTL_API=3

[root@etcd2 ~]# etcdctl --endpoints http://192.168.110.133:2379 get student1

student1

99

etcd单节点搭建完毕，接下来添加两个节点变为etcd集群。

五.安装部署etcd集群

5.1 环境介绍

etcd集群架构：etcd1为leader，etcd2为follower，etcd3为follower

服务器	操作系统版本	CPU架构	进程	功能描述
etcd1/192.168.110.133	CentOS Linux release 7.4.1708 (Core)	x86_64	etcd	leader
etcd2/192.168.110.131	CentOS Linux release 7.4.1708 (Core)	x86_64	etcd	follower
etcd3/192.168.110.132	CentOS Linux release 7.4.1708 (Core)	x86_64	etcd	follower

5.2 把etcd2机器加入集群

首先还原环境变量

[root@etcd1 ~]# unset ETCDCTL_API

停止etcd1机器的etcd服务

[root@etcd1 ~]# systemctl stop etcd

[root@etcd1 ~]# systemctl status etcd

● etcd.service - Etcd Server

   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)

   Active: inactive (dead) since 二 2022-01-11 15:21:30 CST; 49s ago

  Process: 1582 ExecStart=/bin/bash -c GOMAXPROCS=$(nproc) /usr/bin/etcd --name="${ETCD_NAME}" --data-dir="${ETCD_DATA_DIR}" --listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}" (code=killed, signal=TERM)

 Main PID: 1582 (code=killed, signal=TERM)

查看etcd的数据目录

[root@etcd1 ~]# cat /etc/etcd/etcd.conf | grep -i data

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

删除原始数据

[root@etcd1 ~]# rm -rf /var/lib/etcd/*

修改配置文件，把etcd2机器加入配置

[root@etcd1 ~]# vim /etc/etcd/etcd.conf

[root@etcd1 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$"

#配置数据目录

ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"

ETCD_LISTEN_PEER_URLS="http://192.168.110.133:2380,http://localhost:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"

ETCD_NAME="etcd133"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.110.133:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.133:2379,http://localhost:2379"

#目前是两个节点，所以这里是两个节点的etcd

ETCD_INITIAL_CLUSTER="etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380"

#集群token

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#新创建etcd集群的时候ETCD_INITIAL_CLUSTER_STATE="new"，往已经存在的etcd集群添加etcd节点时：ETCD_INITIAL_CLUSTER_STATE="existing"

ETCD_INITIAL_CLUSTER_STATE="new"

复制配置文件到etcd2

[root@etcd1 ~]# scp /etc/etcd/etcd.conf etcd2:/etc/etcd/etcd.conf

root@etcd2's password:

etcd.conf                                                                                                                                                                       100% 1813     1.7MB/s   00:00

etcd2机器修改配置文件

[root@etcd2 ~]# unset ETCDCTL_API

[root@etcd2 ~]# vim /etc/etcd/etcd.conf 

[root@etcd2 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$"

ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"

ETCD_LISTEN_PEER_URLS="http://192.168.110.131:2380,http://localhost:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.110.131:2379,http://localhost:2379"

ETCD_NAME="etcd131"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.110.131:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.131:2379,http://localhost:2379"

ETCD_INITIAL_CLUSTER="etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

两个节点都启动etcd

[root@etcd1 ~]# systemctl start etcd 

[root@etcd1 ~]# systemctl status etcd

[root@etcd2 ~]# systemctl enable etcd --now

Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.

[root@etcd2 ~]# systemctl status etcd

查看etcd集群成员，可以看到192.168.110.133节点是Leader

[root@etcd1 ~]# etcdctl member list

341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false

ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true

集群健康状态

[root@etcd1 ~]# etcdctl cluster-health

member 341a3c460c1c993a is healthy: got healthy result from http://192.168.110.131:2379

member ab23bcc86cf3190b is healthy: got healthy result from http://192.168.110.133:2379

cluster is healthy

现在两个节点的etcd集群搭建完毕，数据也同步了

[root@etcd1 ~]# etcdctl ls /

[root@etcd1 ~]# etcdctl mkdir /public

[root@etcd1 ~]# export ETCDCTL_API=3

[root@etcd1 ~]# etcdctl put student1 59

OK

[root@etcd2 ~]# etcdctl ls /

/public

[root@etcd2 ~]# export ETCDCTL_API=3

[root@etcd2 ~]# etcdctl get student1

student1

59

5.3 把etcd3机器加入集群

现在添加一个节点etcd3到集群

etcd3安装etcd

[root@etcd3 ~]# yum -y install etcd

加入新节点的时候，使用API2版本

[root@etcd1 ~]# export ETCDCTL_API=2

执行添加节点命令，注意：ETCD_INITIAL_CLUSTER_STATE="existing"

[root@etcd1 ~]# etcdctl member add etcd132 http://192.168.110.132:2380

Added member named etcd132 with ID 7d816f4fa2bea295 to cluster

ETCD_NAME="etcd132"

ETCD_INITIAL_CLUSTER="etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380,etcd133=http://192.168.110.133:2380"

ETCD_INITIAL_CLUSTER_STATE="existing"

查看集群成员，发现192.168.110.132显示不正常

[root@etcd1 ~]# etcdctl member list

341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false

7d816f4fa2bea295[unstarted]: peerURLs=http://192.168.110.132:2380

ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true

复制配置文件到etcd3

[root@etcd1 ~]# scp /etc/etcd/etcd.conf etcd3:/etc/etcd/etcd.conf

root@etcd3's password:

etcd.conf                                                                                                                                                                       100% 1813     1.1MB/s   00:00

etcd3修改配置文件

 #注意：添加到一个已经存在的集群，etcd133和etcd131配置文件不变，只修改etcd132配置文件

[root@etcd3 ~]# vim /etc/etcd/etcd.conf 

[root@etcd3 ~]# cat /etc/etcd/etcd.conf | egrep -v "^#|^$"

ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"

ETCD_LISTEN_PEER_URLS="http://192.168.110.132:2380,http://localhost:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.110.132:2379,http://localhost:2379"

ETCD_NAME="etcd132"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.110.132:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.110.132:2379,http://localhost:2379"

ETCD_INITIAL_CLUSTER="etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="existing"

etcd3启动etcd

[root@etcd3 ~]# systemctl enable etcd --now

Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.

查看etcd集群成员，etcd集群有三个节点了

[root@etcd1 ~]# etcdctl member list

341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false

7d816f4fa2bea295: name=etcd132 peerURLs=http://192.168.110.132:2380 clientURLs=http://192.168.110.132:2379,http://localhost:2379 isLeader=false

ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true

自此etcd集群搭建完毕，接下来进行etcd服务的管理。

六.etcd做快照备份数据

生产环境中，有些重要数据是要备份的，以免造成数据丢失。etcd对数据做快照可以进行数据备份。

设置API版本为3

[root@etcd1 ~]# export ETCDCTL_API=3

为了避免数据丢失，可以进行数据备份，即数据做快照，数据快照的help如下

[root@etcd1 ~]# etcdctl snap --help

NAME:

        snapshot - Manages etcd node snapshots

USAGE:

        etcdctl snapshot <subcommand>

API VERSION:

        3.3

COMMANDS:

        save	Stores an etcd node backend snapshot to a given file

        restore	Restores an etcd member snapshot to an etcd directory

        status	Gets backend snapshot status of a given file

GLOBAL OPTIONS:

      --cacert=""				verify certificates of TLS-enabled secure servers using this CA bundle

      --cert=""					identify secure client using this TLS certificate file

      --command-timeout=5s			timeout for short running command (excluding dial timeout)

      --debug[=false]				enable client-side debug logging

      --dial-timeout=2s				dial timeout for client connections

  -d, --discovery-srv=""			domain name to query for SRV records describing cluster endpoints

      --endpoints=[127.0.0.1:2379]		gRPC endpoints

      --hex[=false]				print byte strings as hex encoded strings

      --insecure-discovery[=true]		accept insecure SRV records describing cluster endpoints

      --insecure-skip-tls-verify[=false]	skip server certificate verification

      --insecure-transport[=true]		disable transport security for client connections

      --keepalive-time=2s			keepalive time for client connections

      --keepalive-timeout=6s			keepalive timeout for client connections

      --key=""					identify secure client using this TLS key file

      --user=""					username[:password] for authentication (prompt if password is not supplied)

  -w, --write-out="simple"			set the output format (fields, json, protobuf, simple, table)

进行快照

[root@etcd1 ~]# etcdctl snap save student.data

Snapshot saved at student.data

七.etcd恢复数据

删除数据使用快照进行恢复

[root@etcd1 ~]# etcdctl del student1

1

[root@etcd1 ~]# etcdctl del student2

1

[root@etcd1 ~]# etcdctl get student1

[root@etcd1 ~]# etcdctl get student2

使用快照恢复数据的时候需要把快照复制到其他节点

[root@etcd1 ~]# scp student.data etcd2:~/

root@etcd2's password:

student.data                                                                                                                                                                    100%   20KB  14.5MB/s   00:00

[root@etcd1 ~]# scp student.data etcd3:~/

root@etcd3's password:

student.data                                                                                                                                                                    100%   20KB   8.1MB/s   00:00

在所有节点停止etcd并清空数据

[root@etcd1 ~]# systemctl stop etcd

[root@etcd1 ~]# rm -rf /var/lib/etcd/*

[root@etcd1 ~]# chown etcd:etcd student.data 

[root@etcd2 ~]# systemctl stop etcd

[root@etcd2 ~]# rm -rf /var/lib/etcd/*

[root@etcd2 ~]# chown etcd:etcd student.data

[root@etcd3 ~]# systemctl stop etcd

[root@etcd3 ~]# rm -rf /var/lib/etcd/*

[root@etcd3 ~]# chown etcd:etcd student.data

在每个节点上使用快照恢复数据

[root@etcd1 ~]# etcdctl snapshot restore student.data --name etcd133 --initial-cluster etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380 --initial-advertise-peer-urls http://192.168.110.133:2380 --data-dir /var/lib/etcd/cluster.etcd

2022-01-11 16:44:08.375319 I | etcdserver/membership: added member 341a3c460c1c993a [http://192.168.110.131:2380] to cluster dd7594df5e81191b

2022-01-11 16:44:08.375393 I | etcdserver/membership: added member 4679fe0fcb37326d [http://192.168.110.132:2380] to cluster dd7594df5e81191b

2022-01-11 16:44:08.375404 I | etcdserver/membership: added member ab23bcc86cf3190b [http://192.168.110.133:2380] to cluster dd7594df5e81191b

[root@etcd2 ~]# etcdctl snapshot restore student.data --name etcd131 --initial-cluster etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380 --initial-advertise-peer-urls http://192.168.110.131:2380 --data-dir /var/lib/etcd/cluster.etcd

2022-01-11 16:45:18.378931 I | etcdserver/membership: added member 341a3c460c1c993a [http://192.168.110.131:2380] to cluster dd7594df5e81191b

2022-01-11 16:45:18.378991 I | etcdserver/membership: added member 4679fe0fcb37326d [http://192.168.110.132:2380] to cluster dd7594df5e81191b

2022-01-11 16:45:18.379000 I | etcdserver/membership: added member ab23bcc86cf3190b [http://192.168.110.133:2380] to cluster dd7594df5e81191b

[root@etcd3 ~]# etcdctl snapshot restore student.data --name etcd132 --initial-cluster etcd133=http://192.168.110.133:2380,etcd131=http://192.168.110.131:2380,etcd132=http://192.168.110.132:2380 --initial-advertise-peer-urls http://192.168.110.132:2380 --data-dir /var/lib/etcd/cluster.etcd

2022-01-11 16:46:26.826533 I | etcdserver/membership: added member 341a3c460c1c993a [http://192.168.110.131:2380] to cluster dd7594df5e81191b

2022-01-11 16:46:26.826584 I | etcdserver/membership: added member 4679fe0fcb37326d [http://192.168.110.132:2380] to cluster dd7594df5e81191b

2022-01-11 16:46:26.826595 I | etcdserver/membership: added member ab23bcc86cf3190b [http://192.168.110.133:2380] to cluster dd7594df5e81191b

修改所有节点数据目录的属主

[root@etcd1 ~]# chown -R etcd:etcd /var/lib/etcd/

[root@etcd2 ~]# chown -R etcd:etcd /var/lib/etcd/

[root@etcd3 ~]# chown -R etcd:etcd /var/lib/etcd/

启动etcd

[root@etcd1 ~]# systemctl start etcd

[root@etcd2 ~]# systemctl start etcd

[root@etcd3 ~]# systemctl start etcd

可以发现数据已经恢复

[root@etcd1 ~]# etcdctl get student1

student1

59

[root@etcd3 ~]# etcdctl get student2

student2

62

八.Kubernetes(k8s)中以pod方式运行的etcd

etcd在Kubernetes集群中可以以pod的方式运行，也可以以物理机部署的方式运行，本章讲解以pod方式运行的etcd。

在此之前，需要有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Centos7 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/16686769.html

在k8s中etcd以pod的方式运行，那配置文件在哪里，数据目录在哪里？

查看etcd pod，k8s中etcd为etcd-k8scloude1

[root@k8scloude1 ~]# kubectl get pods

NAME                                       READY   STATUS    RESTARTS   AGE

calico-kube-controllers-6b9fbfff44-4jzkj   1/1     Running   4          2d4h

calico-node-bdlgm                          1/1     Running   2          2d4h

calico-node-hx8bk                          1/1     Running   2          2d4h

calico-node-nsbfs                          1/1     Running   2          2d4h

coredns-545d6fc579-7wm95                   1/1     Running   2          2d4h

coredns-545d6fc579-87q8j                   1/1     Running   2          2d4h

etcd-k8scloude1                            1/1     Running   2          2d4h

kube-apiserver-k8scloude1                  1/1     Running   2          2d4h

kube-controller-manager-k8scloude1         1/1     Running   2          2d4h

kube-proxy-599xh                           1/1     Running   2          2d4h

kube-proxy-lpj8z                           1/1     Running   2          2d4h

kube-proxy-zxlk9                           1/1     Running   2          2d4h

kube-scheduler-k8scloude1                  1/1     Running   2          2d4h

metrics-server-bcfb98c76-k5dmj             1/1     Running   1          33h

在k8s中etcd以pod的形式运行，此etcd的配置文件在哪？在/etc/kubernetes/manifests/etcd.yaml

[root@k8scloude1 ~]# ls /etc/kubernetes/manifests/etcd.yaml

/etc/kubernetes/manifests/etcd.yaml

[root@k8scloude1 ~]# cat !$

cat /etc/kubernetes/manifests/etcd.yaml

apiVersion: v1

kind: Pod

metadata:

  annotations:

    kubeadm.kubernetes.io/etcd.advertise-client-urls: https://192.168.110.130:2379

  creationTimestamp: null

  labels:

    component: etcd

    tier: control-plane

  name: etcd

  namespace: kube-system

spec:

  containers:

  - command:

    - etcd

    - --advertise-client-urls=https://192.168.110.130:2379

    - --cert-file=/etc/kubernetes/pki/etcd/server.crt

    - --client-cert-auth=true

    - --data-dir=/var/lib/etcd

    - --initial-advertise-peer-urls=https://192.168.110.130:2380

    - --initial-cluster=k8scloude1=https://192.168.110.130:2380

    - --key-file=/etc/kubernetes/pki/etcd/server.key

    - --listen-client-urls=https://127.0.0.1:2379,https://192.168.110.130:2379

    - --listen-metrics-urls=http://127.0.0.1:2381

    - --listen-peer-urls=https://192.168.110.130:2380

    - --name=k8scloude1

    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt

    - --peer-client-cert-auth=true

    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key

    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

    - --snapshot-count=10000

    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

    image: registry.aliyuncs.com/google_containers/etcd:3.4.13-0

    imagePullPolicy: IfNotPresent

    livenessProbe:

      failureThreshold: 8

      httpGet:

        host: 127.0.0.1

        path: /health

        port: 2381

        scheme: HTTP

      initialDelaySeconds: 10

      periodSeconds: 10

      timeoutSeconds: 15

    name: etcd

    resources:

      requests:

        cpu: 100m

        ephemeral-storage: 100Mi

        memory: 100Mi

    startupProbe:

      failureThreshold: 24

      httpGet:

        host: 127.0.0.1

        path: /health

        port: 2381

        scheme: HTTP

      initialDelaySeconds: 10

      periodSeconds: 10

      timeoutSeconds: 15

    volumeMounts:

    - mountPath: /var/lib/etcd

      name: etcd-data

    - mountPath: /etc/kubernetes/pki/etcd

      name: etcd-certs

  hostNetwork: true

  priorityClassName: system-node-critical

  volumes:

  - hostPath:

      path: /etc/kubernetes/pki/etcd

      type: DirectoryOrCreate

    name: etcd-certs

  - hostPath:

      path: /var/lib/etcd

      type: DirectoryOrCreate

    name: etcd-data

status: {}

可以发现挂载了数据卷，数据目录在/var/lib/etcd/

[root@k8scloude1 ~]# ls /var/lib/etcd/

member

[root@k8scloude1 ~]# ls /var/lib/etcd/member/

snap  wal

Kubernetes后台数据库etcd：安装部署etcd集群，数据备份与恢复的更多相关文章

kubernetes系列03—kubeadm安装部署K8S集群
本文收录在容器技术学习系列文章总目录 1.kubernetes安装介绍 1.1 K8S架构图 1.2 K8S搭建安装示意图 1.3 安装kubernetes方法 1.3.1 方法1:使用kubeadm ...
supervisor的安装部署及集群管理
supervisor的安装部署及集群管理 supervisor官网:http://www.supervisord.org/ 参考链接: http://blog.csdn.net/xyang81/art ...
centos7.8 安装部署 k8s 集群
centos7.8 安装部署 k8s 集群目录 centos7.8 安装部署 k8s 集群环境说明 Docker 安装 k8s 安装准备工作 Master 节点安装 k8s 版本查看安装 kub ...
4. 利用MySQL Shell安装部署MGR集群 | 深入浅出MGR
GreatSQL社区原创内容未经授权不得随意使用,转载请联系小编并注明来源. 目录 1. 安装准备 2. 利用MySQL Shell构建MGR集群 3. MySQL Shell接管现存的MGR集群 4 ...
使用docker安装部署Spark集群来训练CNN（含Python实例）
使用docker安装部署Spark集群来训练CNN(含Python实例) http://blog.csdn.net/cyh_24/article/details/49683221 实验室有4台神服务器 ...
【大数据之数据仓库】安装部署GreenPlum集群
本篇将向大家介绍如何快捷的安装部署GreenPlum测试集群,大家可以跟着我一块儿实践一把^_^ 1.主机资源申请2台网易云主机,操作系统必须是RedHat或者CentOS,配置尽量高一点.如果是s ...
Docker安装部署es集群
Docker安装部署es集群:环境准备:已安装docker的centos服务器一台1．拉取es版本docker pull elasticsearch:5.6.82．新建文件夹数据挂载目录和配 ...
hadoop 2.5.1单机安装部署伪集群
环境:ubuntu 14.04 server 64版本 hadoop 2.5.1 jdk 1.6 部署的步骤主要参考了http://blog.csdn.net/greensurfer/article/ ...
安装部署Kafka集群
kafka是一个开源的分布式消息订阅系统(消息中间件) 安装过程 1.下载kafka_2.11-0.10.1.0.gz(ps:千万不要下错了,博主就是下到了src文件上去了,kafka中的zookee ...
3. 安装部署MGR集群 | 深入浅出MGR
GreatSQL社区原创内容未经授权不得随意使用,转载请联系小编并注明来源. 目录 1. 安装准备 2. 初始化MySQL Server 3. 初始化MGR第一个节点 4. 继续设置另外两个节点 5. ...

随机推荐

浪姐打分看不够？用几行Python代码模拟评委打分
大家好鸭~我是小熊猫比赛大家都看过吧,每次是不是都对比赛成绩充满期待.特别是浪姐的打分看的简直欲罢不能- 今天就用Python来模拟评委打分,这个案例很短也很简单,很适合新手跟小白练习. 在某次十佳歌 ...
【.NET+MQTT】.NET6 环境下实现MQTT通信，以及服务端、客户端的双边消息订阅与发布的代码演示
前言: MQTT广泛应用于工业物联网.智能家居.各类智能制造或各类自动化场景等.MQTT是一个基于客户端-服务器的消息发布/订阅传输协议,在很多受限的环境下,比如说机器与机器通信.机器与物联网通信等. ...
docker下node环境搭建
初始化⼀个NodeJs程序以下操作必须已经安装了NodeJS. ⾸先创建⼀个空⽂件夹.并创建以下⽂件: server.js package.json Dockerfile .dockerignore ...
关于使用netstat -lantup查看的SSHD 6010端口解释
关于使用netstat -lantup查看的SSHD 6010端口解释: 1.使用netstat -lantup查看当前系统开启的服务端口 tcp6 0 0 ::1:6010 ...
聊聊 C++ 大一统的初始化运算符 {}
一:背景最近发现 C++ 中的类型初始化操作,没有 {} 运算符搞不定的,蛮有意思,今天我们就来逐一列一下各自的用法以及汇编展现,本来想分为值类型和引用类型两大块,但发现在 C++ 中没这种 ...
关于 Python 的 import
好久以前就被 Python 的相对与绝对导入所困扰.去年粗浅探究后自以为完全理解,近来又因 sys.path[0] 和 os.getcwd() 的不一致而刷新了认知... Python 官方文档 5. ...
NOI / 1.4编程基础之逻辑表达式与条件分支讲解-02:输出绝对值
02:输出绝对值总时间限制: 1000ms 内存限制: 65536kB 题目: 描述输入一个浮点数,输出这个浮点数的绝对值. 输入输入一个浮点数,其绝对值不超过10000. 输出输出这个浮点数 ...
AtCoder Beginner Contest 260 F - Find 4-cycle
题目传送门:F - Find 4-cycle (atcoder.jp) 题意: 给定一个无向图,其包含了S.T两个独立点集(即S.T内部间的任意两点之间不存在边),再给出图中的M条边(S中的点与T中的 ...
题解【洛谷 P1246 编码】
题目编码工作常被运用于密文或压缩传输.这里我们用一种最简单的编码方式进行编码:把一些有规律的单词编成数宇. 字母表中共有 \(26\) 个字母 \(\{\tt a,b,\cdots,z\}\),这些 ...
【Azure 应用服务】部署Kafka Trigger Function到Azure Function服务中，解决自定义域名解析难题
问题描述经过前两篇文章,分别使用VM搭建了Kafka服务,创建了Azure Function项目,并且都在本地运行成功. [Azure Developer]在Azure VM (Windows) 中 ...

Kubernetes后台数据库etcd：安装部署etcd集群，数据备份与恢复