使用 etcdadm 快速、弹性部署 etcd 集群

Etcd 是一个可靠的分布式键值存储, 常用于分布式系统关键数据的存储；而 etcdadm 是一个用于操作 etcd 集群的命令行工具，它可以轻松创建集群、向现有集群添加成员、从现有集群中删除成员等操作；其使用方式类似 kubeadm, 即主要操作流程为: 先启动第一个集群节点，后续节点直接 join 即可

建议通过 PC 端，访问运维技术帮 (https://ywjsbang.com) 以获取更好阅读体验，由于精力有限，该文章的后续更新、完善仅限此处，望理解 !!

测试环境

节点主机名	节点 IP 地址	系统版本	etcd 版本	etcdadm 版本
c7	192.168.31.37	CentOS 7.9.2009 ( 5.4.180-1.el7 )	V3.5.5	V0.1.5
c8	192.168.31.38	同上	同上	同上
c9	192.168.31.39	同上	同上	同上

安装 etcdadm

1、预编译二进制安装

wget https://github.com/kubernetes-sigs/etcdadm/releases/download/v0.1.5/etcdadm-linux-amd64

mv etcdadm-linux-amd64 /usr/local/bin/etcdadm

chmod +x /usr/local/bin/etcdadm

scp /usr/local/bin/etcdadm 192.168.31.{38,39}:/usr/local/bin/

2、各节点系统防火墙放行端口 2379，2380

firewall-cmd --add-port=2379/tcp

firewall-cmd --add-port=2380/tcp

初始化 etcd 节点

1、初始化第一个 etcd 集群节点

etcdadm init \

--version "3.5.5" \

--init-system "systemd" \

--install-dir "/opt/bin/" \

--certs-dir "/etc/etcd/pki" \

--data-dir "/var/lib/etcd" \

--release-url "https://github.com/etcd-io/etcd/releases/download"

# 主要选项解析

--version      # 指定部署的 etcd 版本

--init-system  # 设置 etcd 进程管理方式，默认 systemd，取值 kubelet 时，则以容器方法运行 etcd 进程

--install-dir  # etcd 二进制程序安装目录

2、etcdadm init 初始化过程解析

# 下载解压、安装二进制文件 etcd、etcdctl

2022-10-20 14:26:12.781166 I | [install] Artifact not found in cache. Trying to fetch from upstream: https://github.com/etcd-io/etcd/releases/download

INFO[0000] [install] Downloading & installing etcd https://github.com/etcd-io/etcd/releases/download from 3.5.5 to /var/cache/etcdadm/etcd/v3.5.5

INFO[0000] [install] downloading etcd from https://github.com/etcd-io/etcd/releases/download/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz to /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz

INFO[0009] [install] extracting etcd archive /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz to /tmp/etcd641204404

INFO[0009] [install] verifying etcd 3.5.5 is installed in /opt/bin/

# 生成一个自签名的 CA 证书及私钥

INFO[0001] [certificates] creating PKI assets

INFO[0001] creating a self signed etcd CA certificate and key files

[certificates] Generated ca certificate and key.

> /etc/etcd/pki/ca.crt

> /etc/etcd/pki/ca.key

# 生成一个 server 证书及私钥

INFO[0001] creating a new server certificate and key files for etcd

[certificates] Generated server certificate and key.

[certificates] server serving cert is signed for DNS names [c7] and IPs [192.168.31.37 127.0.0.1]

# > /etc/etcd/pki/server.crt

# > /etc/etcd/pki/server.key

# 生成一个 peer 证书及私钥

INFO[0001] creating a new certificate and key files for etcd peering

[certificates] Generated peer certificate and key.

[certificates] peer serving cert is signed for DNS names [c7] and IPs [192.168.31.37]

# > /etc/etcd/pki/peer.crt

# > /etc/etcd/pki/peer.key

# 生成一个用于 etcdctl 的 client 证书及私钥

INFO[0001] creating a new client certificate for the etcdctl

[certificates] Generated etcdctl-etcd-client certificate and key.

# > /etc/etcd/pki/etcdctl-etcd-client.crt

# > /etc/etcd/pki/etcdctl-etcd-client.key

# 生成一个用于 k8s apiserver 调用 etcd 时的 client 证书及私钥

INFO[0002] creating a new client certificate for the apiserver calling etcd

[certificates] Generated apiserver-etcd-client certificate and key.

[certificates] valid certificates and keys now exist in "/etc/etcd/pki"

# > /etc/etcd/pki/apiserver-etcd-client.crt

# > /etc/etcd/pki/apiserver-etcd-client.key

# 检查本地 etcd 端点是否健康

INFO[0003] [health] Checking local etcd endpoint health

INFO[0003] [health] Local etcd endpoint is healthy

# 复制 CA cert/key 到其它 etcd 节点，并在其它 etcd 节点运行 etcdadm join 命令， 将其它 etcd 节点加入集群

INFO[0003] To add another member to the cluster, copy the CA cert/key to its certificate dir and run:

INFO[0003]  etcdadm join https://192.168.31.37:2379

3、向其它节点分发 CA 根证书及私钥

ssh root@192.168.31.38 "mkdir /etc/etcd/pki/"

scp -r /etc/etcd/pki/{ca.crt,ca.key} 192.168.31.38:/etc/etcd/pki/

ssh root@192.168.31.39 "mkdir /etc/etcd/pki/"

scp -r /etc/etcd/pki/{ca.crt,ca.key} 192.168.31.39:/etc/etcd/pki/

添加 etcd 节点

若当前主机无法下载，可提前将 etcd 二进制程序包存放在如下路径: /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz

1、添加节点 192.168.31.38

etcdadm join https://192.168.31.38:2379 \

--version "3.5.5" \

--init-system "systemd" \

--install-dir "/opt/bin/"   \

--certs-dir  "/etc/etcd/pki" \

--data-dir "/var/lib/etcd"  \

--release-url "https://github.com/etcd-io/etcd/releases/download"

2、添加节点 192.168.31.39

etcdadm join https://192.168.31.38:2379 \

--version "3.5.5" \

--init-system "systemd" \

--install-dir "/opt/bin/"   \

--certs-dir  "/etc/etcd/pki" \

--data-dir "/var/lib/etcd"  \

--release-url "https://github.com/etcd-io/etcd/releases/download"

Etcd Server

1、用于 Etcd Server 的环境变量配置 /etc/etcd/etcd.env

ETCD_NAME=c7

# Initial cluster configuration

ETCD_INITIAL_CLUSTER=c7=https://192.168.31.37:2380

ETCD_INITIAL_CLUSTER_TOKEN=dee8095f

ETCD_INITIAL_CLUSTER_STATE=new

# Peer configuration

ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.31.37:2380

ETCD_LISTEN_PEER_URLS=https://192.168.31.37:2380

ETCD_CLIENT_CERT_AUTH=true

ETCD_PEER_CERT_FILE=/etc/etcd/pki/peer.crt

ETCD_PEER_KEY_FILE=/etc/etcd/pki/peer.key

ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

# Client/server configuration

ETCD_ADVERTISE_CLIENT_URLS=https://192.168.31.37:2379

ETCD_LISTEN_CLIENT_URLS=https://192.168.31.37:2379,https://127.0.0.1:2379

ETCD_PEER_CLIENT_CERT_AUTH=true

ETCD_CERT_FILE=/etc/etcd/pki/server.crt

ETCD_KEY_FILE=/etc/etcd/pki/server.key

ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

# Other

ETCD_DATA_DIR=/var/lib/etcd

ETCD_STRICT_RECONFIG_CHECK=true

GOMAXPROCS=8

# Logging configuration

# Profiling/metrics

2、Etcd Server 启动脚本

# cat /etc/systemd/system/etcd.service

[Unit]

Description=etcd

Documentation=https://github.com/coreos/etcd

Conflicts=etcd-member.service

Conflicts=etcd2.service

[Service]

EnvironmentFile=/etc/etcd/etcd.env

ExecStart=/opt/bin/etcd

Type=notify

TimeoutStartSec=0

Restart=on-failure

RestartSec=5s

LimitNOFILE=65536

Nice=-10

IOSchedulingClass=best-effort

IOSchedulingPriority=2

MemoryLow=200M

[Install]

WantedBy=multi-user.target

etcdctl.sh

1、用于 etcdctl 的环境变量配置 /etc/etcd/etcdctl.env

export ETCDCTL_API=3

export ETCDCTL_CACERT=/etc/etcd/pki/ca.crt

export ETCDCTL_CERT=/etc/etcd/pki/etcdctl-etcd-client.crt

export ETCDCTL_KEY=/etc/etcd/pki/etcdctl-etcd-client.key

export ETCDCTL_DIAL_TIMEOUT=3s

2、脚本 etcdctl.sh 是对 etcdctl 命令的简单包装，其用法与 etcdctl 一致

cat /opt/bin/etcdctl.sh

#!/usr/bin/env sh

if ! [ -r "/etc/etcd/etcdctl.env" ]; then

        echo "Unable to read the etcdctl environment file '/etc/etcd/etcdctl.env'. The file must exist, and this wrapper must be run as root."

        exit 1

fi

. "/etc/etcd/etcdctl.env"   # 相当于 source 该环境变量配置文件

"/opt/bin/etcdctl" "$@"     # $@ 表示脚本 etcdctl.sh 的命令行参数

管理命令

# 查看命令行 init 或 join 的帮助信息

etcdadm init|join --help

# 从 etcd 集群移除当前节点

etcdadm reset

# 查看集群节点成员

/opt/bin/etcdctl.sh member list

# > 19fc11a542653f62, started, c9, https://192.168.31.39:2380, https://192.168.31.39:2379, false

# > 9a246c6786d36273, started, c7, https://192.168.31.37:2380, https://192.168.31.37:2379, false

# > a509d3d8e8aa4911, started, c8, https://192.168.31.38:2380, https://192.168.31.38:2379, false

# 查看当前节点是否正常

/opt/bin/etcdctl.sh endpoint health

# 127.0.0.1:2379 is healthy: successfully committed proposal: took = 17.112587ms

# 查看当前节点状态

/opt/bin/etcdctl.sh endpoint status

# > 127.0.0.1:2379, 9a246c6786d36273, 3.5.5, 20 kB, true, false, 3, 10, 10,

由于笔者时间、视野、认知有限，本文难免出现错误、疏漏等问题，期待各位读者朋友、业界大佬指正交流, 共同进步 !!