kubernetes部署 etcd 集群

本文档介绍部署一个三节点高可用 etcd 集群的步骤：

etcd 集群各节点的名称和 IP 如下：

kube-node0：192.168.111.10
kube-node1：192.168.111.11
kube-node2：192.168.111.12

创建 etcd 证书和私钥，所有证书和私钥的操作在/etc/kubernetes/ca/目录。

这里说下题外话：证书和私钥跟程序本身没有什么特定的关系，只是网络传输时的认证和授权而已，就针对etcd服务可以创建一对证书和私钥，也可以为etcd服务器、etcd客户端、etcd集群三个方面创建三对证书和私钥。甚至整个kubernetes集群也可以只用一对证书和私钥，只要配置把所有用到的ip及域名全部添加到hosts。

创建证书签名请求：

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.111.10",
    "192.168.111.11",
    "192.168.111.12"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "ChongQing",
      "L": "ChongQing",
      "O": "k8s",
      "OU": "yunwei"
    }
  ]
}
EOF

hosts 字段指定授权使用该证书的 etcd 节点 IP 或域名列表，这里将 etcd 集群的三个节点 IP 都列在其中；
生成证书和私钥：

# cfssl gencert -ca=/etc/kubernetes/ca/ca.pem \
-ca-key=/etc/kubernetes/ca/ca-key.pem \
-config=/etc/kubernetes/ca/ca-config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd
# ls etcd*
分发生成的证书和私钥到各 etcd 节点的/etc/kubernetes/ca/ 目录下。

# scp /etc/kubernetes/ca/etcd* 192.168.111.11:/etc/kubernetes/ca/
# scp /etc/kubernetes/ca/etcd* 192.168.111.12:/etc/kubernetes/ca/

添加etcd的服务文件，注意修改名称和IP地址，除了--initial-cluster的不变以外，其他的IP地址都是当前主机IP地址。

cat > /lib/systemd/system/etcd.service1 <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --name=kube-node1 \
  --data-dir=/var/lib/etcd \
  --listen-client-urls=https://192.168.111.11:2379,http://127.0.0.1:2379 \
  --client-cert-auth=true \
  --trusted-ca-file=/etc/kubernetes/ca/ca.pem \
  --cert-file=/etc/kubernetes/ca/etcd.pem \
  --key-file=/etc/kubernetes/ca/etcd-key.pem\
  --listen-peer-urls=https://192.168.111.11:2380 \
  --peer-client-cert-auth=true \
  --peer-trusted-ca-file=/etc/kubernetes/ca/ca.pem \
  --peer-cert-file=/etc/kubernetes/ca/etcd.pem \
  --peer-key-file=/etc/kubernetes/ca/etcd-key.pem \
  --initial-advertise-peer-urls=https://192.168.111.11:2380 \
  --advertise-client-urls=https://192.168.111.11:2379 \
  --initial-cluster-token=kubernetes-etcd \
  --initial-cluster=kube-node0=https://192.168.111.10:2380,kube-node1=https://192.168.111.11:2380,kube-node2=https://192.168.111.12:2380 \
  --initial-cluster-state=new
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

对于配置文件的内容，如果你不知道是什么意思，可以使用 etcd --help查看，如果看不懂英文，请使用google翻译https://translate.google.cn，不需要FQ就能访问，比其他翻译好的地方就是能识别出那些是参数，那些是描述。有道翻译也还不错，千万不要用百度翻译，翻译的更看不懂。

启动etcd，先创建工作目录，注意集群少于两台，etcd不报错，但是状态是灰色的。

# mkdir -p /var/lib/etcd/

# for SERVICES in etcd;do systemctl enable $SERVICES;systemctl start $SERVICS;systemctl status $SERVICES;done

如果报错，journalctl -xe，journalctl -u etcd 来定位问题

# etcdctl --version #查看etcd的版本及api的版本

# ETCDCTL_API=3 etcdctl version #查看etcd的版本及api的版本，使用3.0的api，命令不同了。

为什么本地能够不加认证授权就能执行，是因为我们添加有--listen-client-urls=https://192.168.111.11:2379,http://127.0.0.1:2379

# echo 'export ETCDCTL_API=3' >>/etc/profile #永久使用3.0的api，了解，不建议添加
# source /etc/profile

# ETCDCTL_API=3 etcdctl \
--endpoints=https://192.168.111.10:2379,https://192.168.111.11:2379,https://192.168.111.12:2379 \
--cacert=/etc/kubernetes/ca/ca.pem \
--cert=/etc/kubernetes/ca/etcd.pem \
--key=/etc/kubernetes/ca/etcd-key.pem \
endpoint health

https://192.168.111.10:2379 is healthy: successfully committed proposal: took = 1.718331ms
https://192.168.111.12:2379 is healthy: successfully committed proposal: took = 2.897364ms
https://192.168.111.11:2379 is healthy: successfully committed proposal: took = 7.089323ms

# ETCDCTL_API=3 etcdctl \
--endpoints=https://192.168.111.10:2379,https://192.168.111.11:2379,https://192.168.111.12:2379 \
--cacert=/etc/kubernetes/ca/ca.pem \
--cert=/etc/kubernetes/ca/etcd.pem \
--key=/etc/kubernetes/ca/etcd-key.pem \
member list

b9dfbfa5702cc550, started, kube-node2, https://192.168.111.12:2380, https://192.168.111.12:2379
e18dce88c431fa3e, started, kube-node0, https://192.168.111.10:2380, https://192.168.111.10:2379
f7e65517526c5972, started, kube-node1, https://192.168.111.11:2380, https://192.168.111.11:2379

使用2.0的api的参数有很多不一样，可以使用etcdctl --help查看

# etcdctl \
--endpoints=https://192.168.111.10:2379,https://192.168.111.11:2379,https://192.168.111.12:2379 \
--ca-file=/etc/kubernetes/ca/ca.pem \
--cert-file=/etc/kubernetes/ca/etcd.pem \
--key-file=/etc/kubernetes/ca/etcd-key.pem \
member list

b9dfbfa5702cc550: name=kube-node2 peerURLs=https://192.168.111.12:2380 clientURLs=https://192.168.111.12:2379 isLeader=false
e18dce88c431fa3e: name=kube-node0 peerURLs=https://192.168.111.10:2380 clientURLs=https://192.168.111.10:2379 isLeader=true
f7e65517526c5972: name=kube-node1 peerURLs=https://192.168.111.11:2380 clientURLs=https://192.168.111.11:2379 isLeader=false

设置一个字段，来验证数据。
# ETCDCTL_API=3 etcdctl \
--endpoints=https://192.168.111.10:2379,https://192.168.111.11:2379,https://192.168.111.12:2379 \
--cacert=/etc/kubernetes/ca/ca.pem \
--cert=/etc/kubernetes/ca/etcd.pem \
--key=/etc/kubernetes/ca/etcd-key.pem \
put name xu

读取数据，可以在每个机器上执行一遍，看有没有数据

# ETCDCTL_API=3 etcdctl \
--endpoints=https://192.168.111.10:2379,https://192.168.111.11:2379,https://192.168.111.12:2379 \
--cacert=/etc/kubernetes/ca/ca.pem \
--cert=/etc/kubernetes/ca/etcd.pem \
--key=/etc/kubernetes/ca/etcd-key.pem \
get name

也可以每台机器都分别读取数据，你会发现每台都可以写，每台都可以读。

# ETCDCTL_API=3 etcdctl \
--endpoints=https://192.168.111.12:2379\
--cacert=/etc/kubernetes/ca/ca.pem \
--cert=/etc/kubernetes/ca/etcd.pem \
--key=/etc/kubernetes/ca/etcd-key.pem \
put c cn

# ETCDCTL_API=3 etcdctl \
--endpoints=https://192.168.111.12:2379\
--cacert=/etc/kubernetes/ca/ca.pem \
--cert=/etc/kubernetes/ca/etcd.pem \
--key=/etc/kubernetes/ca/etcd-key.pem \
get c