前言
环境准备
安装
静态集群
- 配置
- 启动测试
  - 查看集群状态
生成TLS证书
- etcd证书创建
静态TLS集群
- etcd 配置
- 启动测试
  - 检查TLS集群状态
ETCD 动态集群基于DNS的SRV解析自动发现
ETCD TLS动态集群基于DNS的SRV解析自动发现
报错解决
- 1. 证书报错 bad certificate
  - 解决
- 2. DNS 的 SRV 解析报错 cannot find local etcd member "etcd1" in SRV records
  - http不带证书解析如下
    - bind 的解析
    - dnsmasq 的解析
  - https带证书解析如下
    - bind 的解析
    - dnsmasq 的解析

前言

Etcd 是 CoreOS 基于 Raft 开发的分布式 key-value 存储，可用于服务发现、共享配置以及一致性保障（如数据库选主、分布式锁等）

本次环境，是用于k8s集群，由于在二进制部署 k8s 中，由于 Etcd 集群导致各种各样的问题，特意抽出时间来研究 Etcd 集群。

Etcd 集群配置分为三种:

静态发现
Etcd 动态发现
DNS 动态发现通过DNS的SRV解析动态发现集群

本次主要基于静态发现和 DNS动态发现两种，并结合自签的TLS证书来创建集群。

环境准备

此环境实际用于 k8s 中的ETCD集群使用，用于本次文档

主机名	角色	IP	系统版本	内核版本
node01.k8s.com	node01	192.168.1.91	CentOS 7.7	5.1.4-1.el7.elrepo.x86_64
node02.k8s.com	node02	192.168.1.92	CentOS 7.7	5.1.4-1.el7.elrepo.x86_64
node03.k8s.com	node03	192.168.1.93	CentOS 7.7	5.1.4-1.el7.elrepo.x86_64

安装

在三台机器上均执行

[root@node01 ~]# yum install etcd -y

[root@node01 ~]# rpm -qa etcd

etcd-3.3.11-2.el7.centos.x86_64

创建Etcd所需目录，在三台机器上均执行

mkdir /data/k8s/etcd/{data,wal} -p

mkdir -p /etc/kubernetes/cert

chown -R etcd.etcd /data/k8s/etcd

静态集群

配置

node01 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="http://192.168.1.91:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.1.91:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd1"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.91:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.91:2379"

ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

node02 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="http://192.168.1.92:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.1.92:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd2"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.92:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.92:2379"

ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

node03 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="http://192.168.1.93:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.1.93:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd3"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.93:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.93:2379"

ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

启动测试

[root@node01 etcd]# systemctl start etcd

[root@node01 etcd]# systemctl status etcd

● etcd.service - Etcd Server

   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)

   Active: active (running) since Thu 2019-11-07 09:28:54 CST; 5s ago

 Main PID: 1546 (etcd)

    Tasks: 8

   Memory: 41.3M

   CGroup: /system.slice/etcd.service

           └─1546 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=http://192.168.1.91:2379

Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 [term: 1] received a MsgVote message with higher term from 9c64fba479c5e94 [term: 2]

Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 became follower at term 2

Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 [logterm: 1, index: 3, vote: 0] cast MsgVote for 9c64fba479c5e94 [logterm: 1, index: 3] at term 2

Nov 07 09:28:54 node01.k8s.com etcd[1546]: raft.node: 3b8b38de05e2c497 elected leader 9c64fba479c5e94 at term 2

Nov 07 09:28:54 node01.k8s.com etcd[1546]: published {Name:etcd1 ClientURLs:[http://192.168.1.91:2379]} to cluster 19456f0bfd57284e

Nov 07 09:28:54 node01.k8s.com etcd[1546]: ready to serve client requests

Nov 07 09:28:54 node01.k8s.com etcd[1546]: serving insecure client requests on 192.168.1.91:2379, this is strongly discouraged!

Nov 07 09:28:54 node01.k8s.com systemd[1]: Started Etcd Server.

Nov 07 09:28:54 node01.k8s.com etcd[1546]: set the initial cluster version to 3.3

Nov 07 09:28:54 node01.k8s.com etcd[1546]: enabled capabilities for version 3.3

查看 /var/log/message 日志中，会有日下体现：

Nov  7 09:28:53 node02 etcd: added member 9c64fba479c5e94 [http://192.168.1.92:2380] to cluster 19456f0bfd57284e

Nov  7 09:28:53 node02 etcd: added member 3b8b38de05e2c497 [http://192.168.1.91:2380] to cluster 19456f0bfd57284e

Nov  7 09:28:53 node02 etcd: added member 76ea8679db7365b3 [http://192.168.1.93:2380] to cluster 19456f0bfd57284e

查看集群状态

[root@node01 etcd]# ETCDCTL_API=3 etcdctl --endpoints=http://192.168.1.91:2379,http://192.168.1.92:2379,http://192.168.1.93:2379 endpoint health

http://192.168.1.92:2379 is healthy: successfully committed proposal: took = 1.103545ms

http://192.168.1.93:2379 is healthy: successfully committed proposal: took = 2.122478ms

http://192.168.1.91:2379 is healthy: successfully committed proposal: took = 2.690215ms

[root@node01 etcd]# etcdctl --endpoints=http://192.168.1.91:2379,http://192.168.1.92:2379,http://192.168.1.93:2379 cluster-health

member 9c64fba479c5e94 is healthy: got healthy result from http://192.168.1.92:2379

member 3b8b38de05e2c497 is healthy: got healthy result from http://192.168.1.91:2379

member 76ea8679db7365b3 is healthy: got healthy result from http://192.168.1.93:2379

cluster is healthy

生成TLS证书

使用自签证书

CA(Certificate Authority)是自签名的根证书，用来签名后续创建的其他证书。本文章使用CloudFlare的PKI工具cfssl创建所有证书。

etcd证书创建

整个证书的创建过程均在 node01 上操作;

安装cfssl工具集

mkdir -p /opt/k8s/cert && cd /opt/k8s

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

mv cfssl_linux-amd64 /opt/k8s/bin/cfssl

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

mv cfssljson_linux-amd64 /opt/k8s/bin/cfssljson

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

mv cfssl-certinfo_linux-amd64 /opt/k8s/bin/cfssl-certinfo

chmod +x /opt/k8s/bin/*

echo 'export PATH=/opt/k8s/bin:$PATH' >> ~/.bash_profile

source ~/.bash_profile

生成证书

创建根证书 (CA)

CA证书是集群所有节点共享的，只需要创建一个CA证书，后续创建的所有证书都是由它签名

创建配置文件

CA配置文件用于配置根证书的使用场景(profile)和具体参数

(usage、过期时间、服务端认证、客户端认证、加密等)

cd /opt/k8s/work

cat > ca-config.json <<EOF

{

  "signing": {

    "default": {

      "expiry": "87600h"

    },

    "profiles": {

      "kubernetes": {

        "usages": [

            "signing",

            "key encipherment",

            "server auth",

            "client auth"

        ],

        "expiry": "87600h"

      }

    }

  }

}

EOF

######################

signing 表示该证书可用于签名其它证书，生成的ca.pem证书找中CA=TRUE

server auth 表示client可以用该证书对server提供的证书进行验证

client auth 表示server可以用该证书对client提供的证书进行验证

创建证书签名请求文件

cd /opt/k8s/work

cat > ca-csr.json <<EOF

{

  "CN": "kubernetes",

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "4Paradigm"

    }

  ],

  "ca": {

    "expiry": "876000h"

 }

}

EOF

#######################

CN CommonName,kube-apiserver从证书中提取该字段作为请求的用户名(User Name)，浏览器使用该字段验证网站是否合法

O Organization,kube-apiserver 从证书中提取该字段作为请求用户和所属组(Group)

kube-apiserver将提取的User、Group作为RBAC授权的用户和标识

生成CA证书和私钥

cd /opt/k8s/work

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

ls ca*

创建etcd证书和私钥

cd /opt/k8s/work

cat > etcd-csr.json <<EOF

{

  "CN": "etcd",

  "hosts": [

    "127.0.0.1",

    "192.168.1.91",

    "192.168.1.92",

    "192.168.1.93",

    "k8s.com",

    "etcd1.k8s.com",

    "etcd2.k8s.com",

    "etcd3.k8s.com"

  ],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "4Paradigm"

    }

  ]

}

EOF

#host字段指定授权使用该证书的etcd节点IP或域名列表，需要将etcd集群的3个节点都添加其中

生成证书和私钥

cd /opt/k8s/work

cfssl gencert -ca=/opt/k8s/work/ca.pem \

    -ca-key=/opt/k8s/work/ca-key.pem \

    -config=/opt/k8s/work/ca-config.json \

    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

ls etcd*pem -l

-rw------- 1 root root 1675 Nov  7 09:52 etcd-key.pem

-rw-r--r-- 1 root root 1444 Nov  7 09:52 etcd.pem

etcd 使用的TLS证书创建完成

分发证书到各节点上

要做所有节点上创建对应的目录

mkdir /data/k8s/etcd/{data,wal} -p

mkdir -p /etc/kubernetes/cert

chown -R etcd.etcd /data/k8s/etcd

分发证书

cd /opt/k8s/work

scp ca*.pem ca-config.json 192.168.1.91:/etc/kubernetes/cert

scp ca*.pem ca-config.json 192.168.1.92:/etc/kubernetes/cert

scp ca*.pem ca-config.json 192.168.1.93:/etc/kubernetes/cert

scp etcd*pem 192.168.1.91:/etc/etcd/cert/

scp etcd*pem 192.168.1.92:/etc/etcd/cert/

scp etcd*pem 192.168.1.93:/etc/etcd/cert/

在所有节点上执行:

chown -R etcd.etcd /etc/etcd/cert

静态TLS集群

etcd 配置

node01 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="https://192.168.1.91:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.1.91:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd1"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.91:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.91:2379"

ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"

ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_AUTO_TLS="true"

ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_PEER_AUTO_TLS="true"

node02 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="https://192.168.1.92:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.1.92:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd2"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.92:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.92:2379"

ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"

ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_AUTO_TLS="true"

ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_PEER_AUTO_TLS="true"

node03 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="https://192.168.1.93:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.1.93:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd3"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.93:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.93:2379"

ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"

ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_AUTO_TLS="true"

ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_PEER_AUTO_TLS="true"

启动测试

[root@node01 work]# systemctl start etcd

[root@node01 work]# systemctl status etcd

● etcd.service - Etcd Server

   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)

   Active: active (running) since Thu 2019-11-07 10:15:58 CST; 5s ago

 Main PID: 2078 (etcd)

    Tasks: 8

   Memory: 28.9M

   CGroup: /system.slice/etcd.service

           └─2078 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=https://192.168.1.91:2379

Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe [term: 1] received a MsgVote message with higher term from af05139f75a68867 [term: 2]

Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe became follower at term 2

Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe [logterm: 1, index: 3, vote: 0] cast MsgVote for af05139f75a68867 [logterm: 1, index: 3] at term 2

Nov 07 10:15:58 node01.k8s.com etcd[2078]: raft.node: 2a40d8ba966d12fe elected leader af05139f75a68867 at term 2

Nov 07 10:15:58 node01.k8s.com etcd[2078]: published {Name:etcd1 ClientURLs:[https://192.168.1.91:2379]} to cluster f3e9c54e1aafb3c1

Nov 07 10:15:58 node01.k8s.com etcd[2078]: ready to serve client requests

Nov 07 10:15:58 node01.k8s.com etcd[2078]: serving client requests on 192.168.1.91:2379

Nov 07 10:15:58 node01.k8s.com systemd[1]: Started Etcd Server.

Nov 07 10:15:58 node01.k8s.com etcd[2078]: set the initial cluster version to 3.3

Nov 07 10:15:58 node01.k8s.com etcd[2078]: enabled capabilities for version 3.3

查看 /var/log/message 日志中，会有日下体现：

Nov  7 10:15:57 node01 etcd: added member 2a40d8ba966d12fe [https://192.168.1.91:2380] to cluster f3e9c54e1aafb3c1

Nov  7 10:15:57 node01 etcd: added member af05139f75a68867 [https://192.168.1.92:2380] to cluster f3e9c54e1aafb3c1

Nov  7 10:15:57 node01 etcd: added member c3bab7c20fba3f60 [https://192.168.1.93:2380] to cluster f3e9c54e1aafb3c1

检查TLS集群状态

ETCDCTL_API=3 etcdctl \

    --endpoints=https://etcd1.k8s.com:2379,https://etcd2.k8s.com:2379,https://etcd3.k8s.com:2379 \

    --cacert=/etc/kubernetes/cert/ca.pem \

    --cert=/etc/etcd/cert/etcd.pem \

    --key=/etc/etcd/cert/etcd-key.pem endpoint health

# 输出

https://192.168.1.92:2379 is healthy: successfully committed proposal: took = 1.317022ms

https://192.168.1.91:2379 is healthy: successfully committed proposal: took = 1.59958ms

https://192.168.1.93:2379 is healthy: successfully committed proposal: took = 1.453049ms

etcdctl \

--endpoint=https://etcd1.k8s.com:2379 \

--ca-file=/etc/kubernetes/cert/ca.pem \

--cert-file=/etc/etcd/cert/etcd.pem \

--key-file=/etc/etcd/cert/etcd-key.pem cluster-health

# 输出

member 40a8f19a5db99534 is healthy: got healthy result from https://etcd2.k8s.com:2379

member 9888555207dbf0e0 is healthy: got healthy result from https://etcd3.k8s.com:2379

member a0d541999e9eb3b3 is healthy: got healthy result from https://etcd1.k8s.com:2379

cluster is healthy

ETCD 动态集群基于DNS的SRV解析自动发现

需要局域网内部有DNS服务器

添加SRV解析

目前常用的内部DNS服务有两种，bind、dnsmasq

在下面都会列出具体的配置，但只需要配置其中之一即可；

方法一: 使用`bind`配置SRV解析

如果内部没有bind服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11806962.html

使用域名为 : k8s.com，在bind的zone文件中添加如下解析:

etcd1   IN      A       192.168.1.91

etcd2   IN      A       192.168.1.92

etcd3   IN      A       192.168.1.93

_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd1

_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd2

_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd3

_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd1

_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd2

_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd3

修改之后重新加载配置文件:

[root@jenkins named]# named-checkzone k8s.com k8s.com.zone

zone k8s.com/IN: loaded serial 0

OK

[root@jenkins named]# rndc reload

server reload successful

方法二: 使用`dnsmasq`配置SRV解析

如果内部没有dnsmasq服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11809066.html

使用域名为 : k8s.com，具体修改如下:

在/etc/dnsmasq_hosts新增下面内容

192.168.1.91 etcd1 etcd1.k8s.com

192.168.1.92 etcd2 etcd2.k8s.com

192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析内容

srv-host=_etcd-server._tcp.k8s.com,etcd1.k8s.com,2380,0,100

srv-host=_etcd-server._tcp.k8s.com,etcd2.k8s.com,2380,0,100

srv-host=_etcd-server._tcp.k8s.com,etcd3.k8s.com,2380,0,100

srv-host=_etcd-client._tcp.k8s.com,etcd1.k8s.com,2379,0,100

srv-host=_etcd-client._tcp.k8s.com,etcd2.k8s.com,2379,0,100

srv-host=_etcd-client._tcp.k8s.com,etcd3.k8s.com,2379,0,100

修改之后重启服务 systemctl restart dnsmasq

验证SRV解析是否正常

查询SRV记录

[root@node01 ~]# dig @192.168.1.122 +noall +answer SRV _etcd-server._tcp.k8s.com

_etcd-server._tcp.k8s.com. 3600	IN	SRV	2380 0 100 etcd2.k8s.com.

_etcd-server._tcp.k8s.com. 3600	IN	SRV	2380 0 100 etcd1.k8s.com.

_etcd-server._tcp.k8s.com. 3600	IN	SRV	2380 0 100 etcd3.k8s.com.

查询域名解析结果

[root@node01 ~]# dig @192.168.1.122 +noall +answer etcd1.k8s.com etcd2.k8s.com etcd3.k8s.com

etcd1.k8s.com.		86400	IN	A	192.168.1.91

etcd2.k8s.com.		86400	IN	A	192.168.1.92

etcd3.k8s.com.		86400	IN	A	192.168.1.93

如上述显示，则表示SRV解析正常

配置ETCD

node01 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="http://192.168.1.91:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.1.91:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd1"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd1.k8s.com:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://etcd1.k8s.com:2379"

ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

node02 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="http://192.168.1.92:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.1.92:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd2"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd2.k8s.com:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://etcd2.k8s.com:2379"

ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

node03 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="http://192.168.1.93:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.1.93:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd3"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd3.k8s.com:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://etcd3.k8s.com:2379"

ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

启动并测试

启动

[root@node01 etcd]# systemctl start etcd

[root@node01 etcd]# systemctl status etcd

● etcd.service - Etcd Server

   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)

   Active: active (running) since Thu 2019-11-07 11:25:29 CST; 4s ago

 Main PID: 14203 (etcd)

    Tasks: 8

   Memory: 16.9M

   CGroup: /system.slice/etcd.service

           └─14203 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=http://192.168.1.91:2379

Nov 07 11:25:29 node01.k8s.com etcd[14203]: d79e9ae86b2a1de1 [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections

Nov 07 11:25:29 node01.k8s.com etcd[14203]: d79e9ae86b2a1de1 became leader at term 2

Nov 07 11:25:29 node01.k8s.com etcd[14203]: raft.node: d79e9ae86b2a1de1 elected leader d79e9ae86b2a1de1 at term 2

Nov 07 11:25:29 node01.k8s.com etcd[14203]: published {Name:etcd1 ClientURLs:[http://etcd1.k8s.com:2379 http://etcd1.k8s.com:4001]} to cluster 42cecf80e3791d6c

Nov 07 11:25:29 node01.k8s.com etcd[14203]: ready to serve client requests

Nov 07 11:25:29 node01.k8s.com etcd[14203]: serving insecure client requests on 192.168.1.91:2379, this is strongly discouraged!

Nov 07 11:25:29 node01.k8s.com systemd[1]: Started Etcd Server.

Nov 07 11:25:29 node01.k8s.com etcd[14203]: setting up the initial cluster version to 3.3

Nov 07 11:25:29 node01.k8s.com etcd[14203]: set the initial cluster version to 3.3

Nov 07 11:25:29 node01.k8s.com etcd[14203]: enabled capabilities for version 3.3

日志 vim /var/log/messages 表现如下:

Nov  7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at 0=http://etcd3.k8s.com:2380

Nov  7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at 1=http://etcd2.k8s.com:2380

Nov  7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at etcd1=http://etcd1.k8s.com:2380

Nov  7 11:25:27 node01 etcd: resolving etcd1.k8s.com:2380 to 192.168.1.91:2380

Nov  7 11:25:27 node01 etcd: resolving etcd1.k8s.com:2380 to 192.168.1.91:2380

Nov  7 11:25:28 node01 etcd: name = etcd1

Nov  7 11:25:28 node01 etcd: data dir = /data/k8s/etcd/data

Nov  7 11:25:28 node01 etcd: member dir = /data/k8s/etcd/data/member

Nov  7 11:25:28 node01 etcd: dedicated WAL dir = /data/k8s/etcd/wal

Nov  7 11:25:28 node01 etcd: heartbeat = 100ms

Nov  7 11:25:28 node01 etcd: election = 1000ms

Nov  7 11:25:28 node01 etcd: snapshot count = 100000

Nov  7 11:25:28 node01 etcd: advertise client URLs = http://etcd1.k8s.com:2379,http://etcd1.k8s.com:4001

Nov  7 11:25:28 node01 etcd: initial advertise peer URLs = http://etcd1.k8s.com:2380

Nov  7 11:25:28 node01 etcd: initial cluster = 0=http://etcd3.k8s.com:2380,1=http://etcd2.k8s.com:2380,etcd1=http://etcd1.k8s.com:2380

测试：

[root@node01 etcd]# etcdctl --endpoints=http://192.168.1.91:2379 cluster-health

member 184beca37ca32d75 is healthy: got healthy result from http://etcd2.k8s.com:2379

member d79e9ae86b2a1de1 is healthy: got healthy result from http://etcd1.k8s.com:2379

member f7662e609b7e4013 is healthy: got healthy result from http://etcd3.k8s.com:2379

cluster is healthy

ETCD TLS动态集群基于DNS的SRV解析自动发现

需要局域网内部有DNS服务器

添加SRV解析

目前常用的内部DNS服务有两种，bind、dnsmasq

在下面都会列出具体的配置，但只需要配置其中之一即可；

方法一: 使用`bind`配置SRV解析

如果内部没有bind服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11806962.html

使用域名为 : k8s.com，在bind的zone文件中添加如下解析:

etcd1   IN      A       192.168.1.91

etcd2   IN      A       192.168.1.92

etcd3   IN      A       192.168.1.93

_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd1

_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd2

_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd3

_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd1

_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd2

_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd3

修改之后重新加载配置文件:

[root@jenkins named]# named-checkzone k8s.com k8s.com.zone

zone k8s.com/IN: loaded serial 0

OK

[root@jenkins named]# rndc reload

server reload successful

方法二: 使用`dnsmasq`配置SRV解析

如果内部没有dnsmasq服务，可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11809066.html

使用域名为 : k8s.com，具体修改如下:

在/etc/dnsmasq_hosts新增下面内容

192.168.1.91 etcd1 etcd1.k8s.com

192.168.1.92 etcd2 etcd2.k8s.com

192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析内容

srv-host=_etcd-server-ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100

srv-host=_etcd-server-ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100

srv-host=_etcd-server-ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100

srv-host=_etcd-client-ssl._tcp.k8s.com,etcd1.k8s.com,2379,0,100

srv-host=_etcd-client-ssl._tcp.k8s.com,etcd2.k8s.com,2379,0,100

srv-host=_etcd-client-ssl._tcp.k8s.com,etcd3.k8s.com,2379,0,100

修改之后重启服务 systemctl restart dnsmasq

验证SRV解析是否正常

查询SRV记录

[root@node01 etcd]# dig @192.168.1.122 +noall +answer SRV _etcd-server-ssl._tcp.k8s.com

_etcd-server-ssl._tcp.k8s.com. 3600 IN	SRV	2380 0 100 etcd3.k8s.com.

_etcd-server-ssl._tcp.k8s.com. 3600 IN	SRV	2380 0 100 etcd2.k8s.com.

_etcd-server-ssl._tcp.k8s.com. 3600 IN	SRV	2380 0 100 etcd1.k8s.com.

查询域名解析结果

[root@node01 ~]# dig @192.168.1.122 +noall +answer etcd1.k8s.com etcd2.k8s.com etcd3.k8s.com

etcd1.k8s.com.		86400	IN	A	192.168.1.91

etcd2.k8s.com.		86400	IN	A	192.168.1.92

etcd3.k8s.com.		86400	IN	A	192.168.1.93

ETCD 配置

node01 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="https://192.168.1.91:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.1.91:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd1"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd1.k8s.com:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://etcd1.k8s.com:2379,https://etcd1.k8s.com:4001"

ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"

ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_AUTO_TLS="true"

ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_PEER_AUTO_TLS="true"

node02 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="https://192.168.1.92:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.1.92:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd2"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd2.k8s.com:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://etcd2.k8s.com:2379"

ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"

ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_AUTO_TLS="true"

ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_PEER_AUTO_TLS="true"

node03 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"

ETCD_WAL_DIR="/data/k8s/etcd/wal"

ETCD_LISTEN_PEER_URLS="https://192.168.1.93:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.1.93:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

ETCD_NAME="etcd3"

ETCD_SNAPSHOT_COUNT="100000"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd3.k8s.com:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://etcd3.k8s.com:2379"

ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"

ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_AUTO_TLS="true"

ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

ETCD_PEER_AUTO_TLS="true"

启动测试

启动

[root@node03 etcd]# systemctl restart etcd

[root@node03 etcd]# systemctl status etcd

● etcd.service - Etcd Server

   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)

   Active: active (running) since Thu 2019-11-07 12:38:37 CST; 4s ago

 Main PID: 13460 (etcd)

    Tasks: 8

   Memory: 16.6M

   CGroup: /system.slice/etcd.service

           └─13460 /usr/bin/etcd --name=etcd3 --data-dir=/data/k8s/etcd/data --listen-client-urls=https://192.168.1.93:2379

Nov 07 12:38:36 node03.k8s.com etcd[13460]: established a TCP streaming connection with peer 40a8f19a5db99534 (stream Message writer)

Nov 07 12:38:36 node03.k8s.com etcd[13460]: established a TCP streaming connection with peer 40a8f19a5db99534 (stream MsgApp v2 writer)

Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 [term: 92] received a MsgVote message with higher term from a0d541999e9eb3b3 [term: 98]

Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 became follower at term 98

Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 [logterm: 92, index: 9, vote: 0] cast MsgVote for a0d541999e9eb3b3 [logterm: 92, index: 9] at term 98

Nov 07 12:38:37 node03.k8s.com etcd[13460]: raft.node: 9888555207dbf0e0 elected leader a0d541999e9eb3b3 at term 98

Nov 07 12:38:37 node03.k8s.com etcd[13460]: published {Name:etcd3 ClientURLs:[https://etcd3.k8s.com:2379]} to cluster f445a02ce3dc6a02

Nov 07 12:38:37 node03.k8s.com etcd[13460]: ready to serve client requests

Nov 07 12:38:37 node03.k8s.com etcd[13460]: serving client requests on 192.168.1.93:2379

Nov 07 12:38:37 node03.k8s.com systemd[1]: Started Etcd Server.

日志体现

Nov  7 12:38:36 node01 etcd: added member 40a8f19a5db99534 [https://etcd2.k8s.com:2380] to cluster f445a02ce3dc6a02

Nov  7 12:38:36 node01 etcd: starting peer 40a8f19a5db99534...

Nov  7 12:38:36 node01 etcd: started HTTP pipelining with peer 40a8f19a5db99534

Nov  7 12:38:36 node01 etcd: started streaming with peer 40a8f19a5db99534 (writer)

Nov  7 12:38:36 node01 etcd: started peer 40a8f19a5db99534

Nov  7 12:38:36 node01 etcd: added peer 40a8f19a5db99534

Nov  7 12:38:36 node01 etcd: added member 9888555207dbf0e0 [https://etcd3.k8s.com:2380] to cluster f445a02ce3dc6a02

Nov  7 12:38:36 node01 etcd: starting peer 9888555207dbf0e0...

Nov  7 12:38:36 node01 etcd: started HTTP pipelining with peer 9888555207dbf0e0

Nov  7 12:38:36 node01 etcd: started peer 9888555207dbf0e0

Nov  7 12:38:36 node01 etcd: added peer 9888555207dbf0e0

Nov  7 12:38:36 node01 etcd: added member a0d541999e9eb3b3 [https://etcd1.k8s.com:2380] to cluster f445a02ce3dc6a02

测试集群状态：

ETCDCTL_API=3 etcdctl --endpoints=https://etcd1.k8s.com:2379,https://etcd2.k8s.com:2379,https://etcd3.k8s.com:2379 \

    --cacert=/etc/kubernetes/cert/ca.pem \

    --cert=/etc/etcd/cert/etcd.pem \

    --key=/etc/etcd/cert/etcd-key.pem endpoint health

# 输出

https://etcd1.k8s.com:2379 is healthy: successfully committed proposal: took = 4.269468ms

https://etcd3.k8s.com:2379 is healthy: successfully committed proposal: took = 1.58797ms

https://etcd2.k8s.com:2379 is healthy: successfully committed proposal: took = 1.622151ms

etcdctl \

--endpoint=https://etcd1.k8s.com:2379 \

--ca-file=/etc/kubernetes/cert/ca.pem \

--cert-file=/etc/etcd/cert/etcd.pem \

--key-file=/etc/etcd/cert/etcd-key.pem cluster-health

# 输出

member 40a8f19a5db99534 is healthy: got healthy result from https://etcd2.k8s.com:2379

member 9888555207dbf0e0 is healthy: got healthy result from https://etcd3.k8s.com:2379

member a0d541999e9eb3b3 is healthy: got healthy result from https://etcd1.k8s.com:2379

cluster is healthy

报错解决

1. 证书报错 bad certificate

日志中报错:

Nov  7 12:37:03 node01 etcd: rejected connection from "192.168.1.93:46294" (error "remote error: tls: bad certificate", ServerName "k8s.com")

解决

报错的意思是在生成ETCD的TLS证书的时候，没有把对应的域名加进去

在创建ETCD的TLS证书请求的文件中加入对应的域名

cd /opt/k8s/work

cat > etcd-csr.json <<EOF

{

  "CN": "etcd",

  "hosts": [

    "127.0.0.1",

    "192.168.1.91",

    "192.168.1.92",

    "192.168.1.93",

    "k8s.com",          # 这里的域名查看是否正确

    "etcd1.k8s.com",

    "etcd2.k8s.com",

    "etcd3.k8s.com"

  ],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "4Paradigm"

    }

  ]

}

EOF

2. `DNS` 的 `SRV` 解析报错 cannot find local etcd member "etcd1" in SRV records

DNS 如果配置有问题，会有如下报错:

etcd: error setting up initial cluster: cannot find local etcd member "etcd1" in SRV records

这里是表示DNS在配置SRV解析的时候报错，请仔细查看解析配置:

SRV解析分为两种，一种是http不带证书的解析，一种是https带证书的解析，是有区别的，如果配置错误就会包上述错误

http不带证书解析如下

bind 的解析

编辑 /var/named/k8s.com.zone 文件

etcd1   IN      A       192.168.1.91

etcd2   IN      A       192.168.1.92

etcd3   IN      A       192.168.1.93

_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd1

_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd2

_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd3

_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd1

_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd2

_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd3

dnsmasq 的解析

在/etc/dnsmasq_hosts新增下面内容

192.168.1.91 etcd1 etcd1.k8s.com

192.168.1.92 etcd2 etcd2.k8s.com

192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析内容

srv-host=_etcd-server._tcp.k8s.com,etcd1.k8s.com,2380,0,100

srv-host=_etcd-server._tcp.k8s.com,etcd2.k8s.com,2380,0,100

srv-host=_etcd-server._tcp.k8s.com,etcd3.k8s.com,2380,0,100

srv-host=_etcd-client._tcp.k8s.com,etcd1.k8s.com,2380,0,100

srv-host=_etcd-client._tcp.k8s.com,etcd2.k8s.com,2380,0,100

srv-host=_etcd-client._tcp.k8s.com,etcd3.k8s.com,2380,0,100

https带证书解析如下

bind 的解析

编辑 /var/named/k8s.com.zone 文件

etcd1   IN      A       192.168.1.91

etcd2   IN      A       192.168.1.92

etcd3   IN      A       192.168.1.93

_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd1

_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd2

_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd3

_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd1

_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd2

_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd3

dnsmasq 的解析

在/etc/dnsmasq_hosts新增下面内容

192.168.1.91 etcd1 etcd1.k8s.com

192.168.1.92 etcd2 etcd2.k8s.com

192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析内容

srv-host=_etcd-server-ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100

srv-host=_etcd-server-ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100

srv-host=_etcd-server-ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100

srv-host=_etcd-client-ssl._tcp.k8s.com,etcd1.k8s.com,2379,0,100

srv-host=_etcd-client-ssl._tcp.k8s.com,etcd2.k8s.com,2379,0,100

srv-host=_etcd-client-ssl._tcp.k8s.com,etcd3.k8s.com,2379,0,100

CentOS 7 ETCD集群配置大全的更多相关文章

centos下etcd集群安装
先仔细了解学习etcd 官方: https://github.com/etcd-io/etcd https://www.cnblogs.com/softidea/p/6517959.html http ...
Kubernetes集群搭建之Etcd集群配置篇
介绍 etcd 是一个分布式一致性k-v存储系统,可用于服务注册发现与共享配置,具有以下优点. 简单 : 相比于晦涩难懂的paxos算法,etcd基于相对简单且易实现的raft算法实现一致性,并通过g ...
Docker 搭建 etcd 集群配置
#关闭selinux.防火墙 systemctl stop firewalld.service systemctl disable firewalld.service firewall-cmd --s ...
配置安全证书的Etcd集群
不知在哪篇技术文档中看到,kubernetes master和etcd分开部署模式,因为集群的状态都保存在etcd中,这样当kubernetes master挂掉后,通过API Server交互的Sc ...
Docker 搭建 etcd 集群
阅读目录: 主机安装集群搭建 API 操作 API 说明和 etcdctl 命令说明 etcd 是 CoreOS 团队发起的一个开源项目(Go 语言,其实很多这类项目都是 Go 语言实现的,只能说很 ...
k8s1.13.0二进制部署-ETCD集群（一）
Kubernetes集群中主要存在两种类型的节点:master.minion节点. Minion节点为运行 Docker容器的节点,负责和节点上运行的 Docker 进行交互,并且提供了代理功能.Ma ...
K8s二进制部署单节点 etcd集群，flannel网络配置 ——锥刺股
K8s 二进制部署单节点 master --锥刺股 k8s集群搭建: etcd集群 flannel网络插件搭建master组件搭建node组件 1.部署etcd集群 2.Flannel 网络 ...
使用docker配置etcd集群
docker配置etcd集群与直接部署etcd集群在配置上并没有什么太大差别. 我这里直接使用docker-compose来实现容器化的etcd部署环境如下: HostName IP etcd1 1 ...
kubeadm配置高可用etcd集群
操作系统为ubuntu18 kubernetes版本为v1.15.1 k8s默认在控制平面节点上的kubelet管理的静态pod中运行单个成员的etcd集群,但这不是高可用的方案. etcd高可用集群 ...

随机推荐

论RSA算法的重要性 -RSA 简介
地球上最重要的算法 (这个说法似乎有点夸张了,但是当你了解了RSA算法后,就觉得不夸张了.) 如果没有 RSA 算法,现在的网络世界毫无安全可言,也不可能有现在的网上交易.上一篇文章 ssh 协议为什 ...
tcp通信客户端本地日志查看
最近有一个需求,app要接sdk,只涉及到客户端tcp通信,不涉及服务端接口调用.本文主要从adb环境准备.android/ios本地日志查看实战,进行分析整理. 一.adb查看Android本地日志 ...
【Medium翻译】Java抽象类有什么用？
今天安利一个网站,其实很多朋友应该早就知道了,我之前ARTS打卡,英文文档的很多出处就来自于这个网站,叫「Medium」. 这个网站需要一定的技术去访问,但是为什么说他好呢,因为他号称全球最大的高 ...
python全局变量及局部变量
变量作用域全局变量(global):在函数外部定义,在整个全局范围都有效局部变量(local) 在函数内部定义,局部变量在局部范围内使用数字,字符串,元组,修改其变量值时需要加globle,列表 ...
USACO 07DEC 道路建设(Building Roads)
Farmer John had just acquired several new farms! He wants to connect the farms with roads so that he ...
Go netpoll I/O 多路复用构建原生网络模型之源码深度解析
导言 Go 基于 I/O multiplexing 和 goroutine 构建了一个简洁而高性能的原生网络模型(基于 Go 的I/O 多路复用 netpoll),提供了 goroutine-per- ...
CSPS模拟 95
T_T flag彻底倒了虽然打一开始就没觉得能实现过可是我好桑心T_T skyh那个没素质的还一直bb T_T
Redis持久化--Redis宕机或者出现意外删库导致数据丢失--解决方案
echo编辑整理,欢迎转载,转载请声明文章来源.欢迎添加echo微信(微信号:t2421499075)交流学习. 百战不败,依不自称常胜,百败不颓,依能奋力前行.--这才是真正的堪称强大!!! Red ...
1、第一个TensorFlow程序
import tensorflow as tf import os os.environ[' #去掉警告,将警告级别提升 a = tf.constant(2) #定义一个常量 b = tf.const ...
[转载]1.4 UiPath参数的介绍和使用
一.参数介绍用于将数据从一个项目传递到另一个项目.在全局意义上,它们类似于变量,因为它们动态地存储数据并传递给它.变量在活动之间传递数据,而参数在自动化之间传递数据.因此,它们使你能够一次又一次地重 ...

CentOS 7 ETCD集群配置大全

前言

环境准备

安装

静态集群

配置

node01 配置文件

node02 配置文件

node03 配置文件

启动测试

查看集群状态

生成TLS证书

etcd证书创建

安装cfssl工具集

生成证书

分发证书到各节点上

静态TLS集群

etcd 配置

node01 配置文件

node02 配置文件

node03 配置文件

启动测试

检查TLS集群状态

ETCD 动态集群基于DNS的SRV解析自动发现

添加SRV解析

方法一: 使用bind配置SRV解析

方法二: 使用dnsmasq配置SRV解析

验证SRV解析是否正常

配置ETCD

node01 配置文件

node02 配置文件

node03 配置文件

启动并测试

ETCD TLS动态集群基于DNS的SRV解析自动发现

添加SRV解析

方法一: 使用bind配置SRV解析

方法二: 使用dnsmasq配置SRV解析

验证SRV解析是否正常

ETCD 配置

node01 配置文件

node02 配置文件

node03 配置文件

启动测试

报错解决

1. 证书报错 bad certificate

解决

2. DNS 的 SRV 解析报错 cannot find local etcd member "etcd1" in SRV records

http不带证书解析如下

bind 的解析

dnsmasq 的解析

https带证书解析如下

bind 的解析

dnsmasq 的解析

CentOS 7 ETCD集群配置大全的更多相关文章

随机推荐

热门专题

方法一: 使用`bind`配置SRV解析

方法二: 使用`dnsmasq`配置SRV解析

方法一: 使用`bind`配置SRV解析

方法二: 使用`dnsmasq`配置SRV解析

2. `DNS` 的 `SRV` 解析报错 cannot find local etcd member "etcd1" in SRV records