【云原生 · Kubernetes】部署高可用 kube-controller-manager 集群

个人名片：
因为云计算成为了监控工程师‍
个人博客：念舒_C.ying
CSDN主页️：念舒_C.ying

部署高可用 kube-controller-manager 集群

该集群包含 3 个节点，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当leader 节点不可用时，阻塞的节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。

与 kube-apiserver 的安全端口通信;
在安全端口(https，10257) 输出 prometheus 格式的 metrics；
注意：如果没有特殊指明，本文档的所有操作均在 qist 节点上执行。

12.1 创建 kube-controller-manager 证书和私钥

创建证书签名请求：

cd /opt/k8s/work

cat > /opt/k8s/cfssl/k8s/k8s-controller-manager.json << EOF

{

"CN": "system:kube-controller-manager",

"hosts": [""],

"key": {

"algo": "rsa",

"size": 2048

},

"names": [

{

"C": "CN",

"ST": "$CERT_ST",

"L": "$CERT_L",

"O": "system:kube-controller-manager",

"OU": "Kubernetes-manual"

}

]

}

EOF

hosts 列表包含所有 kube-controller-manager 节点 IP；
CN 和 O 均为 system:kube-controller-manager ，kubernetes 内置的 ClusterRoleBindings
system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。
生成证书和私钥：

cd /opt/k8s/work

cfssl gencert \

-ca=/opt/k8s/cfssl/pki/k8s/k8s-ca.pem \

-ca-key=/opt/k8s/cfssl/pki/k8s/k8s-ca-key.pem \

-config=/opt/k8s/cfssl/ca-config.json \

-profile=kubernetes \

/opt/k8s/cfssl/k8s/k8s-controller-manager.json | \

cfssljson -bare /opt/k8s/cfssl/pki/k8s/k8s-controller-manager

root@Qist work# ll /opt/k8s/cfssl/pki/k8s/k8s-controller-manager*

-rw------- 1 root root 1679 Dec 3 2020 /opt/k8s/cfssl/pki/k8s/k8s-controllermanager-key.pem

-rw-r--r-- 1 root root 1127 Dec 3 2020 /opt/k8s/cfssl/pki/k8s/k8s-controllermanager.csr

-rw-r--r-- 1 root root 1505 Dec 3 2020 /opt/k8s/cfssl/pki/k8s/k8s-controllermanager.pem

将生成的证书和私钥分发到所有 master 节点：

cd /opt/k8s/work

scp -r /opt/k8s/cfssl/pki/k8s/k8s-controller-manager-*

root@192.168.2.175:/apps/k8s/ssl/k8s

scp -r /opt/k8s/cfssl/pki/k8s/k8s-controller-manager-*

root@192.168.2.176:/apps/k8s/ssl/k8s

scp -r /opt/k8s/cfssl/pki/k8s/k8s-controller-manager-*

root@192.168.2.177:/apps/k8s/ssl/k8s

12.2 创建和分发 kubeconfig 文件

kube-controller-manager 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的
CA 证书和 kube-controller-manager 证书等信息：

cd /opt/k8s/kubeconfig

kubectl config set-cluster kubernetes \

--certificate-authority=/opt/k8s/cfssl/pki/k8s/k8s-ca.pem \

--embed-certs=true \

--server=https://127.0.0.1:6443 \

--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \

--client-certificate=/opt/k8s/cfssl/pki/k8s/k8s-controller-manager.pem \

--embed-certs=true \

--client-key=/opt/k8s/cfssl/pki/k8s/k8s-controller-manager-key.pem \

--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context kubernetes \

--cluster=kubernetes \

--user=system:kube-controller-manager \

--kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context kubernetes --kubeconfig=kube-controllermanager.kubeconfig

kube-controller-manager 与 kube-apiserver 混布，故直接通过节点 IP 访问
kube-apiserver；分发 kubeconfig 到所有 master 节点：

cd /opt/k8s/kubeconfig

scp kube-controller-manager.kubeconfig root@192.168.2.175:/apps/k8s/config/

scp kube-controller-manager.kubeconfig root@192.168.2.176:/apps/k8s/config/

scp kube-controller-manager.kubeconfig root@192.168.2.177:/apps/k8s/config/

12.3 创建 kube-controller-manager 启动配置

cd /opt/k8s/work

cat >kube-controller-manager <<EOF

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \

--profiling \

--concurrent-service-syncs=2 \

--concurrent-deployment-syncs=10 \

--concurrent-gc-syncs=30 \

--leader-elect=true \

--bind-address=0.0.0.0 \

--service-cluster-ip-range=10.66.0.0/16 \

--cluster-cidr=10.80.0.0/12 \

--node-cidr-mask-size=24 \

--cluster-name=kubernetes \

--allocate-node-cidrs=true \

--kubeconfig=/apps/k8s/config/kube-controller-manager.kubeconfig \

--authentication-kubeconfig=/apps/k8s/config/kube-controller-manager.kubeconfig \

--authorization-kubeconfig=/apps/k8s/config/kube-controller-manager.kubeconfig \

--use-service-account-credentials=true \

--client-ca-file=/apps/k8s/ssl/k8s/k8s-ca.pem \

--requestheader-client-ca-file=/apps/k8s/ssl/k8s/k8s-ca.pem \

--requestheader-client-ca-file=/apps/k8s/ssl/k8s/k8s-ca.pem \

--requestheader-allowed-names=aggregator \

--requestheader-extra-headers-prefix=X-Remote-Extra- \

--requestheader-group-headers=X-Remote-Group \

--requestheader-username-headers=X-Remote-User \

--node-monitor-grace-period=30s \

--node-monitor-period=5s \

--pod-eviction-timeout=1m0s \

--node-startup-grace-period=20s \

--terminated-pod-gc-threshold=50 \

--alsologtostderr=true \

--cluster-signing-cert-file=/apps/k8s/ssl/k8s/k8s-ca.pem \

--cluster-signing-key-file=/apps/k8s/ssl/k8s/k8s-ca-key.pem \

--deployment-controller-sync-period=10s \

--experimental-cluster-signing-duration=876000h0m0s \

--root-ca-file=/apps/k8s/ssl/k8s/k8s-ca.pem \

--service-account-private-key-file=/apps/k8s/ssl/k8s/k8s-ca-key.pem \

--enable-garbage-collector=true \

--controllers=*,bootstrapsigner,tokencleaner \

--horizontal-pod-autoscaler-sync-period=10s \

--tls-cert-file=/apps/k8s/ssl/k8s/k8s-controller-manager.pem \

--tls-private-key-file=/apps/k8s/ssl/k8s/k8s-controller-manager-key.pem \

--kube-api-qps=100 \

--kube-api-burst=100 \

--tls-ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,

TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDH

E_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES

_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 \

--log-dir=/apps/k8s/log \

--v=2"

EOF

port=0 ：关闭监听非安全端口（http），同时 --address 参数无效， --bind-address 参数有效；
secure-port=10257 端口的 https /metrics 请求；
kubeconfig ：指定 kubeconfig 文件路径，kube-controller-manager 使用它连接和验证 kubeapiserver；
authentication-kubeconfig 和 --authorization-kubeconfig ：kube-controller-manager 使用它连
接 apiserver，对 client 的请求进行认证和授权。 kube-controller-manager 不再使用 --tls-ca-file
对请求 https metrics 的 Client 证书进行校验。如果没有配置这两个 kubeconfig 参数，则 client 连接
kube-controller-manager https 端口的请求会被拒绝(提示权限不足)。
cluster-signing-*-file ：签名 TLS Bootstrap 创建的证书；

分发 kube-controller-manager 配置文件到所有 master 节点：

cd /opt/k8s/work

scp kube-controller-manager root@192.168.2.175:/apps/k8s/conf/

scp kube-controller-manager root@192.168.2.176:/apps/k8s/conf/

scp kube-controller-manager root@192.168.2.177:/apps/k8s/conf/

12.4 创建 kube-controller-manager systemd unit 文件

cd /opt/k8s/work

cat > kube-controller-manager.service <<EOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/kubernetes/kubernetes

[Service]

LimitNOFILE=655350

LimitNPROC=655350

LimitCORE=infinity

LimitMEMLOCK=infinity

EnvironmentFile=-/apps/k8s/conf/kube-controller-manager

ExecStart=/apps/k8s/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-user.target

EOF

12.5 为各节点创建和分发 kube-controller-mananger systemd unit 文件

分发到所有 master 节点：

cd /opt/k8s/work

scp kube-controller-manager.service root@192.168.2.175:/usr/lib/systemd/system/

scp kube-controller-manager.service root@192.168.2.176:/usr/lib/systemd/system/

scp kube-controller-manager.service root@192.168.2.177:/usr/lib/systemd/system/

12.6 启动 kube-controller-manager 服务

# 全局刷新service

systemctl daemon-reload

# 设置kube-controller-manager开机启动

systemctl enable kube-controller-manager

#重启kube-controller-manager

systemctl restart kube-controller-manager

12.7 检查服务运行状态

systemctl status kube-controller-manager|grep Active

kube-controller-manager 监听 10257 端口，接收 https 请求：

[root@k8s-master-1 conf]# netstat -lnpt | grep kube-cont

tcp6 0 0 :::10257 :::* LISTEN

24078/kube-controll

12.8 查看当前的 leader

kubectl -n kube-system get leases kube-controller-manager

NAME HOLDER AGE

kube-controller-manager k8s-master-2_c445a762-adc1-4623-a9b5-4d8ea3d34933 1d

12.9 测试 kube-controller-manager 集群的高可用

停掉一个或两个节点的 kube-controller-manager 服务，观察其它节点的日志，看是否获取了 leader 权限。

期待下次的分享，别忘了三连支持博主呀~
我是 念舒_C.ying ，期待你的关注~