kubernetes部署kube-controller-manager服务

本文档介绍部署高可用 kube-controller-manager 集群的步骤。

该集群包含 3 个节点，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用后，剩余节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。

以下是非认证的的配置文件（用127.0.0.1连接kube-apiserver）：

cat > /lib/systemd/system/kube-controller-manager.service <<"EOF"
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --address=127.0.0.1 \
  --master=http://127.0.0.1:8080 \
  --allocate-node-cidrs=true \
  --service-cluster-ip-range=10.254.0.0/16 \
  --cluster-cidr=172.30.0.0/16 \
  --cluster-name=kubernetes \
  --leader-elect=true \
  --cluster-signing-cert-file=/etc/kubernetes/ca/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ca/ca-key.pem \
  --service-account-private-key-file=/etc/kubernetes/ca/ca-key.pem \
  --root-ca-file=/etc/kubernetes/ca/ca.pem \
  --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF

这个简单很多，但是以后可能会取消，如果要使用认证授权的方式看下面：

------------------------------------------------------------------

为保证通信安全，本文档先生成 x509 证书和私钥，kube-controller-manager 在如下两种情况下使用该证书：

与 kube-apiserver 的安全端口通信时;
在安全端口(https，10252) 输出 prometheus 格式的 metrics；

创建 kube-controller-manager 证书和私钥

cat > kube-controller-manager-csr.json <<EOF
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.111.10",
      "192.168.111.11",
      "192.168.111.12"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "ChongQing",
        "L": "ChongQing",
        "O": "system:kube-controller-manager",
        "OU": "yunwei"
      }
    ]
}
EOF

cfssl gencert -ca=/etc/kubernetes/ca/ca.pem \
  -ca-key=/etc/kubernetes/ca/ca-key.pem \
  -config=/etc/kubernetes/ca/ca-config.json \
  -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

创建和分发 kubeconfig 文件

kubeconfig 文件包含访问 apiserver 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书；

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ca/ca.pem \
--embed-certs=true \
--server=https://192.168.111.9:8443 \
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
--client-certificate=/etc/kubernetes/ca/kube-controller-manager.pem \
--client-key=/etc/kubernetes/ca/kube-controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

分发 kubeconfig 到所有 master 节点：

# scp kube-controller-manager.kubeconfig 192.168.111.11:/etc/kubernetes/
# scp kube-controller-manager.kubeconfig 192.168.111.12:/etc/kubernetes/

创建和分发 kube-controller-manager systemd unit 文件：

cat > /lib/systemd/system/kube-controller-manager.service <<"EOF"
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --port=0 \
  --secure-port=10252 \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --service-cluster-ip-range=10.254.0.0/16 \
  --cluster-signing-cert-file=/etc/kubernetes/ca/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ca/ca-key.pem \
  --root-ca-file=/etc/kubernetes/ca/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/ca/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --horizontal-pod-autoscaler-use-rest-clients=true \
  --horizontal-pod-autoscaler-sync-period=10s \
  --tls-cert-file=/etc/kubernetes/ca/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/ca/kube-controller-manager-key.pem \
  --use-service-account-credentials=true \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on
Restart=on-failure
RestartSec=5
User=k8s

[Install]
WantedBy=multi-user.target
EOF

-------------------------------------------------

启动kube-controller-manager

# systemctl daemon-reload&&for SERVICES in kube-controller-manager;do systemctl enable $SERVICES; systemctl start $SERVICES; systemctl status $SERVICES; done

查看输出的 metric
注意：以下命令在 kube-controller-manager 节点上执行。

kube-controller-manager 监听 10252 端口，接收 https 请求：

# ss -lnpt|grep kube-controll

# curl http://127.0.0.1:10252/metrics |head

测试 kube-controller-manager 集群的高可用
停掉一个或两个节点的 kube-controller-manager 服务，观察其它节点的日志，看是否获取了 leader 权限。
查看当前的 leader
kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml