高可用Kubernetes集群-7. 部署kube-controller-manager

九．部署kube-controller-manager

kube-controller-manager是Kube-Master相关的3个服务之一，是有状态的服务，会修改集群的状态信息。

如果多个master节点上的相关服务同时生效，则会有同步与一致性问题，所以多master节点中的kube-controller-manager服务只能是主备的关系，kukubernetes采用租赁锁（lease-lock）实现leader的选举，具体到kube-controller-manager，设置启动参数"--leader-elect=true"。

1. 创建kube-controller-manager证书

1）创建kube-conftroller-manager证书签名请求

# kube-controller-mamager与kubei-apiserver通信采用双向TLS认证；
# kube-apiserver提取CN作为客户端的用户名，即system:kube-controller-manager。 kube-apiserver预定义的 RBAC使用的ClusterRoleBindings system:kube-controller-manager将用户system:kube-controller-manager与ClusterRole system:kube-controller-manager绑定
[root@kubenode1 ~]# mkdir -p /etc/kubernetes/controller-manager
[root@kubenode1 ~]# cd /etc/kubernetes/controller-manager/
[root@kubenode1 controller-manager]# touch controller-manager-csr.json
[root@kubenode1 controller-manager]# vim controller-manager-csr.json
{
    "CN": "system:kube-controller-manager",
    "hosts": [
      "172.30.200.21",
      "172.30.200.22",
      "172.30.200.23"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "ChengDu",
            "L": "ChengDu",
            "O": "system:kube-controller-manager",
            "OU": "cloudteam"
        }
    ]
}

2）生成kube-controller-mamager证书与私钥

[root@kubenode1 controller-manager]# cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
-ca-key=/etc/kubernetes/ssl/ca-key.pem \
-config=/etc/kubernetes/ssl/ca-config.json \
-profile=kubernetes controller-manager-csr.json | cfssljson -bare controller-manager

# 分发controller-manager.pem，controller-manager-key.pem
[root@kubenode1 controller-manager]# scp controller-manager*.pem root@172.30.200.22:/etc/kubernetes/controller-manager/
[root@kubenode1 controller-manager]# scp controller-manager*.pem root@172.30.200.23:/etc/kubernetes/controller-manager/

2. 创建kube-controller-manager kubeconfig文件

kube-controller-manager kubeconfig文件中包含Master地址信息与必要的认证信息。

# 配置集群参数；
# --server：指定api-server，采用ha之后的vip；
# cluster名自定义，设定之后需保持一致；
# --kubeconfig：指定kubeconfig文件路径与文件名；如果不设置，默认生成在~/.kube/config文件
[root@kubenode1 controller-manager]# kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://172.30.200.10:6443 \
--kubeconfig=controller-manager.conf

# 配置客户端认证参数；
# 认证用户为前文签名中的“system:kube-controller-manager”；
# 指定对应的公钥证书/私钥等
[root@kubenode1 controller-manager]# kubectl config set-credentials system:kube-controller-manager \
--client-certificate=/etc/kubernetes/controller-manager/controller-manager.pem \
--embed-certs=true \
--client-key=/etc/kubernetes/controller-manager/controller-manager-key.pem \
--kubeconfig=controller-manager.conf

# 配置上下文参数
[root@kubenode1 controller-manager]# kubectl config set-context system:kube-controller-manager@kubernetes \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=controller-manager.conf

# 配置默认上下文
[root@kubenode1 controller-manager]# kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=controller-manager.conf

# 分发controller-manager.conf文件到所有master节点；
[root@kubenode1 controller-manager]# scp controller-manager.conf root@172.30.200.22:/etc/kubernetes/controller-manager/
[root@kubenode1 controller-manager]# scp controller-manager.conf root@172.30.200.23:/etc/kubernetes/controller-manager/

3. 配置kube-controller-manager的systemd unit文件

相关可执行文件在部署kubectl时已部署完成。

# kube-controller-manager在kube-apiserver启动之后启动
[root@kubenode1 ~]# touch /usr/lib/systemd/system/kube-controller-manager.service
[root@kubenode1 ~]# vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=kube-apiserver.service

[Service]
EnvironmentFile=/usr/local/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

# 启动参数文件
# --kubeconfig：kubeconfig配置文件路径，配置文件中包含master地址信息与必要的认证信息；
# --allocate-node：设置为true时，使用云服务商为Pod分配的cidrs，一般仅用在公有云；
# --cluster-name：集群名称，默认即kubernetes；
# --cluster-signing-cert-file / --cluster-signing-key-file：用于集群范围的认证；
# --service-account-private-key-file：用于service account token签名的私钥文件路径；
# --root-ca-file：根ca证书路径，被用于service account 的token secret中
# --insecure-experimental-approve-all-kubelet-csrs-for-group：controller-manager自动授权kubelet客户端证书csr组
# --use-service-account-credentials：设置为true时，表示为每个controller分别设置service account；
# --controllers：启动的contrller列表，默认为”*”，启用所有的controller，但不包含” bootstrapsigner”与”tokencleaner”；
# --leader-elect：设置为true时进行leader选举，集群高可用部署时controller-manager必须选举leader，默认即true
[root@kubenode1 ~]# touch /usr/local/kubernetes/kube-controller-manager.conf
[root@kubenode1 ~]# vim /usr/local/kubernetes/kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER_ARGS="--master=https://172.30.200.10:6443 \
  --kubeconfig=/etc/kubernetes/controller-manager/controller-manager.conf \
  --allocate-node-cidrs=true \
  --service-cluster-ip-range=169.169.0.0/16 \
  --cluster-cidr=10.254.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --root-ca-file=/etc/kubernetes/ssl/ca.pem \
  --insecure-experimental-approve-all-kubelet-csrs-for-group=system:bootstrappers \
  --use-service-account-credentials=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --leader-elect=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes/controller-manager \
  --v=2  1>>/var/log/kubernetes/kube-controller-manager.log 2>&1"

# 创建日志目录
[root@kubenode1 ~]# mkdir -p /var/log/kubernetes/controller-manager 

4. 启动并验证

1）kube-conftroller-manager状态验证

[root@kubenode1 ~]# systemctl daemon-reload
[root@kubenode1 ~]# systemctl enable kube-controller-manager
[root@kubenode1 ~]# systemctl start kube-controller-manager
[root@kubenode1 ~]# systemctl status kube-controller-manager

2）kube-conftroller-manager选举查看

# 因kubenode1是第一个启动kube-controller-manager服务的节点，尝试获取leader权限，成功
[root@kubenode1 ~]# cat /var/log/kubernetes/controller-manager/kube-controller-manager.INFO | grep "leaderelection"

# 在kubenode2上观察，kubenode2在尝试获取leader权限，但未成功，后续操作挂起
[root@kubenode2 ~]# tailf /var/log/kubernetes/controller-manager/kube-controller-manager.INFO