K8S集群证书已过期且etcd和apiserver已不能正常使用下的恢复方案

在这种比较极端的情况下，要小心翼翼的规划和操作，才不会让集群彻底死翘翘。首先，几个ca根证书是10年期，应该还没有过期。我们可以基于这几个根证书，来重新生成一套可用的各组件认证证书。

前期，先制定以下方案步骤，能否实现，待验证。

一，制作证书的基本文件。

Ca-csr.json(因为根证书是OK的，所以这个文件，可是列在这里，不会用上)

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",

  },
  "ca": {
    "expiry": "438000h"
  }
}

Ca-config.json(它用来从自签名根ca.crt和ca.key生成新的证书，可以共用)

{
  "signing": {
    "default": {
      "expiry": "43800h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "43800h"
      }
    }
  }
}

二，重新生成etcd系列证书((注意，这是依据/etc/kubernetes/pki/etcd/目录下的ca证书)

Etcd-server.json

{
    "CN": "etcdServer",
    "key": {
        "algo": "rsa",

    },
    "names": [
        {
            "O": "etcd",
            "OU": "etcd Security",
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai"
        }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1,localhost,本机ip,小写主机名 \
  -profile=kubernetes \
  etcd-server.json|cfssljson -bare server

etcd-peer.json

{
    "CN": "etcdPeer",
    "key": {
        "algo": "rsa",

    },
    "names": [
        {
        "O": "etcd",
        "OU": "etcd Security",
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai"
        }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1,localhost,本机ip,小写主机名 \
  -profile=kubernetes \
  etcd-peer.json|cfssljson -bare peer

etcd-client.json

{
    "CN": "etcdClient",
    "key": {
        "algo": "rsa",

    },
    "names": [
        {
        "O": "etcd",
        "OU": "etcd Security",
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai"
        }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -profile=kubernetes \
  etcd-client.json |cfssljson -bare client

三，重新制作apiserver证书(注意，这是依据/etc/kubernetes/pki目录下的ca证书)

Apiserver.json

{
    "CN": "kube-apiserver",
    "key": {
        "algo": "rsa",

    }
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1, kubernetes , kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local,本机ip,小写主机名 \
  -profile=kubernetes \
  apiserver.json |cfssljson -bare apiserver

apiserver-kubelet-client.json

{
    "CN": "kube-apiserver-kubelet-client",
    "key": {
        "algo": "rsa",

    },
    "names": [
        {
        "O": "system:masters"
        }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -profile=kubernetes \
  apiserver-kubelet-client.json |cfssljson -bare apiserver-kubelet-client

三，重新制作front-proxy证书(注意，这是依据/etc/kubernetes/pki目录下的front-proxy-ca证书，它必须和apiserver的ca不一样，牵扯到apiserver的认证顺序，切记)

Front-proxy-client.json

{
    "CN": "front-proxy-client",
    "key": {
        "algo": "rsa",

    }
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -profile=kubernetes \
  front-proxy-client.json |cfssljson -bare front-proxy-client

四，制作scheduler，controller-manager,admin,kubelet,bootstrap证书，此证书只存在于主节点。此证书主要用来生成controller-manager.conf, scheduler.conf, admin.conf, kubelet.conf bootstrap-kubelet.conf。

如果/etc/kubernetes/pki目录下的sa.key,sa.pub存在，则无须更新，因为它没有过期概念。

kube-scheduler-csr.json

{
    "CN": "system:kube-scheduler",
    "key": {
        "algo": "rsa",

    },
    "names": [
      {
        "O": "system:kube-scheduler",
      }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1,localhost,本机ip,小写主机名 \
  -profile=kubernetes \
  kube-scheduler-csr.json|cfssljson -bare kube-scheduler

kube-controller-manager-csr.json

{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",

    },
    "names": [
      {
        "O": "system:kube-controller-manager",
      }
    ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1,localhost,本机ip,小写主机名 \
  -profile=kubernetes \
  kube-controller-manager-csr.json |cfssljson -bare kube-controller-manager

admin-csr.json

{
  "CN": "admin",
  "key": {
    "algo": "rsa",

  },
  "names": [
    {
      "O": "system:masters",
    }
  ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json |cfssljson -bare kube- admin

kubelet-csr.json(这个方法，只适合master上的kubelet运行，不用bootstrap的情况)

{
  "CN": "system:node: 小写主机名",
  "key": {
    "algo": "rsa",

  },
  "names": [
    {
      "O": "system:nodes",
    }
  ]
}

cfssl gencert \
  -ca=ca.crt \
  -ca-key=ca.key \
  -config=ca-config.json \
  -hostname=127.0.0.1,localhost,本机ip,小写主机名 \
  -profile=kubernetes \
  kubelet-csr.json |cfssljson -bare kubelet

如果还需要bootstrap,可以参考下面的url:

https://k2r2bai.com/2018/07/17/kubernetes/deploy/manual-install/

https://www.jianshu.com/p/6650954fa973?tdsourcetag=s_pctim_aiomsg

五，以上文件作好之后，需要根据现在的k8s命令规则改名，还要根据不同的文件，存放于不同的目录。

六，这时，k8s master应该可以启动了。接下来，制作kubeconfig文件，参考url

https://www.cnblogs.com/netsa/p/8134000.html(配置bootstrap及kubelet认证)

https://www.cnblogs.com/charlieroro/p/8489515.html（配置.kube/config文件）

# 设置集群参数

kubectl config set-cluster

# 设置客户端认证参数

kubectl config set-credentials

# 设置上下文参数

kubectl config set-context

# 设置默认上下文

kubectl config use-context

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/cert/ca.pem \
  --embed-certs=true \
  --server=https://ip:port \
  --kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=kube-controller-manager.pem \
  --client-key=kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-context system:kube-controller-manager \
  --cluster=kubernetes \
  --user=system:kube-controller-manager \
  --kubeconfig=kube-controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/cert/ca.pem \
  --embed-certs=true \
  --server=https://ip:port \
  --kubeconfig=kube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler \
  --client-certificate=kube-scheduler.pem \
  --client-key=kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-scheduler.kubeconfig
kubectl config set-context system:kube-scheduler \
  --cluster=kubernetes \
  --user=system:kube-scheduler \
  --kubeconfig=kube-scheduler.kubeconfig
kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-cluster kubernetes \
    --certificate-authority=${PKI_DIR}/ca.pem \
    --embed-certs=true \
    --server=https://ip:port \
    --kubeconfig=${K8S_DIR}/admin.conf

kubectl config set-credentials kubernetes-admin \
    --client-certificate=${PKI_DIR}/admin.pem \
    --client-key=${PKI_DIR}/admin-key.pem \
    --embed-certs=true \
    --kubeconfig=${K8S_DIR}/admin.conf

kubectl config set-context kubernetes-admin@kubernetes \
    --cluster=kubernetes \
    --user=kubernetes-admin \
    --kubeconfig=${K8S_DIR}/admin.conf

kubectl config use-context kubernetes-admin@kubernetes \
    --kubeconfig=${K8S_DIR}/admin.conf

kubectl config set-cluster kubernetes \
  --certificate-authority=${PKI_DIR}/ca.pem \
  --embed-certs=true \
  --server=https://ip:port \
  --kubeconfig=${K8S_DIR}/kubelet.conf && \
kubectl config set-credentials system:node:小写主机名 \
  --client-certificate=${PKI_DIR}/kubelet.pem \
  --client-key=${PKI_DIR}/kubelet-key.pem \
  --embed-certs=true \
  --kubeconfig=${K8S_DIR}/kubelet.conf && \
kubectl config set-context system:node:小写主机名@kubernetes \
  --cluster=kubernetes \
  --user=system:node:小写主机名 \
  --kubeconfig=${K8S_DIR}/kubelet.conf && \
kubectl config use-context system:node:小写主机名@kubernetes \
  --kubeconfig=${K8S_DIR}/kubelet.conf

七，当制作好这些文件之后，按k8s安装的位置，分发文件，重启kubelet，应该就可以重新启动好集群了。