suse 12 二进制部署 Kubernetets 1.19.7 - 第13章 - 部署metrics-server插件

文章目录

• • 1.13.0、创建metrics-server证书和私钥
  • 1.13.1、生成metrics-server证书和私钥
  • 1.13.2、开启kube-apiserver聚合配置
  • 1.13.3、分发配置文件和秘钥到其他节点
  • 1.13.4、重启所有的kube-apiserver组件
  • 1.13.5、下载yaml文件
  • 1.13.6、配置yaml文件
  • 1.13.7、验证metrics-server功能

• suse 12 二进制部署 Kubernetes 集群系列合集：
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第00章 - 环境准备
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第01章 - 创建CA证书和kubectl集群管理命令
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第02章 - 部署etcd集群
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第03章 - 部署flannel插件
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第04章 - 部署docker服务
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第05章 - 部署kube-nginx
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第06章 - 部署kube-apiserver组件
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第07章 - 部署kube-controller-manager组件
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第08章 - 部署kube-scheduler组件
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第09章 - 部署kubelet组件
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第10章 - 部署kube-proxy组件
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第11章 - 部署coredns组件
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第12章 - 部署dashboard插件
  • suse 12 二进制部署 Kubernetets 1.19.7 - 第13章 - 部署metrics-server插件
  • suse 12 编译部署Keepalived + nginx 为 kube-apiserver 提供高可用
  • suse 12 二进制部署 Kubernetets 1.19.7 - 番外篇 - 增加node节点

metrics-server用于监测node,pod等的CPU，内存使用情况（hpa弹性伸缩依赖metrics-server插件）

1.13.0、创建metrics-server证书和私钥

k8s-01:~ # cd /opt/k8s/ssl/
k8s-01:/opt/k8s/ssl # source /opt/k8s/bin/k8s-env.sh
k8s-01:/opt/k8s/ssl # cat > metrics-server-csr.json <<EOF
{
  "CN": "aggregator",
  "hosts": [
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "ShangHai",
      "L": "ShangHai",
      "O": "k8s",
      "OU": "bandian"
    }
  ]
}
EOF

1.13.1、生成metrics-server证书和私钥

k8s-01:/opt/k8s/ssl # cfssl gencert -ca=/opt/k8s/ssl/ca.pem \
-ca-key=/opt/k8s/ssl/ca-key.pem \
-config=/opt/k8s/ssl/ca-config.json \
-profile=kubernetes metrics-server-csr.json | cfssljson -bare metrics-server

1.13.2、开启kube-apiserver聚合配置

• 在kube-apiserver.service文件里面，增加如下内容，用来开启聚合（此操作，后面需要重启kube-apiserver组件，建议在部署kube-apiserver的时候，就开启聚合）

--proxy-client-cert-file=/etc/kubernetes/cert/metrics-server.pem \\
--proxy-client-key-file=/etc/kubernetes/cert/metrics-server-key.pem \\
--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
--requestheader-allowed-names=aggregator \\
--requestheader-extra-headers-prefix="X-Remote-Extra-" \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User

• 为了方便，就重新创建kube-apiserver.service文件了（注意自己kube-apiserver的service文件，别直接复制黏贴我的配置文件）

k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > kube-apiserver.service.template <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=${K8S_DIR}/kube-apiserver
ExecStart=/opt/k8s/bin/kube-apiserver \\
  --v=2 \\
  --advertise-address=##NODE_IP## \\
  --secure-port=6443 \\
  --bind-address=##NODE_IP## \\
  --etcd-servers=${ETCD_ENDPOINTS} \\
  --allow-privileged=true \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
  --authorization-mode=RBAC,Node \\
  --enable-bootstrap-token-auth=true \\
  --token-auth-file=/etc/kubernetes/cert/token.csv \\
  --service-node-port-range=${NODE_PORT_RANGE} \\
  --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\
  --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\
  --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\
  --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\
  --client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --service-account-key-file=/etc/kubernetes/cert/ca.pem \\
  --etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\
  --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\
  --audit-log-maxage=15 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-truncate-enabled \\
  --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\
  --proxy-client-cert-file=/etc/kubernetes/cert/metrics-server.pem \\
  --proxy-client-key-file=/etc/kubernetes/cert/metrics-server-key.pem \\
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-allowed-names=aggregator \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User

Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

1.13.3、分发配置文件和秘钥到其他节点

#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh

# 替换模板文件
for (( i=0; i < 3; i++ ))
do
    sed -e "s/##NODE_IP##/${MASTER_IPS[i]}/" /opt/k8s/conf/kube-apiserver.service.template > \
           /opt/k8s/conf/kube-apiserver-${MASTER_IPS[i]}.service
done

for host in ${MASTER_IPS[@]}
do
    printf "\e[1;34m${host}\e[0m\n"
    scp /opt/k8s/ssl/metrics-server*.pem ${host}:/etc/kubernetes/cert/
    scp /opt/k8s/conf/kube-apiserver-${host}.service ${host}:/etc/systemd/system/kube-apiserver.service
done

1.13.4、重启所有的kube-apiserver组件

#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh

for host in ${MASTER_IPS[@]}
do
    printf "\e[1;34m${host}\e[0m\n"
    ssh root@${host} "systemctl daemon-reload && \
                      systemctl restart kube-apiserver && \
                      systemctl status kube-apiserver | grep Active"
done

1.13.5、下载yaml文件

k8s-01:~ # wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml

1.13.6、配置yaml文件

• 由于github上面拉取的yaml当中，有许多内容需要修改，因此，下面将修改好的yaml文件放上来了，可以直接使用

k8s-01:~ # vim components.yaml

---
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      nodeSelector:
        kubernetes.io/os: linux
        kubernetes.io/arch: "amd64"
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: main-port
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

k8s-01:~ # kubectl apply -f components.yaml

1.13.7、验证metrics-server功能

• metrics-server启动会比较慢，耐心等待1-3分钟，出现如下效果，则成功
• 若没有出现，则使用 kubectl logs -n kube-system metrics-server-xxxx 查看日志

k8s-01:~ # kubectl top node
NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
192.168.72.55   129m         6%     2232Mi          58%
192.168.72.56   119m         5%     1555Mi          40%
192.168.72.57   114m         5%     1425Mi          37%
192.168.72.58   31m          1%     711Mi           18%
192.168.72.59   28m          1%     733Mi           19%
k8s-01:~ # kubectl top pod -A
NAMESPACE     NAME                              CPU(cores)   MEMORY(bytes)
kube-system   coredns-689d7d9f49-s2qjn          2m           13Mi
kube-system   coredns-689d7d9f49-vc9k4          3m           17Mi
kube-system   metrics-server-666566b66d-jfl7v   2m           12Mi