Prometheus 监控K8S 资源状态对象

Prometheus 监控K8S 资源状态对象

• 官方文档：https://github.com/kubernetes/kube-state-metrics

kube-state-metrics是一个简单的服务，它监听Kubernetes API服务器并生成有关对象状态的指标。它不关注单个Kubernetes组件的运行状况，而是关注内部各种对象的运行状况，例如部署，节点和容器。

采集了k8s中各种资源对象的状态信息：

kube-state-metrics
kube_daemonset_*
kube_deployment_*
kube_job_*
kube_namespace_*
kube_node_*
kube_persistentvolumeclaim_*
kube_pod_container_*
kube_pod_*
kube_replicaset_*
kube_service_*
kube_statefulset_*

配置文件

下列是已经修改好的配置文件

• # 主程序
• kube-state-metrics-deployment.yaml

• apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: kube-state-metrics
    namespace: kube-system
    labels:
      k8s-app: kube-state-metrics
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
      version: v1.3.0
  spec:
    selector:
      matchLabels:
        k8s-app: kube-state-metrics
        version: v1.3.0
    replicas: 1
    template:
      metadata:
        labels:
          k8s-app: kube-state-metrics
          version: v1.3.0
        annotations:
          scheduler.alpha.kubernetes.io/critical-pod: ''
      spec:
        priorityClassName: system-cluster-critical
        serviceAccountName: kube-state-metrics
        containers:
        - name: kube-state-metrics
          # 修改为镜像地址
          image: lizhenliang/kube-state-metrics:v1.3.0
          ports:
          - name: http-metrics
            containerPort: 8080
          - name: telemetry
            containerPort: 8081
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 5
        - name: addon-resizer
          # 修改为镜像地址
          image: lizhenliang/addon-resizer:1.8.3
          resources:
            limits:
              cpu: 100m
              memory: 30Mi
            requests:
              cpu: 100m
              memory: 30Mi
          env:
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
          command:
            - /pod_nanny
            - --config-dir=/etc/config
            - --container=kube-state-metrics
            - --cpu=100m
            - --extra-cpu=1m
            - --memory=100Mi
            - --extra-memory=2Mi
            - --threshold=5
            - --deployment=kube-state-metrics
        volumes:
          - name: config-volume
            configMap:
              name: kube-state-metrics-config
  ---
  # Config map for resource configuration.
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: kube-state-metrics-config
    namespace: kube-system
    labels:
      k8s-app: kube-state-metrics
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
  data:
    NannyConfiguration: |-
      apiVersion: nannyconfig/v1alpha1
      kind: NannyConfiguration

  配置文件

• # APIServer授权
• kube-state-metrics-rbac.yaml

• apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: kube-state-metrics
    namespace: kube-system
    labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: kube-state-metrics
    labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
  rules:
  - apiGroups: [""]
    resources:
    - configmaps
    - secrets
    - nodes
    - pods
    - services
    - resourcequotas
    - replicationcontrollers
    - limitranges
    - persistentvolumeclaims
    - persistentvolumes
    - namespaces
    - endpoints
    verbs: ["list", "watch"]
  - apiGroups: ["extensions"]
    resources:
    - daemonsets
    - deployments
    - replicasets
    verbs: ["list", "watch"]
  - apiGroups: ["apps"]
    resources:
    - statefulsets
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources:
    - cronjobs
    - jobs
    verbs: ["list", "watch"]
  - apiGroups: ["autoscaling"]
    resources:
    - horizontalpodautoscalers
    verbs: ["list", "watch"]
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: kube-state-metrics-resizer
    namespace: kube-system
    labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
  rules:
  - apiGroups: [""]
    resources:
    - pods
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources:
    - deployments
    resourceNames: ["kube-state-metrics"]
    verbs: ["get", "update"]
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: kube-state-metrics
    labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: kube-state-metrics
  subjects:
  - kind: ServiceAccount
    name: kube-state-metrics
    namespace: kube-system
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    name: kube-state-metrics
    namespace: kube-system
    labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: kube-state-metrics-resizer
  subjects:
  - kind: ServiceAccount
    name: kube-state-metrics
    namespace: kube-system

  配置文件

• # 暴露端口
• kube-state-metrics-service.yaml

• apiVersion: v1
  kind: Service
  metadata:
    name: kube-state-metrics
    namespace: kube-system
    labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
      kubernetes.io/name: "kube-state-metrics"
    annotations:
      prometheus.io/scrape: 'true'
  spec:
    ports:
    - name: http-metrics
      port: 8080
      targetPort: http-metrics
      protocol: TCP
    - name: telemetry
      port: 8081
      targetPort: telemetry
      protocol: TCP
    selector:
      k8s-app: kube-state-metrics

  配置文件

部署

1、创建容器

kubectl apply -f kube-state-metrics-rbac.yaml
kubectl apply -f kube-state-metrics-deployment.yaml
kubectl apply -f kube-state-metrics-service.yaml

2、查看创建pod

kubectl get pods -n kube-system

NAME                                 READY   STATUS    RESTARTS   AGE
coredns-64479cf49b-lsqqn             1/1     Running   0          5h26m
kube-state-metrics-6459c9f7f-wbwwt   2/2     Running   0          58s
prometheus-0                         2/2     Running   0          4h12m

3、测试访问

4、Grafana可视化导入模板：6417

5、查看监控（其他资源可根据主机情况进行微调）