K8S Canal基于Prometheus进行实时指标监控

部署canal的prometheus监控到k8s中

1、grafana的docker部署方式；https://grafana.com/grafana/download?platform=docker

2、prometheus的docker部署方式： https://github.com/prometheus/prometheus

有了现成的docker镜像后，直接部署即可；

k8s中部署prometheus

yml编排文件如下：

---

apiVersion: apps/v1

kind: Deployment

metadata:

  annotations:

    deployment.kubernetes.io/revision: '16'

    k8s.kuboard.cn/displayName: canal-prometheus

    k8s.kuboard.cn/ingress: 'false'

    k8s.kuboard.cn/service: NodePort

    k8s.kuboard.cn/workload: svc-canal-prometheus

  creationTimestamp: '2020-11-06T03:09:55Z'

  generation: 16

  labels:

    k8s.kuboard.cn/layer: svc

    k8s.kuboard.cn/name: svc-canal-prometheus

  name: svc-canal-prometheus

  namespace: canal-ns

  resourceVersion: '22246892'

  selfLink: /apis/apps/v1/namespaces/canal-ns/deployments/svc-canal-prometheus

  uid: 4ad37eec-3b36-4107-8ed9-07456abba5ba

spec:

  progressDeadlineSeconds: 600

  replicas: 1

  revisionHistoryLimit: 10

  selector:

    matchLabels:

      k8s.kuboard.cn/layer: svc

      k8s.kuboard.cn/name: svc-canal-prometheus

  strategy:

    rollingUpdate:

      maxSurge: 25%

      maxUnavailable: 25%

    type: RollingUpdate

  template:

    metadata:

      annotations:

        kubectl.kubernetes.io/restartedAt: '2020-11-06T14:07:33+08:00'

      labels:

        k8s.kuboard.cn/layer: svc

        k8s.kuboard.cn/name: svc-canal-prometheus

    spec:

      containers:

        - image: prom/prometheus

          imagePullPolicy: Always

          name: canal-prometheus

          terminationMessagePath: /dev/termination-log

          terminationMessagePolicy: File

          volumeMounts:

            - mountPath: /etc/prometheus/prometheus.yml

              name: canal-prometheus-volume

              subPath: etc/prometheus/prometheus.yml

      dnsPolicy: ClusterFirst

      restartPolicy: Always

      schedulerName: default-scheduler

      terminationGracePeriodSeconds: 30

      volumes:

        - configMap:

            defaultMode: 420

            items:

              - key: prometheus.yml

                path: etc/prometheus/prometheus.yml

            name: canal-prometheus

          name: canal-prometheus-volume

status:

  availableReplicas: 1

  conditions:

    - lastTransitionTime: '2020-11-06T05:30:16Z'

      lastUpdateTime: '2020-11-06T05:30:16Z'

      message: Deployment has minimum availability.

      reason: MinimumReplicasAvailable

      status: 'True'

      type: Available

    - lastTransitionTime: '2020-11-06T05:55:49Z'

      lastUpdateTime: '2020-11-06T06:07:43Z'

      message: >-

        ReplicaSet "svc-canal-prometheus-6f7d7b66c5" has successfully

        progressed.

      reason: NewReplicaSetAvailable

      status: 'True'

      type: Progressing

  observedGeneration: 16

  readyReplicas: 1

  replicas: 1

  updatedReplicas: 1

此处有一个有趣的点是，k8s中使用configMap进行目录挂载时，一般情况下我们直接挂载到对应的容器目录后，此时目录将会被覆盖，而此时在配置普罗米修斯的文件映射时，则只是挂载到了具体的文件中；主要的配置则是：subPath: etc/prometheus/prometheus.yml，通过subPath的方式可以直接挂载到具体的文件中；

原创声明：作者：Arnold.zhao 博客园地址：https://www.cnblogs.com/zh94

所挂载的具体配置文件的内容如下：

# my global config test

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration

alerting:

  alertmanagers:

  - static_configs:

    - targets:

      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

  # - "first_rules.yml"

  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: 'canal'

    # metrics_path defaults to '/metrics'

    # scheme defaults to 'http'.

    static_configs:

    - targets: ['svc-canal-deployer:11112']

主要是配置下对应的canal-deployer的地址即可；

targets: ['svc-canal-deployer:11112']

默认情况下canal-deployer的监控端口就是11112，当然如果你修改过该端口，另当别论了。

svc-canal-deployer是canal-deployer的server名称，由于canal-deployer此处也是已经部署在k8s了所以直接使用服务名进行访问，由k8s service自动做转发即可，如果此处不是在k8s的话，则直接配置canal-deployer的ip地址即可，一样的。

对应的configMap的创建yml如下：

---

apiVersion: v1

data:

  prometheus.yml: >-

    # my global config test 

    global:

      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

      # scrape_timeout is set to the global default (10s).

    # Alertmanager configuration

    alerting:

      alertmanagers:

      - static_configs:

        - targets:

          # - alertmanager:9093

    # Load rules once and periodically evaluate them according to the global

    'evaluation_interval'.

    rule_files:

      # - "first_rules.yml"

      # - "second_rules.yml"

    # A scrape configuration containing exactly one endpoint to scrape:

    # Here it's Prometheus itself.

    scrape_configs:

      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

      - job_name: 'canal'

        # metrics_path defaults to '/metrics'

        # scheme defaults to 'http'.

        static_configs:

        - targets: ['svc-canal-deployer:11112']

kind: ConfigMap

metadata:

  creationTimestamp: '2020-11-06T03:15:04Z'

  name: canal-prometheus

  namespace: canal-ns

  resourceVersion: '22246778'

  selfLink: /api/v1/namespaces/canal-ns/configmaps/canal-prometheus

  uid: 2918cb4e-acd6-4c82-9a2e-19e575ba6cea

创建一个专用的Service进行端口映射，此处映射端口为9090

---

apiVersion: v1

kind: Service

metadata:

  annotations:

    k8s.kuboard.cn/displayName: canal-prometheus

    k8s.kuboard.cn/workload: svc-canal-prometheus

  creationTimestamp: '2020-11-06T03:11:34Z'

  labels:

    k8s.kuboard.cn/layer: svc

    k8s.kuboard.cn/name: svc-canal-prometheus

  name: svc-canal-prometheus

  namespace: canal-ns

  resourceVersion: '22204370'

  selfLink: /api/v1/namespaces/canal-ns/services/svc-canal-prometheus

  uid: 6246dafe-f8fd-42ec-8e27-caf38539d35c

spec:

  clusterIP: 10.204.71.228

  externalTrafficPolicy: Cluster

  ports:

    - name: prometheus-9090

      nodePort: 30018

      port: 9090

      protocol: TCP

      targetPort: 9090

  selector:

    k8s.kuboard.cn/layer: svc

    k8s.kuboard.cn/name: svc-canal-prometheus

  sessionAffinity: None

  type: NodePort

好了，部署完成，此时访问端口结果如下：

部署grafana

Deployment的yml如下

---

apiVersion: apps/v1

kind: Deployment

metadata:

  annotations:

    deployment.kubernetes.io/revision: '1'

    k8s.kuboard.cn/ingress: 'false'

    k8s.kuboard.cn/service: NodePort

    k8s.kuboard.cn/workload: web-canal-grafana

  creationTimestamp: '2020-11-06T06:10:27Z'

  generation: 1

  labels:

    k8s.kuboard.cn/layer: web

    k8s.kuboard.cn/name: web-canal-grafana

  name: web-canal-grafana

  namespace: canal-ns

  resourceVersion: '22247823'

  selfLink: /apis/apps/v1/namespaces/canal-ns/deployments/web-canal-grafana

  uid: 484350e4-b408-4361-ac27-633f8d815468

spec:

  progressDeadlineSeconds: 600

  replicas: 1

  revisionHistoryLimit: 10

  selector:

    matchLabels:

      k8s.kuboard.cn/layer: web

      k8s.kuboard.cn/name: web-canal-grafana

  strategy:

    rollingUpdate:

      maxSurge: 25%

      maxUnavailable: 25%

    type: RollingUpdate

  template:

    metadata:

      creationTimestamp: null

      labels:

        k8s.kuboard.cn/layer: web

        k8s.kuboard.cn/name: web-canal-grafana

    spec:

      containers:

        - image: grafana/grafana

          imagePullPolicy: Always

          name: canal-grafana

          resources: {}

          terminationMessagePath: /dev/termination-log

          terminationMessagePolicy: File

      dnsConfig: {}

      dnsPolicy: ClusterFirst

      restartPolicy: Always

      schedulerName: default-scheduler

      securityContext:

        seLinuxOptions: {}

      terminationGracePeriodSeconds: 30

再进行一下server代理，yml如下；映射端口为3000端口

---

apiVersion: v1

kind: Service

metadata:

  annotations:

    k8s.kuboard.cn/workload: web-canal-grafana

  creationTimestamp: '2020-11-06T06:10:27Z'

  labels:

    k8s.kuboard.cn/layer: web

    k8s.kuboard.cn/name: web-canal-grafana

  name: web-canal-grafana

  namespace: canal-ns

  resourceVersion: '22247699'

  selfLink: /api/v1/namespaces/canal-ns/services/web-canal-grafana

  uid: 8cbde138-6855-4eaf-b9cd-8dae72a2efeb

spec:

  clusterIP: 10.204.195.124

  externalTrafficPolicy: Cluster

  ports:

    - name: canal-grafana-3000

      nodePort: 31010

      port: 3000

      protocol: TCP

      targetPort: 3000

  selector:

    k8s.kuboard.cn/layer: web

    k8s.kuboard.cn/name: web-canal-grafana

  sessionAffinity: None

  type: NodePort

OK，启动后，访问对应的应用后，效果如图所示：

此时还没有进行grefana的配置，所以无法获取canal的监控信息

Greafana配置canal监控信息

很简单，基本按照官方的说明即可,

新建一个dataSource，选择prometheus，然后填写对应的prometheus的url地址即可。

填写完成后，最后一步则是，grafana中导入 canal的监控指标

此时按照URL的导入方式：https://raw.githubusercontent.com/alibaba/canal/master/deployer/src/main/resources/metrics/Canal_instances_tmpl.json

导入该json文件即可；

不过我在处理的时候，该url导入不可用，所以，如果你也是不可用的话，可以直接

wget https://raw.githubusercontent.com/alibaba/canal/master/deployer/src/main/resources/metrics/Canal_instances_tmpl.json

获取对应的文件内容后，再填写到第二个框中即可；

最终的监控效果图

原创声明：作者：Arnold.zhao 博客园地址：https://www.cnblogs.com/zh94

具体各监控指标所表示的含义，直接看github canal文档即可；

由于此处我的canal client并没有直接通过tcp的方式和canal -deployer进行交互，而是直接接入的canal-deployer所吐出来的kakfa数据，所以此处所展示的client指标则为空；

参考链接：

关于普罗米修斯（prometheus）的基本概念

 关于Grafana的基本概念

 关于canal安装Grafana&prometheus进行监控的说明

 关于k8s中目录挂载时不覆盖容器原目录的用法