Prometheus Operator 对接 Thanos

文章转载自：https://jishuin.proginn.com/p/763bfbd56ae4

使用 Prometheus Operator 来进行监控，在 Prometheus 高可用的章节中也手动部署了 Thanos，Prometheus Operator 中提供了 ThanosRuler 这样的 CRD 组件，但是如果使用 Prometheus Operator 部署整个监控服务的话，其他 Thanos 组件（比如 Querier、Store 等）需要单独配置，Thanos 和 Prometheus 进行对接是通过 Sidecar 组件完成的，而在 Prometheus 的 CRD 中就提供了对 Sidecar 组件的配置属性，该 Sidecar 可以连接到 Thanos Querier，还可以选择备份对象存储中的数据。

Thanos 的整个架构图如下所示：

Thanos Sidecar

要使用 Prometheus Operator 来对接 Thanos，可以直接查看 Prometheus 这个 CRD 的相关属性：

$ kubectl explain prometheus.spec.thanos
......

现阶段 Prometheus CRD 里面对接 Thanos 的方式是一个实验特性，所以如果你是在生产环境要使用的话需要注意，可能后续版本就变动了，这里我们可以直接通过 thanos 属性来指定使用的镜像版本，以及对应的对象存储配置，这里我们仍然使用 minio 来做对象存储（部署参考前面章节），首先登录 MinIO 创建一个 thanos 的 bucket。然后创建一个对象存储配置文件：

# thanos-storage-minio.yaml
type: s3
config:
  bucket: thanos # bucket 名称
  endpoint: minio.default.svc.cluster.local:9000 # minio 访问地址
  access_key: minio
  secret_key: minio123
  insecure: true
  signature_version2: false

使用上面的配置文件来创建一个 Secret 对象：

$ kubectl create secret generic thanos-objectstorage --from-file=thanos.yaml=thanos-storage-minio.yaml -n monitoring
secret/thanos-objectstorage created

对象存储的配置准备好过后，接下来我们就可以在 Prometheus CRD 中添加对应的 Thanos 配置了，完整的资源对象如下所示：

apiVersion: monitoring.coreos.com/v1
kind: Prometheus # crd 类型，表示资源对象，类似于Pod、Deployment，就是用来管理部署Prometheus应用的
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
      - name: alertmanager-main
        namespace: monitoring
        port: web
  image: prom/prometheus:v2.26.0
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector: # 用来匹配rule规则的selector
    matchLabels: # 匹配的是具有下面两个标签的PrometheusRule这个资源对象
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.26.0
  additionalScrapeConfigs:
    key: prometheus-additional.yaml
    name: additional-configs
  thanos: # 添加 thanos 配置
    image: thanosio/thanos:v0.18.0
    resources:
      limits:
        cpu: 500m
        memory: 500Mi
      requests:
        cpu: 100m
        memory: 500Mi
    objectStorageConfig:
      key: thanos.yaml
      name: thanos-objectstorage
  # storage:
  #   volumeClaimTemplate:
  #     spec:
  #       storageClassName: rook-ceph-block  # 指定storageclass（别用nfs做存储）
  #       resources:
  #         requests:
  #           storage: 20Gi

然后直接更新即可：

$ kubectl apply -f prometheus-prometheus.yaml
prometheus.monitoring.coreos.com/k8s configured

更新完成后我们再次查看更新后的 Prometheus Pod，可以发现已经变成了 3 个容器了：

$ kubectl get pods -n monitoring -l app=prometheus
NAME               READY   STATUS    RESTARTS   AGE
prometheus-k8s-0   3/3     Running   1          117s
prometheus-k8s-1   3/3     Running   1          2m41s

把生成的资源对象导出来查看验证下：

$ kubectl get sts -n monitoring prometheus-k8s -o yaml
- args:  # sidecar 容器
  - sidecar
  - --prometheus.url=http://localhost:9090/
  - --grpc-address=[$(POD_IP)]:10901
  - --http-address=[$(POD_IP)]:10902
  - --objstore.config=$(OBJSTORE_CONFIG)
  - --tsdb.path=/prometheus
  env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: status.podIP
  - name: OBJSTORE_CONFIG
    valueFrom:
      secretKeyRef:
        key: thanos.yaml
        name: thanos-objectstorage
  image: thanosio/thanos:v0.18.0
  imagePullPolicy: IfNotPresent
  name: thanos-sidecar
  ports:
  - containerPort: 10902
    name: http
    protocol: TCP
  - containerPort: 10901
    name: grpc
    protocol: TCP
......

可以看到在原来的基础上新增了一个 sidecar 容器，正常每 2 个小时会上传一次数据，查看 sidecar 可以查看到相关日志：

$ kubectl logs -f prometheus-k8s-0 -c thanos-sidecar -n monitoring
......
level=info ts=2021-04-14T01:48:22.292651206Z caller=sidecar.go:155 msg="successfully loaded prometheus external labels" external_labels="{prometheus=\"monitoring/k8s\", prometheus_replica=\"prometheus-k8s-0\"}"
level=info ts=2021-04-14T01:48:22.292687688Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2021-04-14T04:48:54.276596974Z caller=shipper.go:334 msg="upload new block" id=01F37BB4A8EWDBN1AS25Z8MG5J
level=info ts=2021-04-14T05:00:24.275616176Z caller=shipper.go:334 msg="upload new block" id=01F37C07T9BS3744WKS8KEDZCS

Thanos Querier

Thanos Querier 组件提供了从所有 prometheus 实例中一次性检索指标的能力。它与原 prometheus 的 PromQL 和 HTTP API 是完全兼容的，所以同样可以和 Grafana 一起使用。

因为 Querier 组件是要和 Sidecar 以及 Store 组件进行对接的，所以在 Querier 组件的方向参数中需要配置上上面我们启动的 Thanos Sidecar，同样我们可以通过对应的 Headless Service 来进行发现，自动创建的 Service 名为 prometheus-operated（可以通过 Statefulset 查看）：

$ kubectl describe svc -n monitoring prometheus-operated
Name:              prometheus-operated
Namespace:         monitoring
Labels:            operated-prometheus=true
Annotations:       <none>
Selector:          app=prometheus
Type:              ClusterIP
IP:                None
Port:              web  9090/TCP
TargetPort:        web/TCP
Endpoints:         10.244.1.92:9090,10.244.2.89:9090
Port:              grpc  10901/TCP
TargetPort:        grpc/TCP
Endpoints:         10.244.1.92:10901,10.244.2.89:10901
Session Affinity:  None
Events:            <none>

Thanos Querier 组件完整的资源清单如下所示，需要注意的是 Prometheus Operator 部署的 prometheus 实例多副本的 external_labels 标签为 prometheus_replica：

# thanos-querier.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-querier
  namespace: monitoring
  labels:
    app: thanos-querier
spec:
  selector:
    matchLabels:
      app: thanos-querier
  template:
    metadata:
      labels:
        app: thanos-querier
    spec:
      containers:
        - name: thanos
          image: thanosio/thanos:v0.18.0
          args:
            - query
            - --log.level=debug
            - --query.replica-label=prometheus_replica # prometheus-operator 里面配置的副本标签为 prometheus_replica
            # Discover local store APIs using DNS SRV.
            - --store=dnssrv+prometheus-operated:10901
          ports:
            - name: http
              containerPort: 10902
            - name: grpc
              containerPort: 10901
          resources:
            requests:
              memory: "2Gi"
              cpu: "1"
            limits:
              memory: "2Gi"
              cpu: "1"
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: http
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              path: /-/healthy
              port: http
            initialDelaySeconds: 15
---
apiVersion: v1
kind: Service
metadata:
  name: thanos-querier
  namespace: monitoring
  labels:
    app: thanos-querier
spec:
  ports:
    - port: 9090
      targetPort: http
      name: http
  selector:
    app: thanos-querier
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: thanos-querier
  namespace: monitoring
  labels:
    app: thanos-querier
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`thanos.k8s.local`)
      services:
        - kind: Service
          name: thanos-querier
          port: 9090

直接创建上面的资源对象即可：

$ kubectl apply -f thanos-querier.yaml
$ kubectl get pods -n monitoring -l app=thanos-querier
NAME                              READY   STATUS    RESTARTS   AGE
thanos-querier-69bb75c8cc-5g6t6   1/1     Running   0          105s

部署完成后我们可以在浏览器中打开 Querier 的页面，查看已经关联上的 Stores：

比如在 Graph 页面查询 node_load1 指标，记住勾选上 Use Deduplication 用于去重查询：

Thanos Store

接着需要部署 Thanos Store 组件，该组件和可以 Querier 组件一起协作从指定对象存储的 bucket 中检索历史指标数据，所以自然在部署的时候我们需要指定对象存储的配置，Store 组件配置完成后还需要加入到 Querier 组件里面去：

# thanos-store.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-store
  namespace: monitoring
  labels:
    app: thanos-store
spec:
  selector:
    matchLabels:
      app: thanos-store
  serviceName: thanos-store
  template:
    metadata:
      labels:
        app: thanos-store
    spec:
      containers:
        - name: thanos
          image: thanosio/thanos:v0.18.0
          args:
            - "store"
            - "--log.level=debug"
            - "--data-dir=/data"
            - "--objstore.config-file=/etc/secret/thanos.yaml"
            - "--index-cache-size=500MB"
            - "--chunk-pool-size=500MB"
          ports:
            - name: http
              containerPort: 10902
            - name: grpc
              containerPort: 10901
          livenessProbe:
            httpGet:
              port: 10902
              path: /-/healthy
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              port: 10902
              path: /-/ready
            initialDelaySeconds: 15
          volumeMounts:
            - name: object-storage-config
              mountPath: /etc/secret
              readOnly: false
      volumes:
        - name: object-storage-config
          secret:
            secretName: thanos-objectstorage
---
apiVersion: v1
kind: Service
metadata:
  name: thanos-store
  namespace: monitoring
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: grpc
      port: 10901
      targetPort: grpc
  selector:
    app: thanos-store

直接部署上面的资源对象即可：

$ kubectl apply -f thanos-store.yaml
statefulset.apps/thanos-store created
service/thanos-store created
$ kubectl get pods -n monitoring -l app=thanos-store
NAME             READY   STATUS    RESTARTS   AGE
thanos-store-0   1/1     Running   0          106s

部署完成后为了让 Querier 组件能够发现 Store 组件，我们还需要在 Querier 组件中增加 Store 组件的发现：

containers:
  - name: thanos
    image: thanosio/thanos:v0.18.0
    args:
      - query
      - --log.level=debug
      - --query.replica-label=prometheus_replica
      # Discover local store APIs using DNS SRV.
      - --store=dnssrv+prometheus-operated:10901
      - --store=dnssrv+thanos-store:10901

更新后再次前往 Querier 组件的页面查看发现的 Store 组件正常会多一个 Thanos Store 的组件。

Thanos Compactor

Thanos Compactor 组件可以对我们收集的历史数据进行下采样，可以减少文件的大小。部署方式和之前没什么太大的区别，主要也就是对接对象存储。

# thanos-compactor
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-compactor
  namespace: monitoring
  labels:
    app: thanos-compactor
spec:
  selector:
    matchLabels:
      app: thanos-compactor
  serviceName: thanos-compactor
  template:
    metadata:
      labels:
        app: thanos-compactor
    spec:
      containers:
        - name: thanos
          image: thanosio/thanos:v0.18.0
          args:
            - "compact"
            - "--log.level=debug"
            - "--data-dir=/data"
            - "--objstore.config-file=/etc/secret/thanos.yaml"
            - "--wait"
          ports:
            - name: http
              containerPort: 10902
          livenessProbe:
            httpGet:
              port: 10902
              path: /-/healthy
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              port: 10902
              path: /-/ready
            initialDelaySeconds: 15
          volumeMounts:
            - name: object-storage-config
              mountPath: /etc/secret
              readOnly: false
      volumes:
        - name: object-storage-config
          secret:
            secretName: thanos-objectstorage

同样直接创建上面的资源对象即可：

kubectl apply -f thanos-compactor.yaml

最后如果想通过 Thanos 的 Ruler 组件来配置报警规则，可以直接使用 Prometheus Operator 提供的 ThanosRuler 这个 CRD 对象，不过还是推荐直接和单独的 prometheus 实例配置报警规则，这样调用链路更短，出现问题的时候排查也更方便。Thanos Ruler 组件允许配置记录和告警规则，跨越多个 prometheus 实例进行处理，一个 ThanosRuler 实例至少需要一个 queryEndpoint 指向 Thanos Queriers 或 prometheus 实例的位置，如下所示：

# ThanosRuler Demo
apiVersion: monitoring.coreos.com/v1
kind: ThanosRuler
metadata:
  name: thanos-ruler-demo
  labels:
    example: thanos-ruler
  namespace: monitoring
spec:
  image: thanosio/thanos
  ruleSelector:
    matchLabels: # 匹配 Rule 规则
      role: my-thanos-rules
  queryEndpoints: # querier 地址
    - dnssrv+_http._tcp.my-thanos-querier.monitoring.svc.cluster.local

ThanosRuler 组件使用的记录和警报规则与 Prometheus 里面配置的 PrometheusRule 对象，比如上面的示例中，表示包含 role=my-thanos-rules 标签的 PrometheusRule 对象规则会被添加到 Thanos Ruler Pod 中去。

最后通过 Prometheus Operator 对接上 Thanos 过后的所有资源对象如下所示：

$ kubectl get pods -n monitoring
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2     Running   0          5h14m
alertmanager-main-1                    2/2     Running   0          5h14m
alertmanager-main-2                    2/2     Running   0          5h13m
dingtalk-hook-7c7c5559fd-kwfqk         1/1     Running   0          5h14m
grafana-f8cd57fcf-2q5h6                1/1     Running   0          5h14m
kube-state-metrics-587bfd4f97-pks8n    3/3     Running   0          5h14m
node-exporter-hzfwh                    2/2     Running   18         11d
node-exporter-kpd7q                    2/2     Running   12         5d22h
node-exporter-tbtfk                    2/2     Running   36         11d
prometheus-adapter-69b8496df6-cnhzn    1/1     Running   0          5h14m
prometheus-k8s-0                       3/3     Running   1          3h59m
prometheus-k8s-1                       3/3     Running   1          4h
prometheus-operator-7649c7454f-ls8wc   2/2     Running   0          5h14m
thanos-compactor-0                     1/1     Running   0          113m
thanos-querier-799845897-x4ljh         1/1     Running   0          128m
thanos-store-0                         1/1     Running   0          130m

正常 minio 对象存储上面也会有上传的历史数据了：