kubernetes promethues预警、报警

k8s addon中prometheus为测试事例，官方推荐生产环境使用Prometheus Operator and kube-prometheus.

1.clone 源码

git clone https://github.com/coreos/prometheus-operator.git

2.不进行报警设置

cd prometheus-operator/contrib/kube-prometheus/

　  2.1）设置对应的service对外开放端口NodePort

alertmanager-service.yamld 、grafana-service.yaml 、prometheus-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    alertmanager: main
  name: alertmanager-main
  namespace: monitoring
spec:
  ports:
  - name: web
    port:
    targetPort: web
    nodePort:
  type: NodePort
  selector:
    alertmanager: main
    app: alertmanager

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  ports:
  - name: http
    port:
    targetPort: http
    nodePort:
  type: NodePort
  selector:
    app: grafana

apiVersion: v1
kind: Service
metadata:
  labels:
    prometheus: k8s
  name: prometheus-k8s
  namespace: monitoring
spec:
  ports:
  - name: web
    port:
    targetPort: web
    nodePort:
  type: NodePort
  selector:
    app: prometheus
    prometheus: k8s

　　2.2）快速启动

   kubectl create -f manifests/

　2.3）查看结果

3.进行发送邮件报警设置
cd prometheus-operator/contrib/

mkdir my-kube-prometheus

cd my-kube-prometheus

3.1)安装jb
go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
3.2)安装gojsontoyaml
go get github.com/brancz/gojsontoyaml
3.3）设置环境变量

export PATH=$PATH:/root/go/bin

3.4)install kube-prometheus
$ jb init 
$ jb install github.com/coreos/prometheus-operator/contrib/kube-prometheus/jsonnet/kube-prometheus

配置alertmanager 使之可以发送邮件
cd vendor/kube-prometheus/alertmanager/
vi alertmanager.libsonnet

3.5）vi manifestsContent.jsonnet

local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') + {
  _config+:: {
    namespace: 'monitoring',
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

manifestsContent.jsonnet

3.6)vi generate-manifest.sh

#!/usr/bin/env bash
set -e
set -x
# only exit with zero if all commands of the pipeline exit successfully
set -o pipefail

# Make sure to start with a clean 'manifests' dir
rm -rf manifests
mkdir manifests

                                               # optional, but we would like to generate yaml, not json
jsonnet -J vendor -m manifests manifestsContent.jsonnet | xargs -I{} sh -c 'cat $1 | gojsontoyaml > $1.yaml; rm -f $1' -- {}

generate-manifest.sh

注意： 3.5和3.6 所在目录

3.7）安装jsonnet

git clone https://github.com/google/jsonnet.git

3.7.1)安装gcc-c++

yum install gcc-c++

3.7.2）编译jsonnet

make

3.7.3)查看是否安装成功

./jsonnet -h

3.7.4）设置jsonnet环境变量

export PATH=$PATH:/data/tools/jsonnet

3.8）启动 generate-manifest.sh

cd /data/prometheus-operator/contrib/my-kube-prometheus/

./generate-manifest.sh

生成manifests文件夹

cd manifests

设置对应的service对外暴露端口 NodePort

3.9)启动服务

kubectl create -f manifests/