通过 API 快速创建 AlertManager silence

概述

通常我们要 silence 某个 AlertManager 的 alert 时，需要通过 UI 界面操作，如下图：

效率有点低，而且不够自动化，那么是否可以有一种办法快速创建 AlertManager silence 呢？

-- 有的，通过 API.

API Payload

v1

如下：

{
    "matchers": [
        {
            "name": "alername1",
            "value": ".*",
            "isRegex": true
        }
    ],
    "startsAt": "2022-04-29T22:12:33.533Z",
    "endsAt": "2022-04-29T23:11:44.603Z",
    "createdBy": "api",
    "comment": "Silence",
    "status": {
        "state": "active"
    }
}

v2

{
    "matchers": [
        {
            "name": "service",
            "value": "rancher",
            "isRegex": false,
            "isEqual": true
        },
        {
            "name": "alertname",
            "value": "TargetDown",
            "isRegex": false,
            "isEqual": true
        }
    ],
    "startsAt": "2022-04-29T10:11:35.656Z",
    "endsAt": "2022-04-29T12:11:35.656Z",
    "createdBy": "Casey Cui",
    "comment": "配置错误导致的误报",
    "id": null
}

具体实现

curl 实现

Notes:

以 v1 api 为例

如下：

curl https://alertmanager.ewhisper.cn/api/v1/silences -d '{
      "matchers": [
        {
          "name": "alername1",
          "value": ".*",
          "isRegex": true
        }
      ],
      "startsAt": "2022-04-29T22:12:33.533Z",
      "endsAt": "2022-04-29T23:11:44.603Z",
      "createdBy": "api",
      "comment": "Silence",
      "status": {
        "state": "active"
      }
}'

Notes:

在 K8S 集群内，地址一般为：http://alertmanager:9093

python 实现

如下：

#!/usr/bin/python3
import requests
import socket
import datetime
import time

res = requests.post("http://alertmanager:9093/api/v2/silences", json={
    "matchers": [
        {"name": "job", "value": "myjob", "isRegex": False},
        {"name": "instance", "value": "{}:1234".format(socket.gethostname()), "isRegex": False},
        ],
    "startsAt": datetime.datetime.utcfromtimestamp(time.time()).isoformat(),
    "endsAt": datetime.datetime.utcfromtimestamp(time.time() + 4*3600).isoformat(),
    "comment": "Backups on {}".format(socket.gethostname()),
    "createdBy": "My backup script",
    },
    )
res.raise_for_status()
silenceId = res.json()["silenceID"]

移除 silence 的脚本：

res = requests.delete("http://alertmanager:9093/api/v2/silence/{}".format(silenceId))
res.raise_for_status()

EOF