Kubernetes Heapster

Heapster是容器集群监控和性能分析工具，HPA、Dashborad、Kubectl top都依赖于heapster收集的数据。

但是Heapster从kubernetes 1.8以后已经被遗弃了...... 被metrics-server所替代......

kubernetes 1.11 中部署Heapster 1.5.4版本的过程

Heapster 部署yaml文件

apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: heapster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:heapster
subjects:
- kind: ServiceAccount
  name: heapster
  namespace: kube-system

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  selector:
      matchLabels:
        k8s-app: heapster
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        # image: k8s.gcr.io/heapster-amd64:v1.5.4 将默认google的官方镜像替换为阿里云镜像，否则你懂得
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-amd64:v1.5.4
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
---
apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an add-on, you should comment out this line.
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Heapster
  name: heapster
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8082
  selector:
    k8s-app: heapster

heapster.yaml

heapster 启动参数说明：

• inClusterConfig - Use kube config in service accounts associated with Heapster's namespace. (default: true)
• kubeletPort - kubelet port to use (default: 10255)
• kubeletHttps - whether to use https to connect to kubelets (default: false)
• insecure - whether to trust Kubernetes certificates (default: false)
• auth - client auth file to use. Set auth if the service accounts are not usable.
• useServiceAccount - whether to use the service account token if one is mounted at /var/run/secrets/kubernetes.io/serviceaccount/token (default: false)

使用： kubectl apply -f heapster.yaml 部署

部署后查看log日志，发现一直提示“403 Forbidden”, response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"

[root@node01 heapster-yaml]# kubectl logs --namespace=kube-system heapster-868d5cd7f-855k7
I0912 14:36:48.175115       1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
I0912 14:36:48.175174       1 heapster.go:79] Heapster version v1.5.4
I0912 14:36:48.175366       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0912 14:36:48.175393       1 configs.go:62] Using kubelet port 10250
I0912 14:36:48.185828       1 heapster.go:202] Starting with Metric Sink
I0912 14:36:48.199515       1 heapster.go:112] Starting heapster on port 8082
E0912 14:37:05.000327       1 kubelet.go:288] node node01 is not ready
E0912 14:38:05.016044       1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0912 14:38:25.000643       1 manager.go:152] Failed to get all responses in time (got 0/1)
E0912 14:39:05.008534       1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0912 14:39:25.000674       1 manager.go:152] Failed to get all responses in time (got 0/1)
E0912 14:40:05.009955       1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0912 14:40:25.001231       1 manager.go:152] Failed to get all responses in time (got 0/1)
E0912 14:41:05.017198       1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0912 14:41:25.000846       1 manager.go:152] Failed to get all responses in time (got 0/1)

查看ClusterRole: system:heapster的权限，发现的确没有针对Resource: nodes/stats 的create权限

[root@node01 heapster-yaml]# kubectl describe clusterrole system:heapster
Name:         system:heapster
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{"rbac.authorization.kubernetes.io/autoupdate":"true"},"lab...
              rbac.authorization.kubernetes.io/autoupdate=true
PolicyRule:
  Resources               Non-Resource URLs  Resource Names  Verbs
  ---------               -----------------  --------------  -----
  events                  []                 []              [get list watch]
  namespaces              []                 []              [get list watch]
  nodes                   []                 []              [get list watch]
  pods                    []                 []              [get list watch]
  deployments.extensions  []                 []              [get list watch]

修改ClusterRole: system:heapster的权限:

1. 查看system:heapster yaml格式, 保存为 heapster-clusterrole.yaml

yaml

[root@node01 heapster-yaml]# kubectl get clusterrole system:heapster -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{"rbac.authorization.kubernetes.io/autoupdate":"true"},"labels":{"kubernetes.io/bootstrapping":"rbac-defaults"},"name":"system:heapster","namespace":""},"rules":[{"apiGroups":[""],"resources":["events","namespaces","nodes","pods","nodes/stats"],"verbs":["create","get","list","watch"]},{"apiGroups":["extensions"],"resources":["deployments"],"verbs":["get","list","watch"]}]}
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2018-08-26T02:26:14Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:heapster
  resourceVersion: ""
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Aheapster
  uid: 67ef3689-a8d7-11e8-a891-000c29b52823
rules:
- apiGroups:
  - ""
  resources:
  - events
  - namespaces
  - nodes
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - deployments
  verbs:
  - get
  - list
  - watch

2. 添加Resource: nodes/stats的create权限，并执行 kubectl apply -f heapster-clusterrole.yaml

yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:heapster
rules:
- apiGroups:
  - ""
  resources:
  - events
  - namespaces
  - nodes
  - pods
  - nodes/stats
  verbs:
  - create
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - deployments
  verbs:
  - get
  - list
  - watch

3. 删除heapster重新部署

　　kubectl delete -f heapster.yaml

　　kubectl apply -f heapster.yaml

4. 重新部署后，查看log没有任何报错了

[root@node01 heapster-yaml]# kubectl logs --namespace=kube-system heapster-868d5cd7f-8zgxq
I0912 15:02:38.926068       1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
I0912 15:02:38.926128       1 heapster.go:79] Heapster version v1.5.4
I0912 15:02:38.926565       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0912 15:02:38.926647       1 configs.go:62] Using kubelet port 10250
I0912 15:02:38.938333       1 heapster.go:202] Starting with Metric Sink
I0912 15:02:38.949215       1 heapster.go:112] Starting heapster on port 8082

5. 功能测试

部署前：

[root@node01 heapster-yaml]# kubectl top pod
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

部署后：

[root@node01 tomcat-mysql-yaml]# kubectl top pod
NAME                      CPU(cores)   MEMORY(bytes)
lxcfs-j8bpd               0m           4Mi
mysql-85c85c5668-fr6j7    0m           458Mi
tomcat-78c9778858-424gc   1m           93Mi
tomcat-78c9778858-42xz4   1m           97Mi
tomcat-78c9778858-6t7cd   1m           93Mi            

相关链接：

https://github.com/kubernetes/heapster

https://github.com/kubernetes-incubator/metrics-server

https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/