kubernetes使用metrics-server进行资源监控

kubernetes资源监控

1. 查看集群资源状况

·k8s集群的master节点一般不会跑业务容器·

 kubectl get cs                   #查看master资源状态
 kubectl get node                 #查看节点
 kubectl cluster-info             #查看集群状态
 kubectl  describe pods [pod名]   #查看各类资源状态
 kubectl get pods -o wide         #查看更新信息

2.监控集群资源利用率【metrics-server安装使用】

#可以通过kubectl 来查看资源利用率，但是该命令需要 "heapster" 调用资源才行，如果没有提供资源则会报错如下：
[root@k8s-master1 ~]# kubectl top node k8s-node1
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

#heapster组件已经被弃用,现在使用的是： metrics-server + cAdvisor聚合器来提供资源。
#cAdvisor 已经内置于kubelet组件中。所以需要安装metrics-server

metrics-server架构示意图：

wget https://www.chenleilei.net/soft/k8s/metrics-server.zip
#安装metrics-server 或 上传相关包
git clone https://github.com/kubernetes-incubator/metrics-server
cd metrics-server/
vim metrics-server-deployment.yaml
31,32行改为：
- name: metrics-server
  image: lizhenliang/metrics-server-amd64:v0.3.1

33行 - --secure-port=4443 下方加入以下两行内容[从github中下载需要添加这两处，从博客下载这里就不用改了已经改好了]：
 - --kubelet-insecure-tls
 - --kubelet-preferred-address-types=InternalIP  [这里意思是讲链接方式改为内部IP链接]

[注意：metrics-server是通过主机名来区分主机的,所以说必须要配置 host解析，metrics-server才能正确的采集到目标。]

修改配置文件 vim /etc/kubernetes/manifests/kube-apiserver.yaml
大约20行 - --enable-admission-plugins=NodeRestriction的下方添加1行代码：
- --enable-aggregator-routing=true

root@k8s-master1 metrics-server]# ll
otal 28
rw-r--r-- 1 root root  397 Mar 15 21:04 aggregated-metrics-reader.yaml
rw-r--r-- 1 root root  303 Mar 15 21:04 auth-delegator.yaml
rw-r--r-- 1 root root  324 Mar 15 21:04 auth-reader.yaml
rw-r--r-- 1 root root  298 Mar 15 21:04 metrics-apiservice.yaml           #将merics-server注册到k8s的api中
rw-r--r-- 1 root root 1277 Mar 27 15:57 metrics-server-deployment.yaml    #部署metrics-server
rw-r--r-- 1 root root  297 Mar 15 21:04 metrics-server-service.yaml
rw-r--r-- 1 root root  532 Mar 15 21:04 resource-reader.yaml

修改完毕后直接安装：
[root@k8s-master1 metrics-server]# kubectl apply -f .
提示：
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

# 重启 kubelet：
systemctl restart kubelet

检查安装：
[root@k8s-master1 metrics-server]# kubectl get pods -n kube-system
NAME                                  READY   STATUS    RESTARTS   AGE
coredns-7ff77c879f-kvbbh              1/1     Running   0          6d4h
coredns-7ff77c879f-lqk9q              1/1     Running   5          6d4h
etcd-k8s-master1                      1/1     Running   2          8d
kube-apiserver-k8s-master1            1/1     Running   2          8d
kube-controller-manager-k8s-master1   1/1     Running   2          8d
kube-flannel-ds-amd64-8gssm           1/1     Running   5          8d
kube-flannel-ds-amd64-gtpwc           1/1     Running   2          8d
kube-flannel-ds-amd64-mx4jx           1/1     Running   4          8d
kube-proxy-kzwft                      1/1     Running   5          8d
kube-proxy-rgjmf                      1/1     Running   2          8d
kube-proxy-vhdpp                      1/1     Running   4          8d
kube-scheduler-k8s-master1            1/1     Running   2          8d
metrics-server-5667498b7d-lmbtr       1/1     Running   0          56s    <------metrics-server安装完毕。

查看：
[root@k8s-master1 metrics-server]# kubectl get apiservice
看到这行：
v1beta1.metrics.k8s.io      kube-system/metrics-server   True     17s   #启动成功

验证配置kubectl top配置：
[root@k8s-master1 metrics-server]# kubectl top pods
NAME                    CPU(cores)   MEMORY(bytes)
nginx-f89759699-6jfdp   0m           2Mi        

#最开始执行这条命令报错，现在正常：
[root@k8s-master1 metrics-server]# kubectl top node k8s-node1
NAME        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%     #<---- 成功
k8s-node1   42m          2%     383Mi           13% 

[root@k8s-master1 metrics-server]# kubectl top node
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-master1   94m          4%     809Mi           27%       #<---- 成功
k8s-node1     48m          2%     385Mi           13%
k8s-node2     26m          1%     386Mi           13%     

#资源利用率排序
[root@k8s-master1 metrics-server]# kubectl top pods -l app=nginx --sort-by=memory
NAME                    CPU(cores)   MEMORY(bytes)
nginx-f89759699-6jfdp   0m           2Mi

3. 问题排查

[root@k8s-master1 ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}   

如果kubectl get cs查询失败，可能是apiserver出现了问题。

#apiserver 正常可以通过该命令查看状态
[root@k8s-master1 ~]# kubectl cluster-info
Kubernetes master is running at https://10.0.0.63:6443
KubeDNS is running at https://10.0.0.63:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

# 通过dump.log查看集群问题
[root@k8s-master1 ~]# kubectl cluster-info
Kubernetes master is running at https://10.0.0.63:6443
KubeDNS is running at https://10.0.0.63:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

#状态信息写入到文件：
kubectl cluster-info dump >a.txt

#kubectl describe pod [pod名]

#实时观察pod动态
kubectl get pods -w   [删除创建pod，这条命令里都会输出出来，并可以显示整理流程]