问题

通过minikube启动k8s集群后，我们想安装启动k8s的dashboard插件，执行minikube dashboard后，一直卡住直到提示验证失败。

查看pod状态，我们看到kubernetes-dashboard下的2个pod实例的状态都是ImagePullBackOff

[dev ~]# kubectl get pod --all-namespaces

NAMESPACE              NAME                                         READY   STATUS             RESTARTS       AGE

kube-system            coredns-76fccbbb6b-wbx9v                     1/1     Running            0              110m

kube-system            etcd-minikube                                1/1     Running            0              110m

kube-system            kube-apiserver-minikube                      1/1     Running            0              110m

kube-system            kube-controller-manager-minikube             1/1     Running            0              110m

kube-system            kube-proxy-grk7h                             1/1     Running            0              110m

kube-system            kube-scheduler-minikube                      1/1     Running            0              110m

kube-system            storage-provisioner                          1/1     Running            1 (110m ago)   110m

kubernetes-dashboard   dashboard-metrics-scraper-5d59dccf9b-pzt8r   0/1     ImagePullBackOff   0              108m

kubernetes-dashboard   kubernetes-dashboard-7779f9b69b-zz2wm        0/1     ImagePullBackOff   0              108m

我们查看具体的失败原因可以看到：

查看 dashboard-metrics-scraper 失败原因

[dev ~]# kubectl describe pod dashboard-metrics-scraper-5d59dccf9b-pzt8r --namespace=kubernetes-dashboard

Events:

  Type     Reason   Age                     From     Message

  ----     ------   ----                    ----     -------

  Normal   Pulling  34m (x19 over 109m)     kubelet  Pulling image "docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c"

  Warning  Failed   24m (x21 over 109m)     kubelet  Failed to pull image "docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c": Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.49.1:53: server misbehaving

  Normal   BackOff  4m50s (x442 over 109m)  kubelet  Back-off pulling image "docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c"

  Warning  Failed   4m37s (x443 over 109m)  kubelet  Error: ImagePullBackOff

查看 kubernetes-dashboard 失败原因

[dev ~]# kubectl describe pod kubernetes-dashboard-7779f9b69b-zz2wm --namespace=kubernetes-dashboard

Events:

  Type     Reason   Age                   From     Message

  ----     ------   ----                  ----     -------

  Normal   Pulling  36m (x20 over 116m)   kubelet  Pulling image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93"

  Warning  Failed   31m (x21 over 116m)   kubelet  Failed to pull image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93": Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.49.1:53: server misbehaving

  Normal   BackOff  96s (x486 over 116m)  kubelet  Back-off pulling image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93"

  Warning  Failed   82s (x487 over 116m)  kubelet  Error: ImagePullBackOff

2个pod实例启动失败原因都是dashboard所需镜像pull失败导致。

解决办法

手动pull镜像，并设置tag

拉取失败的总共有2个镜像：

docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c
docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93

因为pull镜像失败的网络问题无法解决，我们只能pull国内镜像，并给镜像打上原始镜像名称的标记。

进入minikube的环境

我们需要先进入minikube的环境，如果直接在host环境中拉取镜像，minikube是获取不到的。

minikube ssh

进入环境后我们可以执行 docker image ls -a查看当前环境的镜像列表，其中没有包含所缺失的镜像。

拉取国内镜像，并打上原始镜像的tag

# 手动pull metrics-scraper:v1.0.8

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.8

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.8 docker.io/kubernetesui/metrics-scraper:v1.0.8

# 手动pull dashboard:v2.7.0

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.7.0

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.7.0 docker.io/kubernetesui/dashboard:v2.7.0

查看现有镜像

[dev ~]# minikube ssh

docker@minikube:~$ docker image ls -a

REPOSITORY                                                                    TAG        IMAGE ID       CREATED         SIZE

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.32.0    c2e17b8d0f4a   5 months ago    97MB

registry.k8s.io/kube-apiserver                                                v1.32.0    c2e17b8d0f4a   5 months ago    97MB

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.32.0    8cab3d2a8bd0   5 months ago    89.7MB

registry.k8s.io/kube-controller-manager                                       v1.32.0    8cab3d2a8bd0   5 months ago    89.7MB

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.32.0    a389e107f4ff   5 months ago    69.6MB

registry.k8s.io/kube-scheduler                                                v1.32.0    a389e107f4ff   5 months ago    69.6MB

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.32.0    040f9f8aac8c   5 months ago    94MB

registry.k8s.io/kube-proxy                                                    v1.32.0    040f9f8aac8c   5 months ago    94MB

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen      v1.4.4     a62eeff05ba5   7 months ago    63.8MB

registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.5.16-0   a9e7e6b294ba   8 months ago    150MB

registry.k8s.io/etcd                                                          3.5.16-0   a9e7e6b294ba   8 months ago    150MB

registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   v1.11.3    c69fa2e9cbf5   9 months ago    61.8MB

registry.k8s.io/coredns/coredns                                               v1.11.3    c69fa2e9cbf5   9 months ago    61.8MB

registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.10       873ed7510279   12 months ago   736kB

registry.k8s.io/pause                                                         3.10       873ed7510279   12 months ago   736kB

kubernetesui/dashboard                                                        v2.7.0     07655ddf2eeb   2 years ago     246MB

registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard                 v2.7.0     07655ddf2eeb   2 years ago     246MB

registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper           v1.0.8     115053965e86   2 years ago     43.8MB

kubernetesui/metrics-scraper                                                  v1.0.8     115053965e86   2 years ago     43.8MB

gcr.io/k8s-minikube/storage-provisioner                                       v5         6e38f40d628d   4 years ago     31.5MB

registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner       v5         6e38f40d628d   4 years ago     31.5MB

缺失的镜像列表都弄好了。

停掉当前的dashboard

这个步骤可以通过kubectl在外部删掉启动失败的2个pod，或者在minikube内部手动运行kubectl删掉都可以。

在外部删掉

kubectl delete pod dashboard-metrics-scraper-5d59dccf9b-pzt8r --namespace kubernetes-dashboard

kubectl delete pod kubernetes-dashboard-7779f9b69b-zz2wm --namespace kubernetes-dashboard

进入minikube环境，手动运行kubectl 删掉dashboard的pod

[dev ~]# minikube ssh

docker@minikube:~$ sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.32.0/kubectl delete -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml

namespace "kubernetes-dashboard" deleted

clusterrole.rbac.authorization.k8s.io "kubernetes-dashboard" deleted

clusterrolebinding.rbac.authorization.k8s.io "kubernetes-dashboard" deleted

configmap "kubernetes-dashboard-settings" deleted

deployment.apps "dashboard-metrics-scraper" deleted

deployment.apps "kubernetes-dashboard" deleted

role.rbac.authorization.k8s.io "kubernetes-dashboard" deleted

rolebinding.rbac.authorization.k8s.io "kubernetes-dashboard" deleted

serviceaccount "kubernetes-dashboard" deleted

secret "kubernetes-dashboard-certs" deleted

secret "kubernetes-dashboard-csrf" deleted

secret "kubernetes-dashboard-key-holder" deleted

service "kubernetes-dashboard" deleted

service "dashboard-metrics-scraper" deleted

编辑dashboard的deployment配置文件，修改其中的镜像地址

我们需要先进入minikube环境

minikube ssh

查看配置

sudo cat /etc/kubernetes/addons/dashboard-dp.yaml

配置太长了，只截取关注的这部分：

kind: Deployment

apiVersion: apps/v1

spec:

    spec:

      containers:

        - name: dashboard-metrics-scraper

          image: docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c

---

kind: Deployment

apiVersion: apps/v1

spec:

    spec:

      containers:

        - name: kubernetes-dashboard

          # WARNING: This must match pkg/minikube/bootstrapper/images/images.go

          image: docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93

          ports:

            - containerPort: 9090

              protocol: TCP

可以看到，pull失败的镜像地址，就是在这个配置文件里配置，接下来，我们需要修改这个配置文件，将image地址中的sha256删掉，并添加镜像pull规则：

    spec:

      containers:

        - name: dashboard-metrics-scraper

          image: docker.io/kubernetesui/metrics-scraper:v1.0.8

          imagePullPolicy: IfNotPresent

----

    spec:

      containers:

        - name: kubernetes-dashboard

          # WARNING: This must match pkg/minikube/bootstrapper/images/images.go

          image: docker.io/kubernetesui/dashboard:v2.7.0

          imagePullPolicy: IfNotPresent

主要是2个字段：

image 将@和后面的字符串删掉，因为我们国内的镜像这个sha256和配置中的看起来并不一样。
imagePullPolicy 设置为 IfNotPresent，表示镜像存在就不会再pull一次。

手动启动dashboard

docker@minikube:~$ sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.32.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml

namespace/kubernetes-dashboard created

clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created

clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created

configmap/kubernetes-dashboard-settings created

Warning: spec.template.metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: non-functional in v1.27+; use the "seccompProfile" field instead

deployment.apps/dashboard-metrics-scraper created

deployment.apps/kubernetes-dashboard created

role.rbac.authorization.k8s.io/kubernetes-dashboard created

rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created

serviceaccount/kubernetes-dashboard created

secret/kubernetes-dashboard-certs created

secret/kubernetes-dashboard-csrf created

secret/kubernetes-dashboard-key-holder created

service/kubernetes-dashboard created

service/dashboard-metrics-scraper created

然后退出minikube环境，查看dashboard的pod运行状态：

[dev ~]# kubectl get pod -A

NAMESPACE              NAME                                          READY   STATUS              RESTARTS         AGE

kube-system            coredns-76fccbbb6b-vtdbf                      1/1     Running             5 (12m ago)      13h

kube-system            etcd-minikube                                 1/1     Running             10 (12m ago)     2d

kube-system            kube-apiserver-minikube                       1/1     Running             60 (13m ago)     2d

kube-system            kube-controller-manager-minikube              1/1     Running             34 (9m42s ago)   2d

kube-system            kube-proxy-4n788                              1/1     Running             1 (12m ago)      2d

kube-system            kube-scheduler-minikube                       1/1     Running             12 (12m ago)     2d

kubernetes-dashboard   dashboard-metrics-scraper-545b5664b6-mcrzh    1/1     Running             0                5m27s

kubernetes-dashboard   kubernetes-dashboard-7f67ccd984-ql4cn         1/1     Running             0                5m27s

minikube dashboard ImagePullBackOff 失败问题的解决方法的更多相关文章

phpStudy启动失败时的解决方法
phpStudy启动失败时的解决方法 phpStudy启动失败,原因一是防火墙拦截,二是80端口已经被别的程序占用,如IIS,迅雷等:三是没有安装VC9运行库,php和apache都是VC9编译.解决 ...
mysql创建外链失败1005错误解决方法
mysql创建外链失败1005错误解决方法错误号:1005错误信息:Can't create table 'webDB.#sql-397_61df' (errno: 150)解决方法错误原因有四: ...
【常见的SQL Server连接失败错误以及解决方法】
[常见的SQL Server连接失败错误以及解决方法] http://blog.csdn.net/feixianxxx/article/details/5523922 ADO连接SQL Server ...
pip安装包失败的常见解决方法
pip安装包失败的常见解决方法方法一当安装包失效的时候可以去网站下载whl文件.下载的时候应该先查看电脑上 python 的版本. 地址:https://www.lfd.uci.edu/~gohl ...
“访问 IIS 元数据库失败”错误的解决方法
1.依次点击“开始”-“运行”. 2.在“运行”栏内输入 “C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -i ...
hanlp在Python环境中的安装失败后的解决方法
Hanlp是由一系列模型与算法组成的javag工具包,目标是普及自然语言处理再生环境中的应用.有很多人在安装hanlp的时候会遇到安装失败的情况,下面就是某大神的分享的在python环境中安装失败的解 ...
【Gradle】配置中引用的jar包版本后面自动加冒号导致引入jar包失败的问题/gradle中引用jar包版本不一致的问题/gradle中引用jar失败的问题解决方法
idea中 gradle中引用jar包,版本后面默认加:的问题 gradle中引用jar包版本不一致的问题 gradle中引用jar失败的问题如上题目所示,三个问题其实都是同一样的简单又恶心,因为 ...
安装VMTools失败的三类解决方法（Windows、Linux、MacOs）
前言写这篇笔记的原因,是前几天在虚拟机 Vmware 中重新安装了几个操作系统,突然发现 VMTools 这个工具成了一个特殊的问题,以前还没有发现,因为通常它就给你自动安装了.但是大多数时候也是需 ...
BurpSuite下提示embedded browser initialisation failed（嵌入式浏览器初始化失败）的解决方法
BurpSuite可谓是渗透测试过程经常使用的神器之一,但使用中经常会碰到奇奇怪怪的问题,比如有时抓http包,发送到Repeater(中继器,也叫重发器)模块后,在右边Render模块下,却无法看到 ...
wordpress定时发布失败Missed Schedule解决方法
今天一位客户的网站设置了文章定时发布但却没有发布成功,提示Missed Schedule错误,把所有的插件都暂停了再设置定时发布也不行,那就只能寻找其他的解决方案了.ytkah查看一下有没相关修复插件 ...

随机推荐

Stream4Graph：动态图上的增量计算
作者:张奇众所周知,当我们需要对数据做关联性分析的时候,一般会采用表连接(SQL join)的方式完成.但是SQL join时的笛卡尔积计算需要维护大量的中间结果,从而对整体的数据分析性能带来巨大影 ...
linux服务问题传文件连不上问题远程问题等
通过iptables相关命令实现防火墙的打开和关闭 1.首先可以在打开的终端使用iptables --help查看帮助使用命令: 2.查看防火墙状态:service iptables status(此 ...
万字解析Golang的map实现原理
0.引言相信大家对Map这个数据结构都不陌生,像C++的map.Java的HashMap.各个语言的底层实现各有不同,在本篇博客中,我将分享个人对Go的map实现的理解,以及深入源码进行分析,相信耐 ...
墨者学院SQL注入（MySQL）的总结：
我们打开靶机在这个界面我们可以看见在密码的下面有一个停机公告,我们点进去,会发现可以发现这个公告是存在id的,又可以会出现漏洞.所以我们尝试着在id=1的后面加上and 1=1 我们可以 ...
Qt使用QAudioInput、QAudioOutput实现局域网的音频通话
Qt使用QAudioInput.QAudioOutput实现局域网的音频通话本文旨在介绍一下用Qt来实现局域网音频通话功能文章目录 Qt使用QAudioInput.QAudioOutput实现局域 ...
[Qt 基础-01] QPushButton
QPushButton 简介 QPushButton是一个很常用的一个按钮控件,主要用于创建一个可按压的按键.它显示了一个文本和一个图标.另外,你也可以在创建时,指定一个快捷键. 基本用法 1. 创 ...
go 遍历修改切片数据
package main import "fmt" type good struct { id int64 sum int64 } func main() { good1 := g ...
go-ini 中文文档
简介地表最强大.最方便和最流行的 Go 语言 INI 文件操作库灵活的数据源不光光可以从文件读取配置,还支持 []byte 类型的纯数据读取和基于 io.ReadCloser 的流式读取 ...
ASP.NET Core 模型验证消息的本地化新姿势
最近在研究系统本地化的问题,不可避免要实现模型类的验证消息本地化.毕竟这些错误消息是要返回给用户的. 疑问产生在MVC模型下,我们会使用模型类对请求参数进行绑定和验证.举个例子: public cl ...
Arrays.asList() 详解
[1. 要点] 该方法是将数组转化成List集合的方法. List list = Arrays.asList("a","b","c"); 注 ...

minikube dashboard ImagePullBackOff 失败问题的解决方法

问题