问题

通过minikube启动k8s集群后，我们想安装启动k8s的dashboard插件，执行minikube dashboard后，一直卡住直到提示验证失败。

查看pod状态，我们看到kubernetes-dashboard下的2个pod实例的状态都是ImagePullBackOff

[dev ~]# kubectl get pod --all-namespaces

NAMESPACE              NAME                                         READY   STATUS             RESTARTS       AGE

kube-system            coredns-76fccbbb6b-wbx9v                     1/1     Running            0              110m

kube-system            etcd-minikube                                1/1     Running            0              110m

kube-system            kube-apiserver-minikube                      1/1     Running            0              110m

kube-system            kube-controller-manager-minikube             1/1     Running            0              110m

kube-system            kube-proxy-grk7h                             1/1     Running            0              110m

kube-system            kube-scheduler-minikube                      1/1     Running            0              110m

kube-system            storage-provisioner                          1/1     Running            1 (110m ago)   110m

kubernetes-dashboard   dashboard-metrics-scraper-5d59dccf9b-pzt8r   0/1     ImagePullBackOff   0              108m

kubernetes-dashboard   kubernetes-dashboard-7779f9b69b-zz2wm        0/1     ImagePullBackOff   0              108m

我们查看具体的失败原因可以看到：

查看 dashboard-metrics-scraper 失败原因

[dev ~]# kubectl describe pod dashboard-metrics-scraper-5d59dccf9b-pzt8r --namespace=kubernetes-dashboard

Events:

  Type     Reason   Age                     From     Message

  ----     ------   ----                    ----     -------

  Normal   Pulling  34m (x19 over 109m)     kubelet  Pulling image "docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c"

  Warning  Failed   24m (x21 over 109m)     kubelet  Failed to pull image "docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c": Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.49.1:53: server misbehaving

  Normal   BackOff  4m50s (x442 over 109m)  kubelet  Back-off pulling image "docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c"

  Warning  Failed   4m37s (x443 over 109m)  kubelet  Error: ImagePullBackOff

查看 kubernetes-dashboard 失败原因

[dev ~]# kubectl describe pod kubernetes-dashboard-7779f9b69b-zz2wm --namespace=kubernetes-dashboard

Events:

  Type     Reason   Age                   From     Message

  ----     ------   ----                  ----     -------

  Normal   Pulling  36m (x20 over 116m)   kubelet  Pulling image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93"

  Warning  Failed   31m (x21 over 116m)   kubelet  Failed to pull image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93": Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.49.1:53: server misbehaving

  Normal   BackOff  96s (x486 over 116m)  kubelet  Back-off pulling image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93"

  Warning  Failed   82s (x487 over 116m)  kubelet  Error: ImagePullBackOff

2个pod实例启动失败原因都是dashboard所需镜像pull失败导致。

解决办法

手动pull镜像，并设置tag

拉取失败的总共有2个镜像：

docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c
docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93

因为pull镜像失败的网络问题无法解决，我们只能pull国内镜像，并给镜像打上原始镜像名称的标记。

进入minikube的环境

我们需要先进入minikube的环境，如果直接在host环境中拉取镜像，minikube是获取不到的。

minikube ssh

进入环境后我们可以执行 docker image ls -a查看当前环境的镜像列表，其中没有包含所缺失的镜像。

拉取国内镜像，并打上原始镜像的tag

# 手动pull metrics-scraper:v1.0.8

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.8

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.8 docker.io/kubernetesui/metrics-scraper:v1.0.8

# 手动pull dashboard:v2.7.0

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.7.0

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.7.0 docker.io/kubernetesui/dashboard:v2.7.0

查看现有镜像

[dev ~]# minikube ssh

docker@minikube:~$ docker image ls -a

REPOSITORY                                                                    TAG        IMAGE ID       CREATED         SIZE

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.32.0    c2e17b8d0f4a   5 months ago    97MB

registry.k8s.io/kube-apiserver                                                v1.32.0    c2e17b8d0f4a   5 months ago    97MB

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.32.0    8cab3d2a8bd0   5 months ago    89.7MB

registry.k8s.io/kube-controller-manager                                       v1.32.0    8cab3d2a8bd0   5 months ago    89.7MB

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.32.0    a389e107f4ff   5 months ago    69.6MB

registry.k8s.io/kube-scheduler                                                v1.32.0    a389e107f4ff   5 months ago    69.6MB

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.32.0    040f9f8aac8c   5 months ago    94MB

registry.k8s.io/kube-proxy                                                    v1.32.0    040f9f8aac8c   5 months ago    94MB

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen      v1.4.4     a62eeff05ba5   7 months ago    63.8MB

registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.5.16-0   a9e7e6b294ba   8 months ago    150MB

registry.k8s.io/etcd                                                          3.5.16-0   a9e7e6b294ba   8 months ago    150MB

registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   v1.11.3    c69fa2e9cbf5   9 months ago    61.8MB

registry.k8s.io/coredns/coredns                                               v1.11.3    c69fa2e9cbf5   9 months ago    61.8MB

registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.10       873ed7510279   12 months ago   736kB

registry.k8s.io/pause                                                         3.10       873ed7510279   12 months ago   736kB

kubernetesui/dashboard                                                        v2.7.0     07655ddf2eeb   2 years ago     246MB

registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard                 v2.7.0     07655ddf2eeb   2 years ago     246MB

registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper           v1.0.8     115053965e86   2 years ago     43.8MB

kubernetesui/metrics-scraper                                                  v1.0.8     115053965e86   2 years ago     43.8MB

gcr.io/k8s-minikube/storage-provisioner                                       v5         6e38f40d628d   4 years ago     31.5MB

registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner       v5         6e38f40d628d   4 years ago     31.5MB

缺失的镜像列表都弄好了。

停掉当前的dashboard

这个步骤可以通过kubectl在外部删掉启动失败的2个pod，或者在minikube内部手动运行kubectl删掉都可以。

在外部删掉

kubectl delete pod dashboard-metrics-scraper-5d59dccf9b-pzt8r --namespace kubernetes-dashboard

kubectl delete pod kubernetes-dashboard-7779f9b69b-zz2wm --namespace kubernetes-dashboard

进入minikube环境，手动运行kubectl 删掉dashboard的pod

[dev ~]# minikube ssh

docker@minikube:~$ sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.32.0/kubectl delete -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml

namespace "kubernetes-dashboard" deleted

clusterrole.rbac.authorization.k8s.io "kubernetes-dashboard" deleted

clusterrolebinding.rbac.authorization.k8s.io "kubernetes-dashboard" deleted

configmap "kubernetes-dashboard-settings" deleted

deployment.apps "dashboard-metrics-scraper" deleted

deployment.apps "kubernetes-dashboard" deleted

role.rbac.authorization.k8s.io "kubernetes-dashboard" deleted

rolebinding.rbac.authorization.k8s.io "kubernetes-dashboard" deleted

serviceaccount "kubernetes-dashboard" deleted

secret "kubernetes-dashboard-certs" deleted

secret "kubernetes-dashboard-csrf" deleted

secret "kubernetes-dashboard-key-holder" deleted

service "kubernetes-dashboard" deleted

service "dashboard-metrics-scraper" deleted

编辑dashboard的deployment配置文件，修改其中的镜像地址

我们需要先进入minikube环境

minikube ssh

查看配置

sudo cat /etc/kubernetes/addons/dashboard-dp.yaml

配置太长了，只截取关注的这部分：

kind: Deployment

apiVersion: apps/v1

spec:

    spec:

      containers:

        - name: dashboard-metrics-scraper

          image: docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c

---

kind: Deployment

apiVersion: apps/v1

spec:

    spec:

      containers:

        - name: kubernetes-dashboard

          # WARNING: This must match pkg/minikube/bootstrapper/images/images.go

          image: docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93

          ports:

            - containerPort: 9090

              protocol: TCP

可以看到，pull失败的镜像地址，就是在这个配置文件里配置，接下来，我们需要修改这个配置文件，将image地址中的sha256删掉，并添加镜像pull规则：

    spec:

      containers:

        - name: dashboard-metrics-scraper

          image: docker.io/kubernetesui/metrics-scraper:v1.0.8

          imagePullPolicy: IfNotPresent

----

    spec:

      containers:

        - name: kubernetes-dashboard

          # WARNING: This must match pkg/minikube/bootstrapper/images/images.go

          image: docker.io/kubernetesui/dashboard:v2.7.0

          imagePullPolicy: IfNotPresent

主要是2个字段：

image 将@和后面的字符串删掉，因为我们国内的镜像这个sha256和配置中的看起来并不一样。
imagePullPolicy 设置为 IfNotPresent，表示镜像存在就不会再pull一次。

手动启动dashboard

docker@minikube:~$ sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.32.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml

namespace/kubernetes-dashboard created

clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created

clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created

configmap/kubernetes-dashboard-settings created

Warning: spec.template.metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: non-functional in v1.27+; use the "seccompProfile" field instead

deployment.apps/dashboard-metrics-scraper created

deployment.apps/kubernetes-dashboard created

role.rbac.authorization.k8s.io/kubernetes-dashboard created

rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created

serviceaccount/kubernetes-dashboard created

secret/kubernetes-dashboard-certs created

secret/kubernetes-dashboard-csrf created

secret/kubernetes-dashboard-key-holder created

service/kubernetes-dashboard created

service/dashboard-metrics-scraper created

然后退出minikube环境，查看dashboard的pod运行状态：

[dev ~]# kubectl get pod -A

NAMESPACE              NAME                                          READY   STATUS              RESTARTS         AGE

kube-system            coredns-76fccbbb6b-vtdbf                      1/1     Running             5 (12m ago)      13h

kube-system            etcd-minikube                                 1/1     Running             10 (12m ago)     2d

kube-system            kube-apiserver-minikube                       1/1     Running             60 (13m ago)     2d

kube-system            kube-controller-manager-minikube              1/1     Running             34 (9m42s ago)   2d

kube-system            kube-proxy-4n788                              1/1     Running             1 (12m ago)      2d

kube-system            kube-scheduler-minikube                       1/1     Running             12 (12m ago)     2d

kubernetes-dashboard   dashboard-metrics-scraper-545b5664b6-mcrzh    1/1     Running             0                5m27s

kubernetes-dashboard   kubernetes-dashboard-7f67ccd984-ql4cn         1/1     Running             0                5m27s

minikube dashboard ImagePullBackOff 失败问题的解决方法的更多相关文章

phpStudy启动失败时的解决方法
phpStudy启动失败时的解决方法 phpStudy启动失败,原因一是防火墙拦截,二是80端口已经被别的程序占用,如IIS,迅雷等:三是没有安装VC9运行库,php和apache都是VC9编译.解决 ...
mysql创建外链失败1005错误解决方法
mysql创建外链失败1005错误解决方法错误号:1005错误信息:Can't create table 'webDB.#sql-397_61df' (errno: 150)解决方法错误原因有四: ...
【常见的SQL Server连接失败错误以及解决方法】
[常见的SQL Server连接失败错误以及解决方法] http://blog.csdn.net/feixianxxx/article/details/5523922 ADO连接SQL Server ...
pip安装包失败的常见解决方法
pip安装包失败的常见解决方法方法一当安装包失效的时候可以去网站下载whl文件.下载的时候应该先查看电脑上 python 的版本. 地址:https://www.lfd.uci.edu/~gohl ...
“访问 IIS 元数据库失败”错误的解决方法
1.依次点击“开始”-“运行”. 2.在“运行”栏内输入 “C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -i ...
hanlp在Python环境中的安装失败后的解决方法
Hanlp是由一系列模型与算法组成的javag工具包,目标是普及自然语言处理再生环境中的应用.有很多人在安装hanlp的时候会遇到安装失败的情况,下面就是某大神的分享的在python环境中安装失败的解 ...
【Gradle】配置中引用的jar包版本后面自动加冒号导致引入jar包失败的问题/gradle中引用jar包版本不一致的问题/gradle中引用jar失败的问题解决方法
idea中 gradle中引用jar包,版本后面默认加:的问题 gradle中引用jar包版本不一致的问题 gradle中引用jar失败的问题如上题目所示,三个问题其实都是同一样的简单又恶心,因为 ...
安装VMTools失败的三类解决方法（Windows、Linux、MacOs）
前言写这篇笔记的原因,是前几天在虚拟机 Vmware 中重新安装了几个操作系统,突然发现 VMTools 这个工具成了一个特殊的问题,以前还没有发现,因为通常它就给你自动安装了.但是大多数时候也是需 ...
BurpSuite下提示embedded browser initialisation failed（嵌入式浏览器初始化失败）的解决方法
BurpSuite可谓是渗透测试过程经常使用的神器之一,但使用中经常会碰到奇奇怪怪的问题,比如有时抓http包,发送到Repeater(中继器,也叫重发器)模块后,在右边Render模块下,却无法看到 ...
wordpress定时发布失败Missed Schedule解决方法
今天一位客户的网站设置了文章定时发布但却没有发布成功,提示Missed Schedule错误,把所有的插件都暂停了再设置定时发布也不行,那就只能寻找其他的解决方案了.ytkah查看一下有没相关修复插件 ...

随机推荐

Scala重写构造函数
package com.wyh.scala.classData object Demo4Class { def main(args: Array[String]): Unit = { val stu ...
Codeforces Round 1006 (Div. 3) 补题+题解
A. New World, New Me, New Array 贪心的想每次都赋值一个 \(p\) 如果正好和为 \(k\) 则答案就是 \(k/p\) ,否则是 \(k/p+1\). #includ ...
【C++】开源：ImGui图形用户界面库配置与使用
项目介绍项目Github地址:https://github.com/ocornut/imgui Dear ImGui (ImGui) 是一个开源的.用 C++ 编写的图形用户界面(GUI)库.它由O ...
【数值计算方法】蒙特卡洛方法积分的Python实现
原理不做赘述,参见[数值计算方法]数值积分&微分-python实现 - FE-有限元鹰 - 博客园,直接上代码,只实现1d,2d积分,N维积分的蒙特卡洛方法也类似. 代码 from typin ...
Delphi edit只允许输入数字
if not (key in ['0'..'9',#8]) then key := #0;
BUUCTF---checkln（ROT加密）
题目 dikqTCpfRjA8fUBIMD5GNDkwMjNARkUwI0BFTg== 解题 1.看见有==,尝试base64解码,但解码失败 2.尝试其它Base加密均失败 3.重新思考有等于号的加 ...
Zabbix Scheduled reports中文乱码
困扰了一天的问题,最后还是靠百度大神的方法解决.原文链接 https://blog.csdn.net/weixin_38587368/article/details/119357516 由于我的是容器 ...
我对TamperMonkey的不满-更新中
我认为我的电脑上的TamperMonkey插件的值得考虑的不足: 没有提供一个把脚本最小化的功能不能编辑热键脚本icon不能使用svg 没有提供一种很好的能够区分别人的脚本和自己的脚本的方式,自己 ...
实现Andriod的APP中文名
让程序编译后,就会自动生成中文名,以及启动界面. 1. 2.AndroidManifest.xml的修改. 如果新建project,自动生成,否则要手动 3. 4.ok了.到手机端看结果吧
java学习-6-核心类：字符串StringJoiner 和数组一起玩
public class Main { public static void main(String[] args) { String[] names = {"Bob", &quo ...

minikube dashboard ImagePullBackOff 失败问题的解决方法

问题