K8s 开始

Kubernetes 是用于自动部署，扩展和管理容器化应用程序的开源系统。本文将介绍如何快速开始 K8s 的使用。

了解 K8s

• Kubernetes / Overview

搭建 K8s

本地开发测试，需要搭建一个 K8s 轻量服务。实际部署时，可以用云厂商的 K8s 服务。

本文以 k3d 为例，于 macOS 搭建 K8s 服务。于 Ubuntu 则推荐 MicroK8s。其他可替代方案有：

• Kubernetes / Install Tools
  • kind, minikube, kubeadm
• Docker Desktop / Deploy on Kubernetes

k3d

k3s 是 Rancher 推出的 K8s 轻量版。而 k3d 即 k3s in docker，以 docker 容器管理 k3s 集群。

以下搭建过程，是于 macOS 的笔记，供参考。其他平台，请依照官方文档进行。

# 安装 kubectl: 命令行工具
brew install kubectl
# 安装 kubecm: 配置管理工具
brew install kubecm

# 安装 k3d
brew install k3d
❯ k3d version
k3d version v4.4.8
k3s version latest (default)

创建集群（1主2从）：

❯ k3d cluster create mycluster --api-port 6550 --servers 1 --agents 2 --port 8080:80@loadbalancer --wait
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-mycluster' (23dc5761582b1a4b74d9aa64d8dca2256b5bc510c4580b3228123c26e93f456e)
INFO[0000] Created volume 'k3d-mycluster-images'
INFO[0001] Creating node 'k3d-mycluster-server-0'
INFO[0001] Creating node 'k3d-mycluster-agent-0'
INFO[0001] Creating node 'k3d-mycluster-agent-1'
INFO[0001] Creating LoadBalancer 'k3d-mycluster-serverlb'
INFO[0001] Starting cluster 'mycluster'
INFO[0001] Starting servers...
INFO[0001] Starting Node 'k3d-mycluster-server-0'
INFO[0009] Starting agents...
INFO[0009] Starting Node 'k3d-mycluster-agent-0'
INFO[0022] Starting Node 'k3d-mycluster-agent-1'
INFO[0030] Starting helpers...
INFO[0030] Starting Node 'k3d-mycluster-serverlb'
INFO[0031] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
INFO[0036] Successfully added host record to /etc/hosts in 4/4 nodes and to the CoreDNS ConfigMap
INFO[0036] Cluster 'mycluster' created successfully!
INFO[0036] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false
INFO[0036] You can now use it like this:
kubectl config use-context k3d-mycluster
kubectl cluster-info

查看集群信息：

❯ kubectl cluster-info
Kubernetes control plane is running at https://0.0.0.0:6550
CoreDNS is running at https://0.0.0.0:6550/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:6550/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

查看资源信息：

# 查看 Nodes
❯ kubectl get nodes
NAME                     STATUS   ROLES                  AGE     VERSION
k3d-mycluster-agent-0    Ready    <none>                 2m12s   v1.20.10+k3s1
k3d-mycluster-server-0   Ready    control-plane,master   2m23s   v1.20.10+k3s1
k3d-mycluster-agent-1    Ready    <none>                 2m4s    v1.20.10+k3s1
# 查看 Pods
❯ kubectl get pods --all-namespaces
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-6488c6fcc6-5n7d9                  1/1     Running     0          2m12s
kube-system   metrics-server-86cbb8457f-dr7lh           1/1     Running     0          2m12s
kube-system   local-path-provisioner-5ff76fc89d-zbxf4   1/1     Running     0          2m12s
kube-system   helm-install-traefik-bfm4c                0/1     Completed   0          2m12s
kube-system   svclb-traefik-zx98g                       2/2     Running     0          68s
kube-system   svclb-traefik-7bx2r                       2/2     Running     0          68s
kube-system   svclb-traefik-cmdrm                       2/2     Running     0          68s
kube-system   traefik-6f9cbd9bd4-2mxhk                  1/1     Running     0          69s

测试 Nginx：

# 创建 Nginx Deployment
kubectl create deployment nginx --image=nginx
# 创建 ClusterIP Service，暴露 Nginx 端口
kubectl create service clusterip nginx --tcp=80:80
# 创建 Ingress Object
#  k3s 以 traefik 为默认 ingress controller
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80
EOF
# 访问 Nginx Service
#  kubectl get pods 确认 nginx STATUS=Running
open http://127.0.0.1:8080

测试 Dashboard：

# 创建 Dashboard
GITHUB_URL=https://github.com/kubernetes/dashboard/releases
VERSION_KUBE_DASHBOARD=$(curl -w '%{url_effective}' -I -L -s -S ${GITHUB_URL}/latest -o /dev/null | sed -e 's|.*/||')
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/${VERSION_KUBE_DASHBOARD}/aio/deploy/recommended.yaml
# 配置 RBAC
#  admin user
cat <<EOF > dashboard.admin-user.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
EOF
#  admin user role
cat <<EOF > dashboard.admin-user-role.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF
# 配置部署
kubectl create -f dashboard.admin-user.yml -f dashboard.admin-user-role.yml
# 获取 Bearer Token
kubectl -n kubernetes-dashboard describe secret admin-user-token | grep ^token
# 访问代理
kubectl proxy
# 访问 Dashboard
#  输入 Token 登录
open http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

删除集群：

k3d cluster delete mycluster

切换集群：

kubecm s

参考：

• k3s / Dashboard
• k3d / Exposing Services

MicroK8s

MicroK8s 是 Ubuntu 官方生态提供的 K8s 轻量版，适合用于开发工作站、IoT、Edge、CI/CD。

以下搭建过程，是于 Ubuntu 18/20 的笔记，供参考。其他平台，请依照官方文档进行。

# 检查 hostname
#  要求不含大写字母和下划线，不然依照后文修改
hostname

# 安装 microk8s
sudo apt install snapd -y
snap info microk8s
sudo snap install microk8s --classic --channel=1.21/stable

# 添加用户组
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube
newgrp microk8s
id $USER

## 一些确保拉到镜像的方法
# 配置代理（如果有）
#  MicroK8s / Installing behind a proxy
#   https://microk8s.io/docs/install-proxy
#  Issue: Pull images from others than k8s.gcr.io
#   https://github.com/ubuntu/microk8s/issues/472
sudo vi /var/snap/microk8s/current/args/containerd-env
  HTTPS_PROXY=http://127.0.0.1:7890
  NO_PROXY=10.1.0.0/16,10.152.183.0/24
# 添加镜像（docker.io）
#  镜像加速器
#   https://yeasy.gitbook.io/docker_practice/install/mirror
#  还可改 args/ 里不同模板的 sandbox_image
sudo vi /var/snap/microk8s/current/args/containerd-template.toml
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://x.mirror.aliyuncs.com", "https://registry-1.docker.io", ]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:32000"]
          endpoint = ["http://localhost:32000"]
# 手动导入，见后文启用插件那

# 重启服务
microk8s stop
microk8s start

检查状态：

$ microk8s status
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    ha-cluster           # Configure high availability on the current node
  disabled:
    ambassador           # Ambassador API Gateway and Ingress
    cilium               # SDN, fast with full network policy
    dashboard            # The Kubernetes dashboard
    dns                  # CoreDNS
    fluentd              # Elasticsearch-Fluentd-Kibana logging and monitoring
    gpu                  # Automatic enablement of Nvidia CUDA
    helm                 # Helm 2 - the package manager for Kubernetes
    helm3                # Helm 3 - Kubernetes package manager
    host-access          # Allow Pods connecting to Host services smoothly
    ingress              # Ingress controller for external access
    istio                # Core Istio service mesh services
    jaeger               # Kubernetes Jaeger operator with its simple config
    keda                 # Kubernetes-based Event Driven Autoscaling
    knative              # The Knative framework on Kubernetes.
    kubeflow             # Kubeflow for easy ML deployments
    linkerd              # Linkerd is a service mesh for Kubernetes and other frameworks
    metallb              # Loadbalancer for your Kubernetes cluster
    metrics-server       # K8s Metrics Server for API access to service metrics
    multus               # Multus CNI enables attaching multiple network interfaces to pods
    openebs              # OpenEBS is the open-source storage solution for Kubernetes
    openfaas             # openfaas serverless framework
    portainer            # Portainer UI for your Kubernetes cluster
    prometheus           # Prometheus operator for monitoring and logging
    rbac                 # Role-Based Access Control for authorisation
    registry             # Private image registry exposed on localhost:32000
    storage              # Storage class; allocates storage from host directory
    traefik              # traefik Ingress controller for external access

如果 status 不正确时，可以如下排查错误：

microk8s inspect
grep -r error /var/snap/microk8s/2346/inspection-report

如果要修改 hostname：

# 改名称
sudo hostnamectl set-hostname ubuntu-vm
# 改 host
sudo vi /etc/hosts

# 云主机的话，还要改下配置
sudo vi /etc/cloud/cloud.cfg
  preserve_hostname: true
  # 如果只修改 preserve_hostname 不生效，那就直接注释掉 set/update_hostname
  cloud_init_modules:
  #  - set_hostname
  #  - update_hostname

# 重启，验证生效
sudo reboot

接着，启用些基础插件：

microk8s enable dns dashboard

# 查看 Pods ，确认 running
microk8s kubectl get pods --all-namespaces
# 不然，详情里看下错误原因
microk8s kubectl describe pod --all-namespaces

直到全部正常 running：

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                         READY   STATUS    RESTARTS   AGE
kube-system   kubernetes-dashboard-85fd7f45cb-snqrv        1/1     Running   1          15h
kube-system   dashboard-metrics-scraper-78d7698477-tmb7k   1/1     Running   1          15h
kube-system   metrics-server-8bbfb4bdb-wlf8g               1/1     Running   1          15h
kube-system   calico-node-p97kh                            1/1     Running   1          6m18s
kube-system   coredns-7f9c69c78c-255fg                     1/1     Running   1          15h
kube-system   calico-kube-controllers-f7868dd95-st9p7      1/1     Running   1          16h

如果拉取镜像失败，可以 microk8s ctr image pull <mirror>。或者，docker pull 后导入 containerd：

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
docker save k8s.gcr.io/pause:3.1 > pause:3.1.tar
microk8s ctr image import pause:3.1.tar

docker pull calico/cni:v3.13.2
docker save calico/cni:v3.13.2 > cni:v3.13.2.tar
microk8s ctr image import cni:v3.13.2.tar

docker pull calico/node:v3.13.2
docker save calico/node:v3.13.2 > node:v3.13.2.tar
microk8s ctr image import node:v3.13.2.tar

如果 calico-node CrashLoopBackOff，可能网络配置问题：

# 查具体日志
microk8s kubectl logs -f -n kube-system calico-node-l5wl2 -c calico-node
# 如果有 Unable to auto-detect an IPv4 address，那么 ip a 找出哪个网口有 IP 。修改：
sudo vi /var/snap/microk8s/current/args/cni-network/cni.yaml
  - name: IP_AUTODETECTION_METHOD
  value: "interface=wlo.*"
# 重启服务
microk8s stop; microk8s start

## 参考
# Issue: Microk8s 1.19 not working on Ubuntu 20.04.1
#  https://github.com/ubuntu/microk8s/issues/1554
# Issue: CrashLoopBackOff for calico-node pods
#  https://github.com/projectcalico/calico/issues/3094
# Changing the pods CIDR in a MicroK8s cluster
#  https://microk8s.io/docs/change-cidr
# MicroK8s IPv6 DualStack HOW-TO
#  https://discuss.kubernetes.io/t/microk8s-ipv6-dualstack-how-to/14507

然后，可以打开 Dashboard 看看：

# 获取 Token （未启用 RBAC 时）
token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
microk8s kubectl -n kube-system describe secret $token
# 转发端口
microk8s kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443
# 打开网页，输入 Token 登录
xdg-open https://127.0.0.1:10443

# 更多说明 https://microk8s.io/docs/addon-dashboard
# Issue: Your connection is not private
#  https://github.com/kubernetes/dashboard/issues/3804

更多操作，请阅读官方文档。本文之后仍以 k3d 为例。

准备 K8s 应用

Go 应用

http_server.go:

package main

import (
	"fmt"
	"log"
	"net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:])
}

func main() {
	http.HandleFunc("/", handler)
	fmt.Println("HTTP Server running ...")
	log.Fatal(http.ListenAndServe(":3000", nil))
}

构建镜像

http_server.dockerfile:

FROM golang:1.17-alpine AS builder
WORKDIR /app
ADD ./http_server.go /app
RUN cd /app && go build http_server.go

FROM alpine:3.14
WORKDIR /app
COPY --from=builder /app/http_server /app/
EXPOSE 3000
ENTRYPOINT ./http_server

# 编译镜像
docker build -t http_server:1.0 -f http_server.dockerfile .
# 运行应用
docker run --rm -p 3000:3000 http_server:1.0
# 测试应用
❯ curl http://127.0.0.1:3000/go
Hi there, I love go!

部署 K8s 应用

了解概念

• Kubernetes / Workloads

之后，参照官方教程，我们将使用 Deployment 运行 Go 应用（无状态）。

导入镜像

首先，我们手动导入镜像进集群：

docker save http_server:1.0 > http_server:1.0.tar
k3d image import http_server:1.0.tar -c mycluster

如果有自己的私有仓库，参见 k3d / Registries 进行配置。

创建 Deployment

# 配置 Deployment （2个副本）
cat <<EOF > go-http-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-http
  labels:
    app: go-http
spec:
  replicas: 2
  selector:
    matchLabels:
      app: go-http
  template:
    metadata:
      labels:
        app: go-http
    spec:
      containers:
      - name: go-http
        image: http_server:1.0
        ports:
        - containerPort: 3000
EOF
# 应用 Deployment
#  --record: 记录命令
kubectl apply -f go-http-deployment.yaml --record

查看 Deployment：

# 查看 Deployment
❯ kubectl get deploy
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
nginx     1/1     1            1           2d
go-http   2/2     2            2           22s

# 查看 Deployment 信息
kubectl describe deploy go-http
# 查看 Deployment 创建的 ReplicaSet （2个）
kubectl get rs
# 查看 Deployment 创建的 Pods （2个）
kubectl get po -l app=go-http -o wide --show-labels
# 查看某一 Pod 信息
kubectl describe po go-http-5848d49c7c-wzmxh

• k8s / Deployments

创建 Service

# 创建 Service，名为 go-http
#  将请求代理到 app=go-http, tcp=3000 的 Pod 上
kubectl expose deployment go-http --name=go-http
# 或
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: go-http
  labels:
    app: go-http
spec:
  selector:
    app: go-http
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 3000
EOF

# 查看 Service
kubectl get svc
# 查看 Service 信息
kubectl describe svc go-http
# 查看 Endpoints 对比看看
#  kubectl get ep go-http
#  kubectl get po -l app=go-http -o wide

# 删除 Service （如果）
kubectl delete svc go-http

访问 Service (DNS)：

❯ kubectl run curl --image=radial/busyboxplus:curl -i --tty

If you don't see a command prompt, try pressing enter.

[ root@curl:/ ]$ nslookup go-http
Server:    10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

Name:      go-http
Address 1: 10.43.102.17 go-http.default.svc.cluster.local

[ root@curl:/ ]$ curl http://go-http:3000/go
Hi there, I love go!

暴露 Service (Ingress)：

# 创建 Ingress Object
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: go-http
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /go
        pathType: Prefix
        backend:
          service:
            name: go-http
            port:
              number: 3000
EOF
# 查看 Ingress
kubectl get ingress
# 查看 Ingress 信息
kubectl describe ingress go-http

# 删除 Ingress （如果）
kubectl delete ingress go-http

访问 Service (Ingress)：

❯ open http://127.0.0.1:8080/go
# 或，
❯ curl http://127.0.0.1:8080/go
Hi there, I love go!
# Nginx 是在 http://127.0.0.1:8080

• k8s / Connecting Applications with Services
• k8s / Exposing an External IP Address

更新

仅当 Deployment Pod 模板发生改变时，例如模板的标签或容器镜像被更新，才会触发 Deployment 上线。其他更新（如对 Deployment 执行扩缩容的操作）不会触发上线动作。

所以，我们准备 http_server:2.0 镜像导入集群，然后更新：

❯ kubectl set image deployment/go-http go-http=http_server:2.0 --record
deployment.apps/go-http image updated

之后，可以查看上线状态：

# 查看上线状态
❯ kubectl rollout status deployment/go-http
deployment "go-http" successfully rolled out

# 查看 ReplicaSet 状态：新的扩容，旧的缩容，完成更新
❯ kubectl get rs
NAME                 DESIRED   CURRENT   READY   AGE
go-http-586694b4f6   2         2         2       10s
go-http-5848d49c7c   0         0         0       6d

测试服务：

❯ curl http://127.0.0.1:8080/go
Hi there v2, I love go!

回滚

查看 Deployment 修订历史：

❯ kubectl rollout history deployment.v1.apps/go-http
deployment.apps/go-http
REVISION  CHANGE-CAUSE
1         kubectl apply --filename=go-http-deployment.yaml --record=true
2         kubectl set image deployment/go-http go-http=http_server:2.0 --record=true

# 查看修订信息
kubectl rollout history deployment.v1.apps/go-http --revision=2

回滚到之前的修订版本：

# 回滚到上一版
kubectl rollout undo deployment.v1.apps/go-http
# 回滚到指定版本
kubectl rollout undo deployment.v1.apps/go-http --to-revision=1

缩放

# 缩放 Deployment 的 ReplicaSet 数
kubectl scale deployment.v1.apps/go-http --replicas=10

# 如果集群启用了 Pod 的水平自动缩放，可以根据现有 Pods 的 CPU 利用率选择上下限
# Horizontal Pod Autoscaler Walkthrough
#  https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
kubectl autoscale deployment.v1.apps/nginx-deployment --min=10 --max=15 --cpu-percent=80

暂停、恢复

# 暂停 Deployment
kubectl rollout pause deployment.v1.apps/go-http
# 恢复 Deployment
kubectl rollout resume deployment.v1.apps/go-http

期间可以更新 Deployment ，但不会触发上线。

删除

kubectl delete deployment go-http

金丝雀部署

灰度部署，用多标签区分多个部署，新旧版可同时运行。部署新版时，用少量流量验证，没问题再全量更新。

• k8s / Canary Deployments

Helm 发布

Helm 是 K8s 的包管理工具，包格式称为 charts。现在来发布我们的 Go 服务吧。

安装 Helm

# macOS
brew install helm
# Ubuntu
sudo snap install helm --classic

执行 helm 了解命令。

创建 Chart

helm create go-http

查看内容：

❯ tree go-http -aF --dirsfirst
go-http
├── charts/     # 包依赖的 charts，称 subcharts
├── templates/  # 包的 K8s 文件模板，用的 Go 模板
│   ├── tests/
│   │   └── test-connection.yaml
│   ├── NOTES.txt       # 包的帮助文本
│   ├── _helpers.tpl    # 模板可重用的片段，模板里 include 引用
│   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── service.yaml
│   └── serviceaccount.yaml
├── .helmignore # 打包忽略说明
├── Chart.yaml  # 包的描述文件
└── values.yaml # 变量默认值，可安装时覆盖，模板里 .Values 引用

修改内容：

• 修改 Chart.yaml 里的描述
  • 更多见 The Chart.yaml File
• 修改 values.yaml 里的变量
  • 修改 image 为发布的 Go 服务
  • 修改 ingress 为 true，及一些配置
  • 删除 serviceAccount autoscaling，templates/ 里也搜索删除相关内容
• 修改 templates/ 里的模板
  • 删除 tests/，剩余 deployment.yaml service.yaml ingress.yaml 有用

结果可见 start-k8s/helm/go-http。

检查错误：

❯ helm lint --strict go-http
==> Linting go-http
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

渲染模板：

# --set 覆盖默认配置，或者用 -f 选择自定的 values.yaml
helm template go-http-helm ./go-http \
--set replicaCount=2 \
--set "ingress.hosts[0].paths[0].path=/helm" \
--set "ingress.hosts[0].paths[0].pathType=Prefix"

安装 Chart：

helm install go-http-helm ./go-http \
--set replicaCount=2 \
--set "ingress.hosts[0].paths[0].path=/helm" \
--set "ingress.hosts[0].paths[0].pathType=Prefix"

# 或，打包后安装
helm package go-http
helm install go-http-helm go-http-1.0.0.tgz \
--set replicaCount=2 \
--set "ingress.hosts[0].paths[0].path=/helm" \
--set "ingress.hosts[0].paths[0].pathType=Prefix"

# 查看安装列表
helm list

测试服务：

❯ kubectl get deploy go-http-helm
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
go-http-helm   2/2     2            2           2m42s

❯ curl http://127.0.0.1:8080/helm
Hi there, I love helm!

卸载 Chart：

helm uninstall go-http-helm

• Helm / Chart Template Guide

发布 Chart

官方仓库 ArtifactHub 上有很多分享的 Helm charts 。可见 velkoz1108/helm-chart 把我们的 Go 服务发布到 Hub 上。

• Helm / Chart Repository Guide

最后

开始 K8s 吧！本文样例在 ikuokuo/start-k8s。

GoCoding 个人实践的经验分享，可关注公众号！