K8S Only IPV6的创建过程之二 完整版

K8S Only IPV6的创建过程之二 完整版

---

整体过程

1. 阿里云新增一台虚拟机, 开通IPV6.与数据库,redis实现物理隔离. 并且进行环境基本处理,安装kubeadm等组件.
2. kubeadm 部署 K8S 1.28.4 并且使用ipv6 SingleStack 的calico网络组件.
3. 修改 nginx-ingress 1.9的部署文件, 修改ipv6支持.
4. 使用endpoint的方式将 mysql/redis 等服务通过service方式暴露到IPV6的K8S内部.
5. 修改myapp的配置文件, 将redis和mysql服务修改为服务名模式, 并且在宿主机使用mysql 的service name 重新注册.
6. docker build myapp 并且使用service的方式进行部署.
7. 使用ingress 进行服务暴露, 增加https证书处理.
8. 地址为 k8sipv6.jnxlh.online . 

8.1 镜像信息
myapp       v1.0        2bed9a6fcf63   7 hours ago     6.77GB

8.2 k8s的服务信息
[root@k8sipv6 ~]# kubectl get svc
NAME              TYPE        CLUSTER-IP            EXTERNAL-IP   PORT(S)          AGE
myapp-service   NodePort    2001:db8:42:1::5040   <none>        80:32063/TCP     10h
kubernetes        ClusterIP   2001:db8:42:1::1      <none>        443/TCP          10h
mysql-service     NodePort    2001:db8:42:1::7b09   <none>        3306:30406/TCP   8h
redis-service     NodePort    2001:db8:42:1::9a1e   <none>        6378:30410/TCP   8h

8.3 ingress信息
[root@k8sipv6 ~]# kubectl get ing
NAME              CLASS    HOSTS                  ADDRESS                                 PORTS     AGE
myapp-ingress   <none>   k8sipv6.jnxlh.online   2408:400a:ed:d4ff:7a7b:f5dc::1   80, 443   7h54m

8.4 pod信息 IPV6
[root@k8sipv6 ~]# kubectl get pod -owide
NAME                                  READY   STATUS    RESTARTS   AGE     IP
myapp-deployment-5487fb6775-96nhh   1/1     Running   0          6h54m   2001:db8:1234:5678:8:2:d1:790c  

第一步机器处理

修改hosts
2408:400a:ed:d4ff:7a7b:f5dc::1   k8sipv6

第二步修改配置文件

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
net.ipv6.ip_forward                 = 1
EOF

sysctl -p

---

安装必备软件

docker
cri-docker
kubeadm 等

---

修改 cri-docker

vim /usr/lib/systemd/system/cri-docker.service
修改为
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9

---

启动docker

systemctl enable --now docker
然后导入镜像
for i in `ls *.tar`; do docker load -i $i ;done

启动cri-docker
systemctl daemon-reload
systemctl restart cri-docker && systemctl enable cri-docker

---

初始化K8S

kubeadm init --node-name=k8sipv6  \
--image-repository=registry.aliyuncs.com/google_containers \
--cri-socket=unix:///var/run/cri-dockerd.sock \
--pod-network-cidr=2001:db8:42:0::/64 \
--service-cidr=2001:db8:42:1::/112  \
--apiserver-advertise-address=2408:400a:ed:d4ff:7a7b:f5dc::1  \
--apiserver-bind-port=6443 

注意需要修改一下 节点的taint
这个适合单节点部署的测试环境

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

---

安装calico

wget https://raw.githubusercontent.com/sgryphon/kubernetes-ipv6/main/calico-ipv6.yaml

直接可以用就可以了
kubectl apply -f calico-ipv6.yaml

---

nginx-ingress的处理

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.0/deploy/static/provider/baremetal/deploy.yaml

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20230407
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.9.0

注意 可以先拉去镜像, 然后在修改 deploy.yaml 里面的镜像信息 再执行 deploy
注意 要将里面的IPv4 修改为IPv6

  ipFamilies:
  - IPv6
  ipFamilyPolicy: SingleStack

---

效果

kubectl get pods -A -owide
NAMESPACE       NAME                                        READY   STATUS      RESTARTS   AGE     IP
ingress-nginx   ingress-nginx-admission-create-6ggqn        0/1     Completed   0          106s    2001:db8:1234:5678:8:2:d1:7903
ingress-nginx   ingress-nginx-admission-patch-2nnfq         0/1     Completed   0          106s    2001:db8:1234:5678:8:2:d1:7904
ingress-nginx   ingress-nginx-controller-7c76845d84-6t6f8   1/1     Running     0          106s    2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system     calico-kube-controllers-77455c6d7c-cg6gr    1/1     Running     0          5m1s    2001:db8:1234:5678:8:2:d1:7902
kube-system     calico-node-dthhl                           1/1     Running     0          5m1s    2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system     coredns-66f779496c-dzh5c                    1/1     Running     0          5m14s   2001:db8:1234:5678:8:2:d1:7900
kube-system     coredns-66f779496c-lrmn7                    1/1     Running     0          5m14s   2001:db8:1234:5678:8:2:d1:7901
kube-system     etcd-k8sipv6                                1/1     Running     0          5m28s   2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system     kube-apiserver-k8sipv6                      1/1     Running     0          5m29s   2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system     kube-controller-manager-k8sipv6             1/1     Running     0          5m29s   2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system     kube-proxy-v8ztl                            1/1     Running     0          5m13s   2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system     kube-scheduler-k8sipv6                      1/1     Running     0          5m28s   2408:400a:ed:d4ff:7a7b:f5dc:a12f

---

redis 与 mysql的处理

apiVersion: v1
kind: Service
metadata:
  name: redis-service
spec:
  ports:
  - port: 6378
  type: NodePort
---
apiVersion: v1
kind: Endpoints
metadata:
  name: redis-service
subsets:
  - addresses:
    - ip: 2408:400a:ed:d4ff:7a7b:f5dc::2
    ports:
    - port: 6378

---

部署service和ingress

kubectl create secret tls k8sipv6_secret --key=k8sipv6.jnxlh.online.key --cert=k8sipv6.jnxlh.online.pem
cat > myapp.service <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: myapp
  name: myapp-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - image: myapp:v1.0
        ports:
        - containerPort: 5200
        name: myapp

---
apiVersion: v1
kind: Service
metadata:
  labels:
   app: myapp
  name: myapp-service
  namespace: default
spec:
  sessionAffinity: ClientIP
  ports:
  - port: 80
    name: myapp-service
    protocol: TCP
    targetPort: 5200
  selector:
    app: myapp
  type: NodePort

EOF

---

部署 ingress

cat >ingress.yaml<<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
    - hosts:
        - k8sipv6.jnxlh.online
      secretName: k8sipv6secret
  rules:
  - host: k8sipv6.jnxlh.online
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-service
            port:
              number: 80
EOF

kubectl apply -f ingress2.yaml

如果没有证书 可能会报错:  可以通过下面的命令来解决:
kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

Warning: annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
Error from server (InternalError): error when creating "ingress2.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": no service port 443 found for service "ingress-nginx-controller-admission"