使用ubuntu charmed kubernetes 部署一套生产环境的集群

官方文档: https://ubuntu.com/kubernetes/docs

搭建一个基本的集群

集群ip规划

hostname	ip
ubuntu-1	10.0.0.10
juju-controller-1	10.0.0.11
juju-master-1	10.0.0.21
juju-master-2	10.0.0.22
juju-master-3	10.0.0.23
juju-worker-1	10.0.0.31
juju-worker-2	10.0.0.32

ubuntu-1 为juju客户端和haproxy机器

juju-controller-1为juju控制器节点(可以做高可用)

以下操作都是在ubuntu-1上执行

安装juju

sudo snap install juju --classic

设置云类型

juju add-cloud
输入manual

添加机器(一共5台)

juju bootstrap
juju add-machine ssh:root@x.x.x.x

机器添加完成后, juju machines 能看到机器id

生成安装core.yaml

在https://jujucharms.com/new/画图，然后导出成yaml

这里是我的yaml

applications:
  containerd:
    charm: 'cs:~containers/containerd-102'
    series: focal
    annotations:
      gui-x: '475'
      gui-y: '800'
  easyrsa:
    charm: 'cs:~containers/easyrsa-345'
    num_units: 1
    series: focal
    annotations:
      gui-x: '90'
      gui-y: '420'
    to:
      - '0'
  etcd:
    charm: 'cs:~containers/etcd-553'
    num_units: 3
    options:
      channel: 3.4/stable
    series: focal
    annotations:
      gui-x: '800'
      gui-y: '420'
    to:
      - '0'
      - '1'
      - '2'
  flannel:
    charm: 'cs:~containers/flannel-518'
    options:
      cidr: 10.24.0.0/16
    series: focal
    annotations:
      gui-x: '475'
      gui-y: '605'
  kubernetes-master:
    charm: 'cs:~containers/kubernetes-master-955'
    num_units: 3
    options:
      enable-dashboard-addons: false
      proxy-extra-args: proxy-mode=ipvs
      service-cidr: 10.153.183.0/24
    series: focal
    annotations:
      gui-x: '800'
      gui-y: '850'
    to:
      - '0'
      - '1'
      - '2'
  kubernetes-worker:
    charm: 'cs:~containers/kubernetes-worker-726'
    num_units: 2
    options:
      proxy-extra-args: proxy-mode=ipvs
    expose: true
    series: focal
    annotations:
      gui-x: '90'
      gui-y: '850'
    to:
      - '3'
      - '4'
relations:
  - - 'kubernetes-master:kube-api-endpoint'
    - 'kubernetes-worker:kube-api-endpoint'
  - - 'kubernetes-master:kube-control'
    - 'kubernetes-worker:kube-control'
  - - 'kubernetes-master:certificates'
    - 'easyrsa:client'
  - - 'etcd:certificates'
    - 'easyrsa:client'
  - - 'kubernetes-master:etcd'
    - 'etcd:db'
  - - 'kubernetes-worker:certificates'
    - 'easyrsa:client'
  - - 'flannel:etcd'
    - 'etcd:db'
  - - 'flannel:cni'
    - 'kubernetes-master:cni'
  - - 'flannel:cni'
    - 'kubernetes-worker:cni'
  - - 'containerd:containerd'
    - 'kubernetes-worker:container-runtime'
  - - 'containerd:containerd'
    - 'kubernetes-master:container-runtime'
machines:
  '0': {}
  '1': {}
  '2': {}
  '3': {}
  '4': {}

根据yml来部署

juju deploy ./core.yaml --map-machines=existing,0=0,1=1,2=2,3=3,4=4

juju status 全部idle就算正常了

kata容器

部署

juju deploy cs:~containers/kata
juju add-relation kata kubernetes-master
juju add-relation kata kubernetes-worker
juju add-relation kata:untrusted containerd:untrusted

使用

部署的时候加上io.kubernetes.cri.untrusted-workload: "true"

apiVersion: v1
kind: Pod
metadata:
  name: nginx-untrusted
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
  - name: nginx
    image: nginx

更换master节点

先删除节点

juju remove-unit etcd/1 --force --no-wait
juju remove-unit kubernetes-master/1 --force --no-wait
juju remove-machine 1 --force --no-wait

etcd集群删除这个member

https://ubuntu.com/kubernetes/docs/charm-etcd

需要先下载证书到本地(要解压)

juju run-action --wait etcd/0 package-client-credentials
juju scp etcd/0:etcd_credentials.tar.gz etcd_credentials.tar.gz

然后使用etcdctl删除节点

export ETCDCTL_KEY_FILE=$(pwd)/client.key
export ETCDCTL_CERT_FILE=$(pwd)/client.crt
export ETCDCTL_CA_FILE=$(pwd)/ca.crt
export ETCDCTL_ENDPOINT=https://10.0.0.21:2379
etcdctl member list

etcdctl member remove c2499df1988d1925

增加节点

juju add-machine ssh:root@100.64.1.167
juju machines
记住节点ID,假如是9

扩容master到节点9

juju add-unit kubernetes-master --to 9

扩容etcd到节点9

juju add-unit etcd --to 9

执行kubectl命令

搭建haproxy负载均衡

haproxy配置文件

...省略

frontend http_ingress_traffic_fe
    bind 0.0.0.0:80
    mode tcp
    default_backend   http_ingress_traffic_be

backend http_ingress_traffic_be
    mode tcp
    balance     roundrobin
    server      juju-worker-1 10.0.0.31:80 check
    server      juju-worker-2 10.0.0.32:80 check

frontend https_ingress_traffic_fe
    bind 0.0.0.0:443
    mode tcp
    default_backend   https_ingress_traffic_be

backend https_ingress_traffic_be
    mode tcp
    balance     roundrobin
    server      juju-worker-1 10.0.0.31:443 check
    server      juju-worker-2 10.0.0.32:443 check

frontend k8s_api_fe
    bind 0.0.0.0:6443
    mode tcp
    default_backend   k8s_api_be

backend k8s_api_be
    mode tcp
    balance     roundrobin
    server      juju-master-1 10.0.0.21:6443 check
    server      juju-master-2 10.0.0.22:6443 check
    server      juju-master-3 10.0.0.23:6443 check

修改负载均衡的ip

juju config kubernetes-master loadbalancer-ips="10.0.0.10"

安装kubectl

sudo snap install kubectl --classic

获取新的kubeconfig配置文件

juju scp kubernetes-master/0:config ~/.kube/config

juju-controller高可用

增加新的controller节点

juju add-machine -m controller ssh:root@100.64.1.169

查看controller机器

juju machines -m controller

开启高可用

juju enable-ha --to 5,6

查看controller信息

juju controllers --refresh
juju controllers