kubernetes-集群构建

本实验参考：https://github.com/gjmzj/kubeasz

kubernetes官方github地址 https://github.com/kubernetes/kubernetes/releases

 

软硬件限制：

①cpu和内存 master：至少1c2g，推荐2c4g；node：至少1c2g

②linux系统 内核版本至少3.10，推荐CentOS7/RHEL7

③docker 至少1.9版本，推荐1.12+

④etcd 至少2.0版本，推荐3.0+

 

 

高可用集群所需节点规划：

①部署节点------x1 : 运行这份 ansible 脚本的节点

②etcd节点------x3 : 注意etcd集群必须是1,3,5,7...奇数个节点

③master节点----x2 : 根据实际集群规模可以增加节点数，需要额外规划一个master VIP(虚地址)

④lb节点--------x2 : 负载均衡节点两个，安装 haproxy+keepalived

⑤node节点------x3 : 真正应用负载的节点，根据需要提升机器配置和增加节点数

 

四台主机规划：

主机	主机名	集色角色
192.168.1.200	master	deploy、etcd、lb1、master1
192.168.1.201	master2	lb2、master2
192.168.1.202	node	etcd2、node1
192.168.1.203	node2	etcd3、node2
192.168.1.250		vip

一、准备工作

1：四台机器都执行安装epel源、更新、安装Python包。（说明：这边是为了做实验，防止出现不必要错误，把防火墙关闭了，生成环境勿学）

 yum install -y  epel-release
 yum install -y  python
 iptables -F
 setenforce 

【deploy节点操作】

 2：安装ansible

[root@master ~]# yum -y install ansible

3：生成密钥对

 [root@master ~]# ssh-keygen
 Generating public/private rsa key pair.
 Enter file in which to save the key (/root/.ssh/id_rsa):
 Created directory '/root/.ssh'.
 Enter passphrase (empty for no passphrase):
 Enter same passphrase again:
 Your identification has been saved in /root/.ssh/id_rsa.
 Your public key has been saved in /root/.ssh/id_rsa.pub.
 The key fingerprint is:
 SHA256:cfoSPSgeEkAkgY08UIVWK2t2eNJIrKph5wkRkZX7AKs root@master
 The key's randomart image is:
 +---[RSA ]----+
 |BOB=+            |
 |oB=o .           |
 | oB +   . .      |
 | +.O .   *       |
 |o.B B o S o      |
 |Eo.+ + o o .     |
 |oo .  . . .      |
 |o.+ .    .       |
 |.  o             |
 +----[SHA256]-----+

4：拷贝秘钥到四台机器中

 [root@master ~]# for ip in    ; do ssh-copy-id 192.168..$ip; done

5：测试是否可以免密登录

 [root@master ~]# ssh 192.168.1.200
 Last login: Wed Dec  ::  from 192.168.1.2
 [root@master ~]# exit
 登出
 Connection to 192.168.1.200 closed.
 [root@master ~]# ssh 192.168.1.201
 Last login: Wed Dec  ::  from 192.168.1.2
 [root@master2 ~]# exit
 登出
 Connection to 192.168.1.201 closed.
 [root@master ~]# ssh 192.168.1.202
 Last login: Wed Dec  ::  from 192.168.1.200
 [root@node1 ~]# exit
 登出
 Connection to 192.168.1.202 closed.
 [root@master ~]# ssh 192.168.1.203
 Last login: Wed Dec  ::  from 192.168.1.2
 [root@node2 ~]#  exit
 登出
 Connection to 192.168.1.203 closed.

6：下载脚本文件，安装kubeasz代码、二进制、离线镜像

脚本下载链接：https://pan.baidu.com/s/1GLoU9ntjUL2SP4R_Do7mlQ

提取码：96eg

 [root@master ~]# chmod +x easzup
 [root@master ~]# ./easzup -D
 [root@master ~]# ls /etc/ansible/
 .prepare.yml     .docker.yml       .network.yml        .upgrade.yml  .setup.yml  bin          down       pics       tools
 .etcd.yml        .kube-master.yml  .cluster-addon.yml  .backup.yml   .clean.yml  dockerfiles  example    README.md
 .containerd.yml  .kube-node.yml    .harbor.yml         .restore.yml  ansible.cfg   docs         manifests  roles

7：配置hosts集群参数

 [root@master ~ ]# cd /etc/ansible
 [root@master ansible]# cp example/hosts.multi-node hosts
 [root@master ansible]# vim hosts
 [etcd]   ##设置etcd节点ip
 192.168.1.200 NODE_NAME=etcd1
 192.168.1.202 NODE_NAME=etcd2
 192.168.1.203 NODE_NAME=etcd3

 [kube-master]   ##设置master节点ip
 192.168.1.200
 192.168.1.201

 [kube-node]  ##设置node节点ip
 192.168.1.202
 192.168.1.203

 [ex-lb]  ##设置lb节点ip和VIP
 192.168.1.200 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=
 192.168.1.201 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT= 

8：修改完hosts，测试连通性

 [root@master ansible]# ansible all -m ping
 [DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user
 configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

 192.168.1.201 | SUCCESS => {
     "ansible_facts": {
         "discovered_interpreter_python": "/usr/bin/python"
     },
     "changed": false,
     "ping": "pong"
 }
 192.168.1.202 | SUCCESS => {
     "ansible_facts": {
         "discovered_interpreter_python": "/usr/bin/python"
     },
     "changed": false,
     "ping": "pong"
 }
 192.168.1.203 | SUCCESS => {
     "ansible_facts": {
         "discovered_interpreter_python": "/usr/bin/python"
     },
     "changed": false,
     "ping": "pong"
 }
 192.168.1.200 | SUCCESS => {
     "ansible_facts": {
         "discovered_interpreter_python": "/usr/bin/python"
     },
     "changed": false,
     "ping": "pong"
 }

二、开始部署集群

【deploy节点操作】手动安装方式

1：安装ca证书

[root@master ansible]# ansible-playbook .prepare.yml

2：安装etcd

[root@master ansible]# ansible-playbook .etcd.yml

检查etcd健康状态，显示healthy: successfully表示节点正常

 [root@master ansible]# for ip in    ; do ETCDCTL_API= etcdctl --endpoints=https://192.168.1.$ip:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint healt; done
 https://192.168.1.200:2379 is healthy: successfully committed proposal: took = 5.658163ms
 https://192.168.1.202:2379 is healthy: successfully committed proposal: took = 6.384588ms
 https://192.168.1.203:2379 is healthy: successfully committed proposal: took = 7.386942ms

3：安装docker

[root@master ansible]# ansible-playbook .docker.yml

4：安装master

  [root@master ansible]# ansible-playbook .kube-master.yml 

查看集群状态

 [root@master ansible]# kubectl get componentstatus
 NAME                 STATUS    MESSAGE             ERROR
 scheduler            Healthy   ok
 controller-manager   Healthy   ok
 etcd-               Healthy   {"health":"true"}
 etcd-               Healthy   {"health":"true"}
 etcd-               Healthy   {"health":"true"}

5：安装node节点

[root@master ansible]# ansible-playbook .kube-node.yml

查看node节点

 [root@master ansible]# kubectl get nodes
 NAME            STATUS                     ROLES    AGE     VERSION
 192.168.1.200   Ready,SchedulingDisabled   master   4m45s   v1.15.0
 192.168.1.201   Ready,SchedulingDisabled   master   4m45s   v1.15.0
 192.168.1.202   Ready                      node     12s     v1.15.0
 192.168.1.203   Ready                      node     12s     v1.15.0

6：部署集群网络

  [root@master ansible]# ansible-playbook .network.yml

查看kube-system namespace上的pod，从中可以看到flannel相关的pod

 [root@master ansible]# kubectl get pod -n kube-system
 NAME                          READY   STATUS    RESTARTS   AGE
 kube-flannel-ds-amd64-7bk5w   /     Running             61s
 kube-flannel-ds-amd64-blcxx   /     Running             61s
 kube-flannel-ds-amd64-c4sfx   /     Running             61s
 kube-flannel-ds-amd64-f8pnz   /     Running             61s

7：安装集群插件

[root@master ansible]# ansible-playbook .cluster-addon.yml

查看kube-system namespace下的服务

 [root@master ansible]# kubectl get svc -n kube-system
 NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                       AGE
 heapster                  ClusterIP   10.68.191.0     <none>        /TCP                        13m
 kube-dns                  ClusterIP   10.68.0.2       <none>        /UDP,/TCP,/TCP        15m
 kubernetes-dashboard      NodePort    10.68.115.45    <none>        :/TCP                 13m
 metrics-server            ClusterIP   10.68.116.163   <none>        /TCP                       15m
 traefik-ingress-service   NodePort    10.68.106.241   <none>        :/TCP,:/TCP   12m

【自动安装方式】

一步执行上面所有手动安装操作

[root@master ansible]# ansible-playbook .setup.yml

查看node/pod使用资源情况

 [root@master ansible]# kubectl top node
 NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
 192.168.1.200   58m          %     960Mi           %
 192.168.1.201   34m          %     1018Mi          %
 192.168.1.202   76m          %     549Mi           %
 192.168.1.203   89m          %    568Mi           % 

 [root@master ansible]# kubectl top pod --all-namespaces
 NAMESPACE     NAME                                          CPU(cores)   MEMORY(bytes)
 kube-system   coredns-797455887b-9nscp                      5m           22Mi
 kube-system   coredns-797455887b-k92wv                      5m           19Mi
 kube-system   heapster-5f848f54bc-vvwzx                     1m           11Mi
 kube-system   kube-flannel-ds-amd64-7bk5w                   3m           20Mi
 kube-system   kube-flannel-ds-amd64-blcxx                   2m           19Mi
 kube-system   kube-flannel-ds-amd64-c4sfx                   2m           18Mi
 kube-system   kube-flannel-ds-amd64-f8pnz                   2m           10Mi
 kube-system   kubernetes-dashboard-5c7687cf8-hnbdp          1m           22Mi
 kube-system   metrics-server-85c7b8c8c4-6q4vj               1m           16Mi
 kube-system   traefik-ingress-controller-766dbfdddd-98trv   4m           17Mi

查看集群信息

 [root@master ansible]# kubectl cluster-info
 Kubernetes master is running at https://192.168.1.200:6443
 CoreDNS is running at https://192.168.1.200:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
 kubernetes-dashboard is running at https://192.168.1.200:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy
 Metrics-server is running at https://192.168.1.200:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

：测试DNS

①创建一个nginx.service

 [root@master ansible]# kubectl run nginx --image=nginx --expose --port=
 kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
 service/nginx created
 deployment.apps/nginx created

②创建busybox测试pod，可以看到nginx监听的虚拟地址10.68.243.55

 [root@master ansible]# kubectl run busybox --rm -it --image=busybox /bin/sh
 kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
 If you don't see a command prompt, try pressing enter.
 / # nslookup nginx.default.svc.cluster.local
 Server:    10.68.0.2
 Address:    10.68.0.2:

 Name:    nginx.default.svc.cluster.local
 Address: 10.68.243.55

三、增加node节点，IP：192.168.1.204

【deploy节点操作】

1：拷贝公钥到新的node节点机器上

[root@master ansible]# ssh-copy-id 192.168.1.204

2：修改hosts文件，添加新的node节点IP

[root@master ansible]# vim hosts
[kube-node]
192.168.1.202
192.168.1.203
192.168.1.204

3：执行添加node安装文件，并指导节点的IP

[root@master ansible]# ansible-playbook tools/.addnode.yml -e NODE_TO_ADD=192.168.1.204 

 4：验证node节点是否添加成功

 [root@master ansible]# kubectl get node
 NAME            STATUS                     ROLES    AGE     VERSION
 192.168.1.200   Ready,SchedulingDisabled   master   9h      v1.15.0
 192.168.1.201   Ready,SchedulingDisabled   master   9h      v1.15.0
 192.168.1.202   Ready                      node     9h      v1.15.0
 192.168.1.203   Ready                      node     9h      v1.15.0
 192.168.1.204   Ready                      node     2m11s   v1.15.0 

 [root@master ansible]# kubectl get pod -n kube-system -o wide
 NAME                                          READY   STATUS    RESTARTS   AGE   IP              NODE            NOMINATED NODE   READINESS GATES
 coredns-797455887b-9nscp                      /     Running             31h   172.20.3.2      192.168.1.203   <none>           <none>
 coredns-797455887b-k92wv                      /     Running             31h   172.20.2.2      192.168.1.202   <none>           <none>
 heapster-5f848f54bc-vvwzx                     /     Running             31h   172.20.2.4      192.168.1.202   <none>           <none>
 kube-flannel-ds-amd64-7bk5w                   /     Running             31h   192.168.1.202   192.168.1.202   <none>           <none>
 kube-flannel-ds-amd64-blcxx                   /     Running             31h   192.168.1.200   192.168.1.200   <none>           <none>
 kube-flannel-ds-amd64-c4sfx                   /     Running             31h   192.168.1.203   192.168.1.203   <none>           <none>
 kube-flannel-ds-amd64-f8pnz                   /     Running             31h   192.168.1.201   192.168.1.201   <none>           <none>
 kube-flannel-ds-amd64-vdd7n                   /     Running             21h   192.168.1.204   192.168.1.204   <none>           <none>
 kubernetes-dashboard-5c7687cf8-hnbdp          /     Running             31h   172.20.3.3      192.168.1.203   <none>           <none>
 metrics-server-85c7b8c8c4-6q4vj               /     Running             31h   172.20.2.3      192.168.1.202   <none>           <none>
 traefik-ingress-controller-766dbfdddd-98trv   /     Running             31h   172.20.3.4      192.168.1.203   <none>           <none>