kubernetes的搭建（一）

集群的搭建

集群的类型

kubunetes的集群类型大致上分为两类：一主多从和多主多从。

一主多从：一台master节点和多台node节点，搭建简单，但是有单机故障的风险，适用于测试环境
多主多从：多台master节点和多台node节点，搭建麻烦，安全性高，适用于生产环境

为了测试简单，本次搭建的是：一主两从

安装方式

kubernetes有多种部署方式，目前主流的方式有如下：

minikube: 一个用于快速搭建单节点kubernetes的工具
kubeadm：一个用于快速搭建kubernetes集群的工具
二进制包：从官网下载每个组件的二进制包，依次去安装，次方式对与理解kubernetes组件更加有效

安装前准备

主机规划：

作用	ip	操作系统	配置
master	192.168.109.100	centos7 基础设施服务器	2 cpu 2G内存 50G硬盘
node1	192.168.109.101	centos7 基础设施服务器	2 cpu 2G内存 50G硬盘
node2	192.168.109.102	centos7 基础设施服务器	2 cpu 2G内存 50G硬盘

搭建基础环境

配置网卡ip和yum源

查看系统版本

[root@master ~]# cat /etc/redhat-release

CentOS Linux release 7.5.1804 (Core)

[root@node1 ~]# cat /etc/redhat-release

CentOS Linux release 7.5.1804 (Core)

[root@node2 ~]# cat /etc/redhat-release

CentOS Linux release 7.5.1804 (Core)

配置域名解析

master

[root@master ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.109.100 master

192.168.109.101 node1

192.168.109.102 node2

node1

[root@node1 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.109.100 master

192.168.109.101 node1

192.168.109.102 node2

node2

[root@node2 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.109.100 master

192.168.109.101 node1

192.168.109.102 node2

配置时间同步

启动chronyd服务并且设置开机自启

master

[root@master ~]# systemctl start  chronyd

[root@master ~]# systemctl enable chronyd

node1

[root@node1 ~]# systemctl start chronyd

[root@node1 ~]# systemctl enable chronyd

node2

[root@node2 ~]# systemctl start chronyd

[root@node2 ~]#

[root@node2 ~]# systemctl enable chronyd

禁用iptable和firewalld服务

1.关闭firewalld

master

[root@master ~]# systemctl enable chronyd

[root@master ~]# systemctl stop firewalld.service

[root@master ~]# systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon

Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)

Active: inactive (dead) since 三 2024-05-01 14:52:19 CST; 51s ago

Docs: man:firewalld(1)

Process: 717 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)

Main PID: 717 (code=exited, status=0/SUCCESS)

5月 01 13:51:29 master systemd[1]: Starting firewalld - dynamic fi....

月 01 13:51:30 master systemd[1]: Started firewalld - dynamic fir....

月 01 14:52:16 master systemd[1]: Stopping firewalld - dynamic fi....

5月 01 14:52:19 master systemd[1]: Stopped firewalld - dynamic fir....

Hint: Some lines were ellipsized, use -l to show in full.

[root@master ~]# systemctl disable firewalld.service

Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

node1

[root@node1 ~]# systemctl stop firewalld.service

[root@node1 ~]# systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon

Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)

Active: inactive (dead) since 三 2024-05-01 14:52:20 CST; 51s ago

 Docs: man:firewalld(1)

Process: 724 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)

Main PID: 724 (code=exited, status=0/SUCCESS)

5月 01 13:51:37 master systemd[1]: Starting firewalld - dynamic firewall daemon...

5月 01 13:51:37 master systemd[1]: Started firewalld - dynamic firewall daemon.

5月 01 14:52:16 node1 systemd[1]: Stopping firewalld - dynamic firewall daemon...

5月 01 14:52:20 node1 systemd[1]: Stopped firewalld - dynamic firewall daemon.

[root@node1 ~]# systemctl disable firewalld.service

Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

node2

[root@node2 ~]# systemctl stop firewalld.service

[root@node2 ~]# systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon

Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)

Active: inactive (dead) since 三 2024-05-01 14:52:19 CST; 52s ago

 Docs: man:firewalld(1)

Process: 724 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)

Main PID: 724 (code=exited, status=0/SUCCESS)

5月 01 13:51:40 master systemd[1]: Starting firewalld - dynamic firewall daemon...

5月 01 13:51:41 master systemd[1]: Started firewalld - dynamic firewall daemon.

5月 01 14:52:16 node2 systemd[1]: Stopping firewalld - dynamic firewall daemon...

5月 01 14:52:19 node2 systemd[1]: Stopped firewalld - dynamic firewall daemon.

[root@node2 ~]# systemctl disable firewalld.service

Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

关闭iptables

当前虚拟机并没有iptables服务，所以不用关闭

比如：

[root@master ~]# systemctl stop  iptables

Failed to stop iptables.service: Unit iptables.service not loaded.

禁用selinux

master

[root@master ~]# sed -i 's/'SELINUX=enforcing'/'SELINUX=disabled/g'' /etc/selinux/config

node1

[root@node1 ~]# sed -i 's/'SELINUX=enforcing'/'SELINUX=disabled/g'' /etc/selinux/config

node2

[root@node2 ~]# sed -i 's/'SELINUX=enforcing'/'SELINUX=disabled/g'' /etc/selinux/config

禁止使用swap分区

kubernetes不支持swap分区，所以需要禁止使用swap分区

直接将/etc/fstab文件中的swap相关配置注释掉

master

[root@master ~]# cat /etc/fstab

#

# /etc/fstab

# Created by anaconda on Wed May  1 21:37:04 2024

#

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

#

/dev/mapper/centos-root /                       xfs     defaults        0 0

UUID=d97390d4-c671-411c-9371-015002de02a5 /boot                   xfs     defaults        0 0

#/dev/mapper/centos-swap swap                    swap    defaults        0 0

[root@master ~]#

node1

[root@node1 ~]# cat /etc/fstab

#

# /etc/fstab

# Created by anaconda on Wed May  1 21:37:04 2024

#

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

#

/dev/mapper/centos-root /                       xfs     defaults        0 0

UUID=d97390d4-c671-411c-9371-015002de02a5 /boot                   xfs     defaults        0 0

#/dev/mapper/centos-swap swap                    swap    defaults        0 0

node2

[root@node2 ~]# cat /etc/fstab

#

# /etc/fstab

# Created by anaconda on Wed May  1 21:37:04 2024

#

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

#

/dev/mapper/centos-root /                       xfs     defaults        0 0

UUID=d97390d4-c671-411c-9371-015002de02a5 /boot                   xfs     defaults        0 0

#/dev/mapper/centos-swap swap                    swap    defaults        0 0

修改linux内核参数,添加网桥过滤和地址转发功能

master

[root@master ~]# cat /etc/  sysctl.d/kubernetes.conf

net.bridge. bridge-nf-call-ip6tables     = 1

net.bridge. bridge-nf-call-iptables =    1

net.ipv4.ip_forward = 1

[root@master ~]#

node1

[root@node1 ~]# cat /etc/   sysctl.d/kubernetes.conf

net.bridge. bridge-nf-call-ip6tables     = 1

net.bridge. bridge-nf-call-iptables =    1

net.ipv4.ip_forward = 1

node2

[root@node2 ~]# cat /etc/   sysctl.d/kubernetes.conf

net.bridge. bridge-nf-call-ip6tables     = 1

net.bridge. bridge-nf-call-iptables =    1

net.ipv4.ip_forward = 1

重新加载配置和加载网桥过滤模块

master

[root@master ~]#       sysctl -p

[root@master ~]#       modprobe br_netfilter

node1

[root@node1 ~]#        sysctl -p

[root@node1 ~]#        modprobe           br_netfilter

node2

[root@node2 ~]#        sysctl -p

[root@node2 ~]#        modprobe           br_netfilter

查看是否加载成功,使用如下命令
```
lsmod |grep br_netfil
```

配置ipvs功能

在kubernetes中service有两种代理模型，一种是基于iptables的，一种是基于ipvs的。两者比较的话，ipvs的性能，明显要高一些，但是如果要是用它，需要手动载入ipvs模块

安装ipset和ipvsadmin （如果没有指定虚拟机，三台同样的操作）

[root@master ~]# yum -y     install ipset ipvsadmin

添加需要加载的模块写入脚本文件

[root@master ~]# cat /etc/  sysconfig/modules/ipvs.   modules

#!/bin/bash

modprobe -- ip_vs

modprobe -- ip_vs_rr

modprobe -- ip_vs_wrr

modprobe -- ip_vs_sh

modprobe --     nf_conntrack_ipv4

设置脚本文件可执行权限

[root@master ~]# chmod  +x /etc/sysconfig/modules/   ipvs.modules

执行脚本文件

[root@master ~]# /bin/  bash /etc/sysconfig/  modules/ipvs.modules

查看是否加载成功

[root@master ~]# lsmod  |   grep -e ip_vs -e   nf_conntrack_ipv4

nf_conntrack_ipv4       15053  0

nf_defrag_ipv4          12729  1 nf_conntrack_ipv4

ip_vs_sh                12688  0

ip_vs_wrr               12697  0

ip_vs_rr                12600  0

ip_vs                   141432  6 ip_vs_rr,   ip_vs_sh,ip_vs_wrr

nf_conntrack            133053  2 ip_vs,  nf_conntrack_ipv4

libcrc32c               12644  3 xfs,ip_vs,  nf_conntrack

重启虚拟机

reboot

检查swap分区和selinux

[root@master ~]# free -m

              total        used        free      shared  buff/cache   available

Mem:           1982         100        1752           9         130        1726

Swap:             0           0           0

[root@master ~]# getenforce

Disabled

[root@master ~]#

环境搭建（二）

安装docker

切换镜像源

  [root@master yum.repos.d]# wget docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

查看当前镜像源中支持的docker版本

yum list docker-ce --showduplicates

安装指定版本的docker

ps: 安装指定版本的docker必须使用--setopt=obsoletes=1选项，否则yum会自动安装更高版本
```
[root@master yum.repos.d]   # yum install  --setopt=obsoletes=0     docker-ce-18.06.2.ce-3. el7 -y
```

添加一个配置文件,设置docker开机自启

docker在默认情况下使用cgroupdiver为cgroupfs，而kubernetes推荐使用systemd来代替cgroupfs，所以需要修改配置文件

[root@master ~]# mkdir /    etc/docker

[root@master ~]# vim /etc/  docker/daemon.json

[root@master ~]#    systemctl restart docker.  service

[root@master ~]#    systemctl enable docker

Created symlink from /etc/  systemd/system/multi-user.    target.wants/docker.    service to /usr/lib/    systemd/system/docker.  service.

[root@master ~]# docker     --version

Docker version 18.06.   2-ce, build 6d37f41

[root@master ~]#

安装kubernetes

配置yum源

[root@master yum.repos.d]   # cat k8s.repo

[kubernetes]

name=kubernetes

baseurl=http://mirrors.   aliyun.com/kubernetes/ yum/repos/   kubernetes-el7-x86_64

enabled=1

gpgcheck=0

repo_gpgcheck=0

gpgkey=http://mirrors.    aliyun.com/kubernetes/  yum/doc/yum-key.gpg

       http://mirrors.    aliyun.com/ kubernetes/yum/  doc/  rpm-package-key.  gpg

 //yum仓库的配置文件一定要顶格写

安装kubelet、kubeadm、kubectl

[root@master yum.repos.d]# yum -y install --setopt=obsoletes=0 kubeadm-1.17.4-0 kubelet-1.17.4-0 kubectl-1.17.4-0 -y

配置kubelet的cgroup

[root@master yum.repos.d]# cat /etc/sysconfig/kubelet

KUBELET_CGROUP_ARGS="--cgroup-drive=systemd"

KUBE_PROXT_MODE="ipvs"

//手动指定使用ipvs代理

设置kubelet开机自启

[root@master yum.repos.d]# systemctl enable kubelet.service

Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

[root@master yum.repos.d]#  //后面配置集群启动的时候，会自动启动kubelet，不需要手动启动

准备集群镜像

#在安装集群之前，必须提前准备好集群需要的镜像，所需镜像可以使用·kubeadm config images list·命令查看

# 下载镜像

# 次镜像在k8s的仓库中，由于网络问题，无法连接，所以尝试使用aliyun仓库中的镜像。然后改名成k8s的镜像，最后在删除阿里云的镜像

images=(

    kube-apiserver:v1.17.4

    kube-controller-manager:v1.17.4

    kube-scheduler:v1.17.4

    kube-proxy:v1.17.4

    pause:3.1

    etcd:3.4.3-0

    coredns:1.6.5

)

for imagename in ${images[@]}; do

      docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imagename

      docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imagename   k8s.gcr.io/$imagename

      docker  rmi  registry.cn-hangzhou.aliyuncs.com/google_containers/$imagename

done

docker images

docker images 查看出来7个镜像

[root@master yum.repos.d]# docker images

REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE

k8s.gcr.io/kube-proxy                v1.17.4             6dec7cfde1e5        4 years ago         116MB

k8s.gcr.io/kube-apiserver            v1.17.4             2e1ba57fe95a        4 years ago         171MB

k8s.gcr.io/kube-controller-manager   v1.17.4             7f997fcf3e94        4 years ago         161MB

k8s.gcr.io/kube-scheduler            v1.17.4             5db16c1c7aff        4 years ago         94.4MB

k8s.gcr.io/coredns                   1.6.5               70f311871ae1        4 years ago         41.6MB

k8s.gcr.io/etcd                      3.4.3-0             303ce5db0e90        4 years ago         288MB

k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        6 years ago         742kB

[root@master yum.repos.d]#

从这里开始只使用master节点

集群初始化

初始化，只需要在master上执行

kubeadm  init \

    --kubernetes-version=v1.17.4 \

    --pod-network-cidr=10.24 4.0.0/16 \

    --service-cidr=10.96.0.0/12 \

    --apiserver-advertise-address=192.168.200.128

看到successful表示成功

如果遇到报错，需要再次执行初始化，删除掉之前生成的重复文件，关闭服务。再次init执行
配置kubectl

 wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

 kubectl apply -f  kube-flannel.yml

 （kubectl delete -f kube-flannel.yml    删除）

测试（nginx）

kubectl create deployment nginx  --image=nginx:1.14-alpine

kubectl expose deployment nginx  --port=80  --type=NodePort

kubectl get pod

kubectl get service

curl http://192.168.109.101:31775