kubernetes 集群部署

环境
JiaoJiao_Centos7-1(152.112) 192.168.152.112
JiaoJiao_Centos7-2(152.113) 192.168.152.113
JiaoJiao_Centos7-3(152.114) 192.168.152.114
已开通 4C+8G+80G

集群规划

部署方式

环境准备：基于主机名称通信，时间同步，关闭firewall和iptables.service

方式一：yum ,rpm 安装。复杂。

1. etcd cluster ：安装在master节点上2. flannel：安装在所有节点上
2. 安装配置master
kube-apiserver,kube-scheduler,kube-controller-manager
3. 安装配置node
kube-proxy
kubelet

方式二：使用kubeadm工具安装。简单。

1. master和node 都用yum 安装kubelet，kubeadm，docker
2. master 上初始化：kubeadm init
3. master 上启动一个flannel的pod
4. node上加入集群:kubeadm join

部署步骤

采取方式二：使用kubeadm工具安装步骤

准备环境

修改主机名（3台机器都需要修改）

# hostnamectl set-hostname master

配置hosts和dns解析（3台机器都需要配置）

[root@node02 ~]# cat /etc/hosts

127.0.0.1 node02 localhost localhost.localdomain localhost4 localhost4.localdomain4

:: node02 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.152.112 master

192.168.152.113 node01

192.168.152.114 node02

默认的dns服务器没法解析到opsx.alibaba.com和mirrors.aliyun.com，使用nameserver 192.168.?.?（具体根据你所在公司环境设置）这个可以。

[root@master ~]# cat /etc/resolv.conf

# Generated by NetworkManager

nameserver 192.168.?.?

关闭防火墙和iptables，检查iptables配置值是否为1（3台机器都需要）

# systemctl stop firewalld.service

# systemctl stop iptables.service

# systemctl disable firewalld.service

[root@master docker]# cat /proc/sys/net/bridge/bridge-nf-call-ip6tables 

[root@master docker]# cat /proc/sys/net/bridge/bridge-nf-call-iptables

配置docker yum源

#yum update

#yum install -y yum-utils device-mapper-persistent-data lvm2 wget

#cd /etc/yum.repos.d

#wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

配置kubernetes yum 源

#wget https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg

# wget https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

# rpm --import rpm-package-key.gpg

# rpm --import yum-key.gpg

[root@master yum.repos.d]# cat kubernetes.repo

[kubernetes]

name=Kubernetes Repo

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/

gpgcheck=

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg

enabled=

# yum repolist

master和node 安装kubelet，kubeadm，docker

# yum install docker-ce kubelet kubeadm

master 上安装kubectl ，node节点可以选择安装或者不安装

kubectl 是API server 客户端程序，通过连接master端的API server来实现k8s对象资源的增删改查等基本操作。

#yum install kubectl

docker的配置

配置私有仓库和镜像加速地址

[root@master docker]# cat /etc/docker/daemon.json

{"registry-mirrors": ["http://295c6a59.m.daocloud.io"],

"insecure-registries":["自己搭建的私有仓库地址"]}

设置代理

[root@master docker]# cat /usr/lib/systemd/system/docker.service |grep -v "^#"

[Unit]

Description=Docker Application Container Engine

Documentation=https://docs.docker.com

BindsTo=containerd.service

After=network-online.target firewalld.service containerd.service

Wants=network-online.target

Requires=docker.socket

[Service]

Type=notify

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

ExecReload=/bin/kill -s HUP $MAINPID

TimeoutSec=

RestartSec=

Restart=always

#设置代理地址，和不让代理的地址

Environment="HTTPS_PROXY=http://www.ik8s.io:10080" "NO_PROXY=localhost,127.0.0.1,192.168.152.0/24"

StartLimitBurst=

StartLimitInterval=60s

LimitNOFILE=infinity

LimitNPROC=infinity

LimitCORE=infinity

TasksMax=infinity

Delegate=yes

KillMode=process

[Install]

WantedBy=multi-user.target

启动docker

# systemctl daemon-reload

# systemctl start docker

[root@master docker]# docker info

Containers:

Running:

Paused:

Stopped:

Images:

Server Version: 18.09.

Storage Driver: overlay2

Backing Filesystem: xfs

Supports d_type: true

Native Overlay Diff: false

Logging Driver: json-file

Cgroup Driver: cgroupfs

Plugins:

Volume: local

Network: bridge host macvlan null overlay

Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog

Swarm: inactive

Runtimes: runc

Default Runtime: runc

Init Binary: docker-init

containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84

runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30

init version: fec3683

Security Options:

seccomp

Profile: default

Kernel Version: 3.10.-.el7.x86_64

Operating System: CentOS Linux  (Core)

OSType: linux

Architecture: x86_64

CPUs:

Total Memory: .64GiB

Name: master

ID: KEBQ:4XXU:LTA7:J5QH:HK4F:JS26:RKI4:YFPJ:RY45:Q647:SJI6:VZDQ

Docker Root Dir: /var/lib/docker

Debug Mode (client): false

Debug Mode (server): false

HTTPS Proxy: http://www.ik8s.io:10080 #代理地址

No Proxy: localhost,127.0.0.1,192.168.152.0/

Registry: https://index.docker.io/v1/

Labels:

Experimental: false

Insecure Registries:

XXXXX #本公司私有库地址

127.0.0.0/

Registry Mirrors:

http://295c6a59.m.daocloud.io/ #镜像加速地址

Live Restore Enabled: false

Product License: Community Engine

# systemctl enable docker

master 上初始化：kubeadm init

修改配置文件，忽略swap报错

[root@master docker]# cat /etc/sysconfig/kubelet

KUBELET_EXTRA_ARGS="--fail-swap-on=false"

初始化

#kubeadm init [command] --help 可以先查看下帮助

# kubeadm init --kubernetes-version=v1.14.2 --pod-network-cidr=10.244.0.0/ --service-cidr=10.96.0.0/ --ignore-preflight-errors=Swap

--kubernetes-version=v1. # k8s的版本

版本通过：https://github.com/kubernetes/kubernetes/releases 来确定你需要的版本。

--pod-network-cidr=10.244.0.0/ #pod网络。flannel 的默认是10.244.0./

--service-cidr=10.96.0.0/ # service 网络，也是用了默认值

--ignore-preflight-errors=Swap #忽略Swap报错

[root@master ~]# kubeadm init --kubernetes-version=v1.14.2 --pod-network-cidr=10.244.0.0/ --service-cidr=10.96.0.0/ --ignore-preflight-errors=Swap

[init] Using Kubernetes version: v1.14.2

[preflight] Running pre-flight checks

[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/

[WARNING Swap]: running with swap on is not supported. Please disable swap

[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'

[preflight] Pulling images required for setting up a Kubernetes cluster

[preflight] This might take a minute or two, depending on the speed of your internet connection

[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

error execution phase preflight: [preflight] Some fatal errors occurred:

[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.14.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 172.96.236.117:10080: connect: connection refused

, error: exit status

[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.14.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 172.96.236.117:10080: connect: connection refused

, error: exit status

[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.14.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 172.96.236.117:10080: connect: connection refused

, error: exit status

报错了。拉不到镜像。
解决办法：docker.io仓库对google的容器做了镜像，可以通过下列命令下拉取相关镜像

#docker pull mirrorgooglecontainers/kube-apiserver:v1.14.2

#docker pull mirrorgooglecontainers/kube-controller-manager:v1.14.2

#docker pull mirrorgooglecontainers/kube-scheduler:v1.14.2

#docker pull mirrorgooglecontainers/kube-proxy:v1.14.2

#docker pull mirrorgooglecontainers/pause:3.1

#docker pull mirrorgooglecontainers/etcd:3.3.

#docker pull coredns/coredns:1.3.

通过docker tag命令来修改镜像的标签：

# docker tag mirrorgooglecontainers/kube-apiserver:v1.14.2 k8s.gcr.io/kube-apiserver:v1.14.2

# docker tag mirrorgooglecontainers/kube-controller-manager:v1.14.2 k8s.gcr.io/kube-controller-manager:v1.14.2

# docker tag mirrorgooglecontainers/kube-scheduler:v1.14.2 k8s.gcr.io/kube-scheduler:v1.14.2

# docker tag mirrorgooglecontainers/kube-proxy:v1.14.2 k8s.gcr.io/kube-proxy:v1.14.2

# docker tag mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1

# docker tag mirrorgooglecontainers/etcd:3.3. k8s.gcr.io/etcd:3.3.

# docker tag coredns/coredns:1.3. k8s.gcr.io/coredns:1.3.

初始化过程中有提示，请按照提示进行操作并且将下面内容保存下来。

kubeadm join 192.168.152.112: --token qlu35g.rfxl6k8mvry8nxfp \

> --discovery-token-ca-cert-hash sha256:8febc1b1402b471187e55186b4c3937ac23cfe830fb6ccfbd2e9f212e011218d

部署网络：master 上启动一个flannel的pod

下载镜像。由于不能直接下载官方镜像，自己百度找了个地址：

#docker pull jmgao1983/flannel:v0.11.0-amd64

#docker tag jmgao1983/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64

#docker rmi jmgao1983/flannel:v0.11.0-amd64

启动：

# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

检查下：

[root@master ~]# kubectl get pods -n kube-system

NAME READY STATUS RESTARTS AGE

coredns-fb8b8dccf-f7db6 / Running  53m

coredns-fb8b8dccf-sd8hk / Running  53m

etcd-master / Running  52m

kube-apiserver-master / Running  52m

kube-controller-manager-master / Running  52m

kube-flannel-ds-amd64-zq5zr / Running  19m

kube-proxy-8mtbc / Running  53m

kube-scheduler-master / Running  52m

到此，master 就安装好了! 保存镜像

docker save -o master.tar k8s.gcr.io/kube-proxy:v1.14.2 k8s.gcr.io/kube-apiserver:v1.14.2 k8s.gcr.io/kube-controller-manager:v1.14.2 k8s.gcr.io/kube-scheduler:v1.14.2 quay.io/coreos/flannel:v0.11.0-amd64 k8s.gcr.io/coredns:1.3. k8s.gcr.io/etcd:3.3. k8s.gcr.io/pause:3.1

node节点加入集群

将master上保存的镜像同步到节点上

[root@master ~]#scp master.tar node01:/root/

[root@master ~]#scp master.tar node02:/root/

将镜像导入本地

[root@node01 ~]# docker load< master.tar

修改配置，忽略报错：

[root@node01 docker]# cat /etc/sysconfig/kubelet

KUBELET_EXTRA_ARGS="--fail-swap-on=false"

执行加入集群命令

[root@node01 ~]# kubeadm join 192.168.152.112: --token qlu35g.rfxl6k8mvry8nxfp --discovery-token-ca-cert-hash sha256:8febc1b1402b471187e55186b4c3937ac23cfe830fb6ccfbd2e9f212e011218d --ignore-preflight-errors=Swap

在master上查看节点情况

[root@master ~]# kubectl get nodes

NAME STATUS ROLES AGE VERSION

master Ready master 107m v1.14.2

node01 Ready <none> 23m v1.14.2

node02 Ready <none> 23m v1.14.2