k8s搭建

K8s官方文档地址:https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/  如果用云主机部署，一定要提前开启端口

1. 服务器规划

角色	IP	组件
k8s-master1	192.168.31.63	kube-apiserver kube-controller-manager kube-scheduler etcd
k8s-master2	192.168.31.64	kube-apiserver kube-controller-manager kube-scheduler
k8s-node1	192.168.31.65	kubelet kube-proxy docker etcd
k8s-node2	192.168.31.66	kubelet kube-proxy docker etcd
Load Balancer（Master）	192.168.31.61 192.168.31.60 (VIP)	Nginx L4
Load Balancer（Backup）	192.168.31.62	Nginx L4

1.系统初始化

修改主机名称：

hostnamectl set-hostname k8s-master1

关闭防火墙：

# systemctl stop firewalld

# systemctl disable firewalld

关闭selinux：

# setenforce 0 # 临时

# sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久

关闭swap：

# swapoff -a  # 临时

# vim /etc/fstab  # 永久

同步系统时间：

# ntpdate time.windows.com

2.2 部署三个Etcd节点   

TLS、etcd地址:

链接：https://pan.baidu.com/s/1kyC5KgsF5DB2fZK5UGPaQg
提取码：o101

# tar zxvf etcd.tar.gz

# cd etcd

# cp TLS/etcd/ssl/{ca,server,server-key}.pem ssl

分别拷贝到Etcd三个节点：

# scp –r etcd root@192.168.31.63:/opt

# scp etcd.service root@192.168.31.63:/usr/lib/systemd/system/

登录三个节点修改配置文件 名称和IP：

# vi /opt/etcd/cfg/etcd.conf

#[Member]

ETCD_NAME="etcd-1"  名称一定要替换

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="https://192.168.31.63:2380"  内网ip

ETCD_LISTEN_CLIENT_URLS="https://192.168.31.63:2379"

#[Clustering]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.31.63:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.31.63:2379"

ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.31.63:2380,etcd-2=https://192.168.31.64:2380,etcd-3=https://192.168.31.65:2380"   部署3个节点的内网ip

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"  集群状态

# systemctl daemon-reload

# systemctl start etcd

# ps -ef|grep etcd 查看etcd进程

# systemctl enable etcd 设置开机启动

# tail /var/log/messages -f 查看系统日志

2.3 查看集群状态

# /opt/etcd/bin/etcdctl \

> --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \

> --endpoints="https://192.168.31.63:2379,https://192.168.31.64:2379,https://192.168.31.65:2379" \    部署3个节点的内网ip一定要替换

> cluster-health

如果出现下面字段，说明集群状态是健康的

member 37f20611ff3d9209 is healthy: got healthy result from https://192.168.31.63:2379

member b10f0bac3883a232 is healthy: got healthy result from https://192.168.31.64:2379

member b46624837acedac9 is healthy: got healthy result from https://192.168.31.65:2379

cluster is healthy

1.部署Master Node

1.1 生成apiserver证书

# cd TLS/k8s

修改请求文件中hosts字段包含所有etcd节点IP：

# vi server-csr.json

{

"CN": "kubernetes",

"hosts": [

"10.0.0.1",

"127.0.0.1",

"kubernetes",

"kubernetes.default",

"kubernetes.default.svc",

"kubernetes.default.svc.cluster",

"kubernetes.default.svc.cluster.local",

"192.168.31.60",    你的内网ip

"192.168.31.61",

"192.168.31.62",

"192.168.31.63",

"192.168.31.64",

"192.168.31.65",

"192.168.31.66"

],

"key": {

"algo": "rsa",

"size": 2048

},

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing",

"O": "k8s",

"OU": "System"

}

]

}

# ./generate_k8s_cert.sh

# ls *pem

ca-key.pem  ca.pem  kube-proxy-key.pem  kube-proxy.pem  server-key.pem  server.pem

3.2 部署apiserver，controller-manager和scheduler

在Master节点完成以下操作。

二进制包下载地址：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md#v1161

master安装包下载地址:链接：https://pan.baidu.com/s/1kyC5KgsF5DB2fZK5UGPaQg 
提取码：o101

二进制文件位置：kubernetes/serverr/bin

# tar zxvf k8s-master.tar.gz

# cd kubernetes

# cp TLS/k8s/ssl/*.pem ssl

# cp  –r  kubernetes /opt

# cp kube-apiserver.service kube-controller-manager.service kube-scheduler.service /usr/lib/systemd/system

# cat /opt/kubernetes/cfg/kube-apiserver.conf

KUBE_APISERVER_OPTS="--logtostderr=false \

--v=2 \

--log-dir=/opt/kubernetes/logs \

--etcd-servers=https://192.168.31.63:2379,https://192.168.31.64:2379,https://192.168.31.65:2379 \  替换etcd节点的内网ip

--bind-address=192.168.31.63 \ 替换master节点的ip

--secure-port=6443 \

--advertise-address=192.168.31.63 \ 替换master节点的ip

……

# systemctl start kube-apiserver

# systemctl start kube-controller-manager

# systemctl start kube-scheduler

# systemctl enable kube-apiserver

# systemctl enable kube-controller-manager

# systemctl enable kube-scheduler

# systemctl  start kube-apiserver

# ls /opt/kubernetes/logs  查看日志

# less /opt/kubernetes/logs/kube-apiserver.INFO

# tail -f /opt/kubernetes/logs/kube-controller-manager.INFO

# for i in $(ls /opt/kubernetes/bin);do systemctl enable $i;done开机启动

# mv /opt/kubernetes/bin/kubectl  /usr/local/bin/kubectl 移动到环境变量

# chmod a+x /usr/local/bin/kubect

# kubectl get cs查看组件状态

# 查看3个组件的进程 ps -ef|grep kube

3.3 启用TLS Bootstrapping

为kubelet TLS Bootstrapping 授权：

# cat /opt/kubernetes/cfg/token.csv

c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"

格式：token,用户,uid,用户组

给kubelet-bootstrap授权：

kubectl create clusterrolebinding kubelet-bootstrap \

--clusterrole=system:node-bootstrapper \

--user=kubelet-bootstrap

token也可自行生成替换：

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

但apiserver配置的token必须要与node节点bootstrap.kubeconfig配置里一致。

1.部署Worker Node

1.1 安装Docker

二进制包下载地址：https://download.docker.com/linux/static/stable/x86_64/

docker下载地址: 链接：https://pan.baidu.com/s/1kyC5KgsF5DB2fZK5UGPaQg 
提取码：o101

# tar zxvf k8s-node.tar.gz

# tar zxvf docker-18.09.6.tgz

# mv docker/* /usr/bin

# mkdir /etc/docker

# mv daemon.json /etc/docker

# mv docker.service /usr/lib/systemd/system

# systemctl start docker

# systemctl enable docker

# docker info 通过docker info查看docker是否启动成功

执行docker info出现如下警告

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

解决办法：

vi /etc/sysctl.conf

添加以下内容

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

最后再执行

sysctl -p

此时docker info就看不到此报错了

执行docker info出现如下警告

4.2 部署kubelet和kube-proxy

拷贝证书到Node：

# cd TLS/k8s

# scp ca.pem kube-proxy*.pem root@192.168.31.65:/opt/kubernetes/ssl/

# cp kube-apiserver.service kube-controller-manager.service kube-

# tar zxvf k8s-node.tar.gz

# mv kubernetes /opt

# cp kubelet.service kube-proxy.service /usr/lib/systemd/system

查看以下三个文件中IP地址：

[root@k8s-node2 kubernetes]# grep 192 *

修改以下两个文件中主机名：

[root@k8s-node2 cfg]# vim bootstrap.kubeconfig

[root@k8s-node2 cfg]# vim kubelet.conf

[root@k8s-node2 cfg]# vim kubelet.kubeconfig

[root@k8s-node2 cfg]# vim kube-proxy-config.yml

[root@k8s-node2 cfg]# vim kube-proxy.kubeconfig

# systemctl start kubelet

# systemctl start kube-proxy

# systemctl enable kubelet

# systemctl enable kube-proxy

# tail /opt/kubernetes/logs/kubelet.INFO 查看日志

4.3 允许给Node颁发证书

# kubectl get csr

# kubectl certificate approve node-csr-MYUxbmf_nmPQjmH3LkbZRL2uTO-_FCzDQUoUfTy7YjI  替换你的node名称

# kubectl get node