Kubernetes : 多节点 k8s 集群实践

需要说明的是, 这本书里有很多描述的东西和实践的案例不吻合.

Kubernets 集群架构

架构图

Server List

节点名称	节点 IP
k8s-master	10.10.10.10
k8s-node1	10.10.10.11
k8s-node2	10.10.10.12

配置主机 hosts

需要说明的是, 这个不是必须的. 但是如果不配置主机 hosts, 后续配置 kubelet 参数KUBELET_API_SERVER 以及 proxy 中参数 KUBE_PROXY_ARGS 的时候指定 api-server 的具体 IP 地址即可.

在三台节点上分别配置 hosts 文件:

/etc/hosts

10.10.10.10 k8s-master

10.10.10.11 k8s-node1

10.10.10.12 k8s-nodes

节点服务说明

Master : etcd , kube-apiserver , kube-controller-manager, kube-scheduler.

Node : docker, kubelet, kube-proxy

配置 Master

安装 kubernetes

yum install etcd kubernetes

关闭防火墙

systemctl stop firewalld

ssytemctl disable firewalld

sed -i "s/SELINUX=enforcing/SELINUX=disabled/1" /etc/selinux/config

setenforce  0

配置文件

/etc/kubernetes/apiserver

KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"

KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,,ResourceQuota"

KUBE_API_ARGS="--insecure-port=8080 --service-node-port-range=1-65535"

/etc/kubernetes/config

KUBE_LOGTOSTDERR="--logtostderr=false"

KUBE_LOG_LEVEL="--v=0"

KUBE_ALLOW_PRIV="--allow-privileged=false"

KUBE_MASTER="--master=http://127.0.0.1:8080"

/etc/kubernetes/controller-manager

KUBE_CONTROLLER_MANAGER_ARGS="--master=http://127.0.0.1:8080"

/etc/kubernetes/scheduler

KUBE_SCHEDULER_ARGS="--master=http://127.0.0.1:8080"

启动服务

kube-controller-manager 和 kube-scheduler 依赖 kube-apiserver , kube-apiserver 依赖 etcd. 所以启动顺序如下:

systemctl start etcd

systemctl start kube-apiserver

systemctl start kube-controller-manager

systemctl start kube-scheduler

systemctl enable etcd

systemctl enable kube-apiserver

systemctl enable kube-controller-manager

systemctl enable kube-scheduler

或者在 shell CLI 如下启动:

for svc in etcd kube-apiserver kube-controller-manager kube-scheduler; do systemctl start $svc; systemctl enable $svc;done

至此, master 节点已经配置完成了.

配置 Node

安装 kubernetes 和 Docker

yum install kubernetes docker

关闭防火墙

systemctl stop firewalld

ssytemctl disable firewalld

sed -i "s/SELINUX=enforcing/SELINUX=disabled/1" /etc/selinux/config

setenforce  0

配置文件

/etc/sysconfig/docker

OPTIONS='--selinux-enabled=false --log-driver=journald --signature-verification=false --insecure-registry gcr.io'

if [ -z "${DOCKER_CERT_PATH}" ]; then

    DOCKER_CERT_PATH=/etc/docker

fi

/etc/kubernetes/kubelet (k8s-node1)

KUBELET_ADDRESS="--address=10.10.10.11"

KUBELET_HOSTNAME="--hostname-override=k8s-node1"

KUBELET_API_SERVER="--api-servers=http://k8s-master:8080"

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

KUBELET_ARGS="--cluster-dns=8.8.8.8 --cluster-domain=kubelet-node1"

ps: k8s-node2 配置不同之处在于 KUBELET_ADDRESS & KUBELET_HOSTNAME

/etc/kubetnetes/proxy

KUBE_PROXY_ARGS="--master=http://k8s-master:8080"

启动服务

kubelet 依赖 docker 服务,所以先启动 docker.

systemctl start docker

systemctl start kubelet

systemctl start kube-proxy

systemctl enable docker

systemctl enable kubelet

systemctl enable kube-proxy

或者在 shell CLI 执行:

for svc in docker kubelet kube-proxy; do systemctl start $svc; systemctl enable $svc;done

这样一个 multi node k8s cluster 就搭建完了. 可以在 master 节点执行: kubectl get nodes 来查看节点.

node 是注册机制, 所以只要一个网络中的 kubelet 的 apiserver 配置的是 master 节点, master 就会将其纳入.

搭建过程中遇到问题, 多使用 describe 去查看具体信息.

比如创建 rc 一直处于 pending 状态, 可以执行: kubectl describe rc <rc name> .

Q&A

报错信息均使用 kubelet describe 获得

no API token found for service account default/default, retry after the token is automatically created and added to the service account

/etc/kubernetes/apiserver 中 KUBE_ADMISSION_CONTROL 参数去掉 ServiceAccount.

KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"

<br>

****

```sh

Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed for registry.access.redhat.com/rhel7/pod-infrastructure:latest, this may be because there are no credentials on this request.  details: (open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory)"

yum -y install python-rhsm-certificates

创建文件: /etc/rhsm/ca/redhat-uep.pem