centos7之使用最新版的kubeadm体验k8s1.12.0

1、环境准备

centos7 、docker-ce18.06.1-ce、kubeadm、kubelet、kubectl

2、安装

yum安装，准备repo文件

docker：

[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/centos/7/$basearch/stable
enabled=
gpgcheck=
gpgkey=https://download.docker.com/linux/centos/gpg

[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://download.docker.com/linux/centos/7/debug-$basearch/stable
enabled=
gpgcheck=
gpgkey=https://download.docker.com/linux/centos/gpg

[docker-ce-stable-source]
name=Docker CE Stable - Sources
baseurl=https://download.docker.com/linux/centos/7/source/stable
enabled=
gpgcheck=
gpgkey=https://download.docker.com/linux/centos/gpg

kubeadm、kubelet、kubctl

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=
gpgcheck=
repo_gpgcheck=
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

yum -y install docker-ce-18.06..ce-.el7.x86_64
yum -y install  kubelet-1.12.2-0.x86_64  kubectl-1.12.2-0.x86_64 kubeadm-1.12.2-0.x86_64

3、配置docker

#vim /usr/lib/systemd/system/docker.service
 ExecStart=/usr/bin/dockerd --graph=/data/docker --storage-driver=overlay2

# mkdir -p /etc/systemd/system/docker.service.d
#vim http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://10.10.23.74:8118" "NO_PROXY=localhost,127.0.0.1,0.0.0.0,10.10.29.43,10.10.25.49,172.11.0.0,172.10.0.0,172.11.0.0/16,172.10.0.0/16,10.,172.,.evo.get.com,.kube.hpp.com,charts.gitlab.io,.mirror.ucloud.cn"

#cat https-proxy.conf
 [Service]

Environment="HTTPS_PROXY=http://10.10.23.74:8118" "NO_PROXY=localhost,127.0.0.1,0.0.0.0,10.10.29.43,10.10.25.49,172.11.0.0,172.10.0.0,172.11.0.0/16,172.10.0.0/16,10.,172.,.evo.get.com,.kube.hpp.com,charts.gitlab.io,.mirror.ucloud.cn"

shadowsocks的安装参考我的另外的一篇博客：https://www.cnblogs.com/cuishuai/p/8463458.html

4、初始化

创建/etc/sysctl.d/k8s.conf文件

cat /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables =
net.bridge.bridge-nf-call-iptables =
net.ipv4.ip_forward =
vm.swappiness=0

sysctl -p  /etc/sysctl.d/k8s.conf

swapoff -a 
systemctl enable kubelet
systemctl start kubelet
kubeadm init  --kubernetes-version=v1.12.0 --pod-network-cidr=10.244.0.0/  --apiserver-advertise-address=192.18.1.12     

--apiserver-advertise-address

是master的apiserver的监听地址，默认是本机ip。

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

默认master节点是node-role.kubernetes.io/master:NoSchedule，需要做一个修改临时的，为了测试和后面部署flannel。

kubectl taint nodes ku node-role.kubernetes.io/master-

ku是我的master的节点名称。当然可以制定为--all。

安装flannel

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

vim kube-flannel.yml
   args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=eth0

kubectl apply -f kube-flannel.yml

由于前面已经修改了master的taint，可以直接部署，如果没有修改的话，可以修改flannel的压马路文件来实现部署，否则会部署失败。

修改

spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
tolerations:
- operator: Exists
effect: NoSchedule

tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoSchedule

测试DNS

kubectl run curl --image=radial/busyboxplus:curl -it

[ root@curl-5cc7b478b6-6cfqr:/ ]$ nslookup kubernetes.default
Server:    10.96.0.10
Address : 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address : 10.96.0.1 kubernetes.default.svc.cluster.local

进入pod

kubectl exec -it curl-5cc7b478b6-6cfqr -n default -- /bin/sh

！添加node节点到集群：

使用初始化得到的命令直接加入即可，node节点需要安装kubelet、kubeadm

kubeadm默认创建的token24小时失效，再次添加节点的时候，就会报错unauthorized。这里有两种方法：

1、直接生成一个永不过期的token（不推荐这种方式，线上集群的token还是要定时更换）

kubeadm  token create  --ttl  

2、定时更换，设置一个合理的ttl，定时更换集群的token

# cat uptk.sh
#!/bin/bash
token=` kubeadm token list| awk '{print $1}' | grep -v TOKEN`

 for i in $token
 do
  kubeadm token delete $i
 done
kubeadm token create --ttl 72h

设置crontab，每两天更新一次，这个可以根据自己的需求做，因为我生成的token是72h，即3天。

crontab -e
* * */ * * /data/scripts/uptk.sh >& &

获取kubeadm  join需要的信息：

1）token

kubeadm token list | grep authentication,signing | awk '{print $1}'

2）discovery-token-ca-cert-hash

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der >/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

3）加入节点

kubeadm join --token c04f89.b781cdb55d83c1ef 10.10.3.4: --discovery-token-ca-cert-hash sha256:986e83a9cb948368ad0552b95232e31d3b76e2476b595bd1d905d5242ace29af

！集群移除节点

在master节点：

kubectl drain node2 --delete-local-data --force --ignore-daemonsets
kubectl delete node node2

node2是要删除的节点的名称。

在node2节点上执行：

kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel. down
ip link delete flannel.
rm -rf /var/lib/cni/

安装helm

wget https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz
tar xf helm-v2.11.0-linux-amd64.tar.gz
cp  helm tiller  /usr/local/bin

创建tiller需要的用户，这里为了可以使用helm部署到所有的namespace里面，赋予clusterole权限，创建rbac-tiller.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

kubectl  apply  -f  rbac-tiller.yaml

创建tiller部署在特定的namespace，并且helm部署的程序也在这个namespace里面：

参考：https://whmzsu.github.io/helm-doc-zh-cn/quickstart/rbac-zh_cn.html

创建namespace：

kubectl create namespace tiller-world

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: tiller-world
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: tiller-manager
  namespace: tiller-world
rules:
- apiGroups: ["","extensions","apps"]
  resources: ["*"]
  verbs: ["*"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: tiller-binding
  namespace: tiller-world
subjects:
- kind: ServiceAccount
  name: tiller
  namespace: tiller-world
roleRef:
  kind: Role
  name: tiller-manager
  apiGroup: rbac.authorization.k8s.io

初始化：

helm init --service-account tiller  --upgrade

默认安装到kube-system空间下，可以自己指定namespace和image等。

如果将tiller安装到tiller-world下面则需要执行如下命令：

helm init --service-account tiller --tiller-namespace tiller-world  --upgrade

镜像：

# kubernetes
k8s.gcr.io/kube-apiserver:v1.12.0
k8s.gcr.io/kube-controller-manager:v1.12.0
k8s.gcr.io/kube-scheduler:v1.12.0
k8s.gcr.io/kube-proxy:v1.12.0
k8s.gcr.io/etcd:3.2.
k8s.gcr.io/pause:3.1

# network and dns
quay.io/coreos/flannel:v0.10.0-amd64
k8s.gcr.io/coredns:1.2.

# helm and tiller
gcr.io/kubernetes-helm/tiller:v2.11.0

# nginx ingress
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.
k8s.gcr.io/defaultbackend:1.4

# dashboard and metric-sever
k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0
gcr.io/google_containers/metrics-server-amd64:v0.3.0

QA

1.12.1版本也测试过了，但是启动coredns的时候总是报错，起不来。

// :: [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
// :: [FATAL] plugin/loop: Seen "HINFO IN 3256902131464476443.1309143030470211725." more than twice, loop detected

按照google上找到的方法在/etc/systemd/system/kubelet.service.d/10-kubeadm.conf添加如下内容：

Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --resolv-conf=/etc/resolv.conf"

并没有解决，看github上面的issue也是没有解决，有的是停掉systemd-resolved。但是是ubuntu上的。我是的是centos，感觉也是不好，于是回退到1.12.0没有出现那个问题。