K8s集群部署（一）------ETCD集群部署

环境说明

三台主机：

k8s-master  　　10.0.3.225

k8s-node1 　　  10.0.3.226

k8s-node2 　　  10.0.3.227

配置主机名解析

[root@k8s-master kubernetes]# vim /etc/hosts
10.0.3.225 k8s-master
10.0.3.226 k8s-node1
10.0.3.227 k8s-node2

关闭防火墙和SElinux

[root@k8s-master kubernetes]# systemctl stop firewalld

[root@k8s-master kubernetes]# getenforce
Disabled

设置环境变量

vim  /root/.bash_profile

PATH=$PATH:$HOME/bin:/opt/kubernetes/bin

source /root/.bash_profile

为了拷贝方便，设置SSH免密

[root@k8s-master src]# ssh-keygen
[root@k8s-master src]# ssh-copy-id 10.0.3.225
[root@k8s-master src]# ssh-copy-id 10.0.3.226
[root@k8s-master src]# ssh-copy-id 10.0.3.227

系统环境初始化

1.安装Docker（三台操作）

第一步：使用国内Docker源

[root@linux-node1 ~]# cd /etc/yum.repos.d/
[root@linux-node1 yum.repos.d]# wget \
 https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

第二步：Docker安装：

[root@linux-node1 ~]# yum install -y docker-ce

第三步：启动后台进程：

[root@linux-node1 ~]# systemctl start docker

2.准备部署目录（三台操作）

    mkdir -p /opt/kubernetes/{cfg,bin,ssl,log}

3.准备软件包百度网盘下载地址：（在mastrt操作）

[https://pan.baidu.com/s/1zs8sCouDeCQJ9lghH1BPiw](https://pan.baidu.com/s/1zs8sCouDeCQJ9lghH1BPiw)

cd  /usr/local/
unzip k8s-v1.10.1-manual.zip
cd k8s-v1.10.1-manual/k8s-v1.10.1/
mv * /usr/local/src/

 k8s官网地址：https://github.com/kubernetes

4.解压软件包（在mastrt操作）

cd /usr/local/src/
tar zxvf kubernetes.tar.gz  && \
tar zxvf kubernetes-server-linux-amd64.tar.gz  && \
tar zxvf kubernetes-client-linux-amd64.tar.gz && \
tar zxvf kubernetes-node-linux-amd64.tar.gz 

所有软件包都在kubernetes目录下了

[root@k8s-master src]# ls
cfssl-certinfo_linux-amd64  cfssl_linux-amd64             etcd-v3.2.18-linux-amd64.tar.gz     kubernetes                            kubernetes-node-linux-amd64.tar.gz    kubernetes.tar.gz
cfssljson_linux-amd64       cni-plugins-amd64-v0.7.1.tgz  flannel-v0.10.0-linux-amd64.tar.gz  kubernetes-client-linux-amd64.tar.gz  kubernetes-server-linux-amd64.tar.gz
[root@k8s-master src]# cd kubernetes/
[root@k8s-master kubernetes]# ls
addons  client  cluster  docs  examples  hack  kubernetes-src.tar.gz  LICENSES  node  README.md  server  third_party  version
[root@k8s-master kubernetes]# 

手动制作CA证书

1.安装 CFSSL

上面的安装包，已经下载好cfssl了。移动/opt/kubernetes/bin/即可。

cd /usr/local/src
chmod +x cfssl*  && \
mv cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfo && \
mv cfssljson_linux-amd64  /opt/kubernetes/bin/cfssljson  && \
mv cfssl_linux-amd64  /opt/kubernetes/bin/cfssl

复制cfssl命令文件到k8s-node1和k8s-node2节点。如果实际中多个节点，就都需要同步复制。

[root@k8s-master src]# scp /opt/kubernetes/bin/cfssl* 10.0.3.226:/opt/kubernetes/bin/
[root@k8s-master src]# scp /opt/kubernetes/bin/cfssl* 10.0.3.227:/opt/kubernetes/bin/

CFSSL下载地址：

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

2.初始化cfssl

[root@k8s-master src]# cd /usr/local/src/
[root@k8s-master src]# mkdir ssl && cd ssl
[root@k8s-master ssl]# cfssl print-defaults config > config.json
[root@k8s-master ssl]# cfssl print-defaults csr > csr.json

3.创建用来生成 CA 文件的 JSON 配置文件

[root@k8s-master ssl]# vim ca-config.json
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "8760h"
      }
    }
  }
}

4.创建用来生成 CA 证书签名请求（CSR）的 JSON 配置文件

[root@k8s-master ssl]#  vim ca-csr.json
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size":
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

5.生成CA证书（ca.pem）和密钥（ca-key.pem）

[root@k8s-master ssl]#  cfssl gencert -initca ca-csr.json | cfssljson -bare ca
// :: [INFO] generating a new CA key and certificate from CSR
// :: [INFO] generate received request
// :: [INFO] received CSR
// :: [INFO] generating key: rsa-
// :: [INFO] encoded CSR
// :: [INFO] signed certificate with serial number 

[root@k8s-master ssl]# ls -l ca*
-rw-r--r--  root root   Nov  : ca-config.json
-rw-r--r--  root root  Nov  : ca.csr
-rw-r--r--  root root   Nov  : ca-csr.json
-rw-------  root root  Nov  : ca-key.pem
-rw-r--r--  root root  Nov  : ca.pem

6.分发证书

[root@k8s-master ssl]# cp ca.csr ca.pem ca-key.pem ca-config.json /opt/kubernetes/ssl

#scp证书到k8s-node1和k8s-node2节点
[root@k8s-master ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 10.0.3.226:/opt/kubernetes/ssl
ca.csr                                                                                                                                                                           %      .0KB/s   :
ca.pem                                                                                                                                                                           %      .3KB/s   :
ca-key.pem                                                                                                                                                                       %      .6KB/s   :
ca-config.json                                                                                                                                                                   %       .3KB/s   :
[root@k8s-master ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 10.0.3.227:/opt/kubernetes/ssl
ca.csr                                                                                                                                                                           %      .0KB/s   :
ca.pem                                                                                                                                                                           %      .3KB/s   :
ca-key.pem                                                                                                                                                                       %      .6KB/s   :
ca-config.json                                                                                                                                                                   %       .3KB/s   :    

部署ETCD集群

详细介绍查看etcd官网：  https://github.com/coreos/etcd-operator

0.准备etcd软件包

[root@k8s-master src]# pwd
/usr/local/src
[root@k8s-master src]# tar zxvf etcd-v3.2.18-linux-amd64.tar.gz
[root@k8s-master src]# cd etcd-v3.2.18-linux-amd64/
[root@k8s-master etcd-v3.2.18-linux-amd64]# ls
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md
[root@k8s-master etcd-v3.2.18-linux-amd64]# cp etcd etcdctl /opt/kubernetes/bin/
[root@k8s-master etcd-v3.2.18-linux-amd64]# cp etcd etcdctl 10.0.3.226:/opt/kubernetes/bin/
[root@k8s-master etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 10.0.3.227:/opt/kubernetes/bin/

下载地址：wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz

1.创建 etcd 证书签名请求：

[root@k8s-master ssl]# cd /usr/local/src/ssl/
[root@k8s-master ssl]# vim etcd-csr.json
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
"10.0.3.225",         #指定etcd节点的IP地址
"10.0.3.226",
"10.0.3.227"
  ],
  "key": {
    "algo": "rsa",
    "size":
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

2.生成 etcd 证书和私钥：

[root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
>   -ca-key=/opt/kubernetes/ssl/ca-key.pem \
>   -config=/opt/kubernetes/ssl/ca-config.json \
>   -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

会生成一下证书文件
[root@k8s-master ssl]# ls -l etcd*
-rw-r--r--  root root  Nov  : etcd.csr
-rw-r--r--  root root   Nov  : etcd-csr.json
-rw-------  root root  Nov  : etcd-key.pem
-rw-r--r--  root root  Nov  : etcd.pem

3.将证书移动到/opt/kubernetes/ssl目录下

[root@k8s-master ssl]# cp etcd*.pem /opt/kubernetes/ssl
[root@k8s-master ssl]# scp etcd*.pem 10.0.3.226:/opt/kubernetes/ssl
etcd-key.pem                                                                                                                                                                     %      .6KB/s   :
etcd.pem                                                                                                                                                                         %      .4KB/s   :
[root@k8s-master ssl]# scp etcd*.pem 10.0.3.227:/opt/kubernetes/ssl
etcd-key.pem                                                                                                                                                                     %      .6KB/s   :
etcd.pem 

4.设置ETCD配置文件

[root@k8s-master ssl]# vim /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node1"  #唯一的，每个节点不能一样
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"   #数据存放目录
#ETCD_SNAPSHOT_COUNTER=""
#ETCD_HEARTBEAT_INTERVAL=""
#ETCD_ELECTION_TIMEOUT=""
ETCD_LISTEN_PEER_URLS="https://10.0.3.225:2380"   #集群之间通信使用
ETCD_LISTEN_CLIENT_URLS="https://10.0.3.225:2379,https://127.0.0.1:2379"  #是给客户端使用
#ETCD_MAX_SNAPSHOTS=""
#ETCD_MAX_WALS=""
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.3.225:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node1=https://10.0.3.225:2380,etcd-node2=https://10.0.3.226:2380,etcd-node3=https://10.0.3.227:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.3.225:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"

5.创建ETCD系统服务

[root@k8s-master ssl]# vim /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd"
Type=notify

[Install]
WantedBy=multi-user.target

6.重新加载系统服务

[root@k8s-master ssl]# systemctl daemon-reload
[root@k8s-master ssl]# systemctl enable etcd.service 

etcd.conf和etcd.service拷贝到其他节点

[root@k8s-master ssl]# scp /opt/kubernetes/cfg/etcd.conf 10.0.3.226:/opt/kubernetes/cfg/
etcd.conf                                                                                                                                                                        %      .1KB/s   :
[root@k8s-master ssl]# scp /opt/kubernetes/cfg/etcd.conf 10.0.3.227:/opt/kubernetes/cfg/
etcd.conf                                                                                                                                                                        %      .1KB/s   :
[root@k8s-master ssl]# scp /etc/systemd/system/etcd.service 10.0.3.226:/opt/kubernetes/cfg/
etcd.service                                                                                                                                                                     %       .3KB/s   :
[root@k8s-master ssl]# scp /etc/systemd/system/etcd.service 10.0.3.227:/opt/kubernetes/cfg/
etcd.service   

其他节点需要修改一下配置。以node1为例，node2一样要修改。

[root@k8s-node1 yum.repos.d]# vim /opt/kubernetes/cfg/etcd.conf
   ETCD_NAME="etcd-node2"
   ETCD_LISTEN_PEER_URLS="https://10.0.3.226:2380"
   ETCD_LISTEN_CLIENT_URLS="https://10.0.3.226:2379,https://127.0.0.1:2379"
  ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.3.226:2380"
  ETCD_ADVERTISE_CLIENT_URLS="https://10.0.3.226:2379"

同样需要reload一下。

systemctl daemon-reload
systemctl enable etcd

在所有节点上创建etcd存储目录并启动etcd

mkdir /var/lib/etcd
systemctl start etcd
systemctl status  etcd
netstat -lntup|grep 23

需要在所有的 etcd 节点重复上面的步骤，直到所有机器的 etcd 服务都已启动。

7.验证集群

[root@k8s-node1 ~]#  etcdctl --endpoints=https://10.0.3.225:2379 \
>   --ca-file=/opt/kubernetes/ssl/ca.pem \
>   --cert-file=/opt/kubernetes/ssl/etcd.pem \
>   --key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health
member 981424cb437f053 is healthy: got healthy result from https://10.0.3.226:2379
member 7a30be4a7549b699 is healthy: got healthy result from https://10.0.3.227:2379
member d12a89096ab8db09 is healthy: got healthy result from https://10.0.3.225:2379
cluster is healthy