二进制部署K8S-2集群部署

感谢老男孩教育王导的公开视频，文档整理自https://www.yuque.com/duduniao/k8s。

因为在后期运行容器需要有大量的物理硬件资源使用的环境是用的Wmware EXSI如果使用笔记本可适当调整虚拟机配置。

角色 主机名 IP CPU Mem OS
(主)负载均衡、内网DNS hdss7-11.host.com 10.4.7.11 4C 4G CentOS Linux release 7.8
(备)负载均衡、内网DNS hdss7-12.host.com 10.4.7.12 4C 4G CentOS Linux release 7.8
k8s master、k8s node、etcd hdss7-21.host.com 10.4.7.21 4C 8G CentOS Linux release 7.8
k8s master、k8s node、etcd hdss7-22.host.com 10.4.7.22 4C 8G CentOS Linux release 7.8
harbor私有仓库、nfs hdss7-200.host.com 10.4.7.200 4C 4G CentOS Linux release 7.8

2. 安装前准备

2.1. 环境准备

所有机器都需要执行

[root@hdss7-11 ~]# systemctl stop firewalld

[root@hdss7-11 ~]# systemctl disable firewalld

[root@hdss7-11 ~]# setenforce 0

[root@hdss7-11 ~]# sed -ir '/^SELINUX=/s/=.+/=disabled/' /etc/selinux/config

[root@hdss7-11 ~]# yum install -y epel-release

[root@hdss7-11 ~]# yum install -y wget net-tools telnet tree nmap sysstat lrzsz dos2unix bind-utils vim less ntpdate

# 所有机器时间同步，生产环境需要有时间同步服务器。

ntpdate ntp1.aliyun.com

2.2. bind安装

2.2.1. hdss7-11 安装bind

[root@hdss7-11 ~]# yum install -y bind

2.2.2. hdss7-11 配置bind

  • 主配置文件
[root@hdss7-11 ~]# vim /etc/named.conf  # 确保以下配置正确

  listen-on port 53 { 10.4.7.11; };

    directory   "/var/named";

    allow-query     { any; };

  forwarders      { 10.4.7.254; };      # 上级DNS，需要手动添加

  recursion yes;

  dnssec-enable no;

  dnssec-validation no;
  • 在 hdss7-11.host.com 配置区域文件
# 增加两个zone配置，od.com为业务域，host.com.zone为主机域

[root@hdss7-11 ~]# cat>>/etc/named.rfc1912.zones<<EOF

zone "host.com" IN {

        type  master;

        file  "host.com.zone";

        allow-update { 10.4.7.11; };

};

zone "od.com" IN {

        type  master;

        file  "od.com.zone";

        allow-update { 10.4.7.11; };

};

EOF
  • 在 hdss7-11.host.com 配置主机域文件
# line6中时间需要修改

[root@hdss7-11 ~]# vim /var/named/host.com.zone

$ORIGIN host.com.

$TTL 600    ; 10 minutes

@       IN SOA  dns.host.com. dnsadmin.host.com. (

                2021043001 ; serial

                10800      ; refresh (3 hours)

                900        ; retry (15 minutes)

                604800     ; expire (1 week)

                86400      ; minimum (1 day)

                )

            NS   dns.host.com.

$TTL 60 ; 1 minute

dns                A    10.4.7.11

HDSS7-11           A    10.4.7.11

HDSS7-12           A    10.4.7.12

HDSS7-21           A    10.4.7.21

HDSS7-22           A    10.4.7.22

HDSS7-200          A    10.4.7.200
  • 在 hdss7-11.host.com 配置业务域文件
[root@hdss7-11 ~]# vim /var/named/od.com.zone

$ORIGIN od.com.

$TTL 600    ; 10 minutes

@           IN SOA  dns.od.com. dnsadmin.od.com. (

                2021043001 ; serial

                10800      ; refresh (3 hours)

                900        ; retry (15 minutes)

                604800     ; expire (1 week)

                86400      ; minimum (1 day)

                )

                NS   dns.od.com.

$TTL 60 ; 1 minute

dns                A    10.4.7.11
  • 在 hdss7-11.host.com 启动bind服务，并测试
[root@hdss7-11 ~]# named-checkconf  # 检查配置文件

[root@hdss7-11 ~]# systemctl start named ; systemctl enable named

[root@hdss7-11 ~]# host HDSS7-200 10.4.7.11

Using domain server:

Name: 10.4.7.11

Address: 10.4.7.11#53

Aliases: 

HDSS7-200.host.com has address 10.4.7.200

2.2.3. 修改宿主机DNS

  • 修改所有主机的dns服务器地址
[root@hdss7-11 ~]# sed -i '/DNS1/s/10.4.7.254/10.4.7.11/' /etc/sysconfig/network-scripts/ifcfg-ens32

[root@hdss7-11 ~]# systemctl restart network

[root@hdss7-11 ~]# cat /etc/resolv.conf

# Generated by NetworkManager

search host.com

nameserver 10.4.7.11
  • 这里的客户机是windows，直接更改的hosts

10.4.7.11              HDSS7-11.host.com

10.4.7.12              HDSS7-12.host.com

10.4.7.21              HDSS7-21.host.com

10.4.7.22              HDSS7-22.host.com

10.4.7.200             HDSS7-200.host.com harbor.od.com k8s-yaml.od.com wlw.od.com

10.4.7.10              traefik.od.com dashboard.od.com jenkins.od.com dubbo-monitor.od.com demo.od.com

2.3. 根证书准备

  • 在 hdss7-200 下载工具
[root@hdss7-200 ~]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl

[root@hdss7-200 ~]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssl-json

[root@hdss7-200 ~]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/local/bin/cfssl-certinfo

[root@hdss7-200 ~]# chmod u+x /usr/local/bin/cfssl*
  • 在 hdss7-200 签发根证书
[root@hdss7-200 ~]# mkdir /opt/certs/ ; cd /opt/certs/

# 根证书配置：

# CN 一般写域名，浏览器会校验

# names 为地区和公司信息

# expiry 为过期时间

[root@hdss7-200 certs]# cat>>/opt/certs/ca-csr.json<<EOF

{

    "CN": "OldboyEdu",

    "hosts": [

    ],

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "ST": "beijing",

            "L": "beijing",

            "O": "od",

            "OU": "ops"

        }

    ],

    "ca": {

        "expiry": "175200h"

    }

}

EOF

[root@hdss7-200 certs]# cfssl gencert -initca ca-csr.json | cfssl-json -bare ca

2020/01/05 10:42:07 [INFO] generating a new CA key and certificate from CSR

2020/01/05 10:42:07 [INFO] generate received request

2020/01/05 10:42:07 [INFO] received CSR

2020/01/05 10:42:07 [INFO] generating key: rsa-2048

2020/01/05 10:42:08 [INFO] encoded CSR

2020/01/05 10:42:08 [INFO] signed certificate with serial number 451005524427475354617025362003367427117323539780

[root@hdss7-200 certs]# ls -l ca*

-rw-r--r-- 1 root root  993 Jan  5 10:42 ca.csr

-rw-r--r-- 1 root root  328 Jan  5 10:39 ca-csr.json

-rw------- 1 root root 1675 Jan  5 10:42 ca-key.pem

-rw-r--r-- 1 root root 1346 Jan  5 10:42 ca.pem

2.4. docker环境准备

需要安装docker的机器：hdss7-21 hdss7-22 hdss7-200，以hdss7-21为例

[root@hdss7-21 ~]# wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

[root@hdss7-21 ~]# yum install -y docker-ce

# 不安全的registry中增加了harbor地址

# 各个机器上bip网段不一致，bip中间两段与宿主机最后两段相同，目的是方便定位问题

[root@hdss7-21 ~]# mkdir /etc/docker/

[root@hdss7-21 ~]# vim /etc/docker/daemon.json

{

  "graph": "/data/docker",

  "storage-driver": "overlay2",

  "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],

  "registry-mirrors": ["https://registry.docker-cn.com"],

  "bip": "172.7.21.1/24",

  "exec-opts": ["native.cgroupdriver=systemd"],

  "live-restore": true

}

[root@hdss7-21 ~]# mkdir /data/docker

[root@hdss7-21 ~]# systemctl start docker ; systemctl enable docker

2.5. harbor安装

参考地址：https://www.yuque.com/duduniao/trp3ic/ohrxds#9Zpxx

官方地址：https://goharbor.io/

下载地址：https://github.com/goharbor/harbor/releases

2.5.1. hdss7-200 安装harbor

# 目录说明：

# /opt/src : 源码、文件下载目录

# /opt/release : 各个版本软件存放位置

# /opt/apps : 各个软件当前版本的软链接

[root@hdss7-200 ~]# mkdir /opt/{src,apps,release}

[root@hdss7-200 ~]# cd /opt/src

[root@hdss7-200 src]# wget https://github.com/goharbor/harbor/releases/download/v1.9.4/harbor-offline-installer-v1.9.4.tgz

[root@hdss7-200 src]# tar -xvf harbor-offline-installer-v1.9.4.tgz

[root@hdss7-200 src]# mv harbor /opt/release/harbor-v1.9.4

[root@hdss7-200 src]# ln -s /opt/release/harbor-v1.9.4 /opt/apps/harbor

[root@hdss7-200 src]# ll /opt/apps/

total 0

lrwxrwxrwx 1 root root 26 Jan  5 11:13 harbor -> /opt/release/harbor-v1.9.4

# 实验环境仅修改以下配置项，生产环境还得修改密码

[root@hdss7-200 src]# vim /opt/apps/harbor/harbor.yml

hostname: harbor.od.com

http:

  port: 180

data_volume: /data/harbor

location: /data/harbor/logs

[root@hdss7-200 src]# yum install -y docker-compose

[root@hdss7-200 src]# cd /opt/apps/harbor/

[root@hdss7-200 harbor]# ./install.sh

......

 ----Harbor has been installed and started successfully.----

[root@hdss7-200 harbor]# docker-compose ps

      Name                     Command               State             Ports

--------------------------------------------------------------------------------------

harbor-core         /harbor/harbor_core              Up

harbor-db           /docker-entrypoint.sh            Up      5432/tcp

harbor-jobservice   /harbor/harbor_jobservice  ...   Up

harbor-log          /bin/sh -c /usr/local/bin/ ...   Up      127.0.0.1:1514->10514/tcp

harbor-portal       nginx -g daemon off;             Up      8080/tcp

nginx               nginx -g daemon off;             Up      0.0.0.0:180->8080/tcp

redis               redis-server /etc/redis.conf     Up      6379/tcp

registry            /entrypoint.sh /etc/regist ...   Up      5000/tcp

registryctl         /harbor/start.sh                 Up
  • 设置harbor开机启动
 # 增加以下内容

[root@hdss7-200 harbor]# cat>>/etc/rc.d/rc.local<<EOF

# start harbor

cd /opt/apps/harbor

/usr/bin/docker-compose stop

/usr/bin/docker-compose start

EOF

# 注意要给执行权限

chmod +x /etc/rc.d/rc.local

2.5.2. hdss7-200 安装nginx

  • 安装Nginx反向代理harbor
# 当前机器中Nginx功能较少，使用yum安装即可。如有多个harbor考虑源码编译且配置健康检查

# nginx配置此处忽略，仅仅使用最简单的配置。

[root@hdss7-200 ~]# yum -y install nginx

[root@hdss7-200 harbor]# cat>>/etc/nginx/conf.d/harbor.conf<<EOF

server {

    listen       80;

    server_name  harbor.od.com;

    # 避免出现上传失败的情况

    client_max_body_size 1000m;

    location / {

        proxy_pass http://127.0.0.1:180;

    }

}

EOF

[root@hdss7-200 harbor]# systemctl start nginx ; systemctl enable nginx
  • hdss7-11 配置DNS解析
# serial序列号需要滚动一个,根据日期编写的。只是一个数字而已

[root@hdss7-11 ~]# vim /var/named/od.com.zone

$ORIGIN od.com.

$TTL 600    ; 10 minutes

@           IN SOA  dns.od.com. dnsadmin.od.com. (

                2021043002 ; serial    # 更改这个数字

                10800      ; refresh (3 hours)

                900        ; retry (15 minutes)

                604800     ; expire (1 week)

                86400      ; minimum (1 day)

                )

                NS   dns.od.com.

$TTL 60 ; 1 minute

dns                A    10.4.7.11

harbor             A    10.4.7.200

[root@hdss7-11 ~]# systemctl restart named.service  # reload 无法使得配置生效

[root@hdss7-11 ~]# host harbor.od.com

harbor.od.com has address 10.4.7.200

  • 新建项目: public

  • 测试harbor
[root@hdss7-200 ~]# docker pull nginx:1.17.9

[root@hdss7-200 ~]# docker tag nginx:1.17.9 harbor.od.com/public/nginx:v1.17.9

[root@hdss7-200 ~]# docker login -u admin harbor.od.com

                    # 默认密码: Harbor12345

[root@hdss7-200 ~]# docker image push harbor.od.com/public/nginx:v1.17.9

3. 主控节点安装

3.1. etcd安装

etcd 的leader选举机制，要求至少为3台或以上的奇数台。本次安装涉及：hdss7-12，hdss7-21，hdss7-22

3.1.1. 签发etcd证书

证书签发服务器 hdss7-200:

  • 创建ca的json配置: /opt/certs/ca-config.json

    • server 表示服务端连接客户端时携带的证书，用于客户端验证服务端身份
    • client 表示客户端连接服务端时携带的证书，用于服务端验证客户端身份
    • peer 表示相互之间连接时使用的证书，如etcd节点之间验证
cat>>/opt/certs/ca-config.json<<EOF

{

    "signing": {

        "default": {

            "expiry": "175200h"

        },

        "profiles": {

            "server": {

                "expiry": "175200h",

                "usages": [

                    "signing",

                    "key encipherment",

                    "server auth"

                ]

            },

            "client": {

                "expiry": "175200h",

                "usages": [

                    "signing",

                    "key encipherment",

                    "client auth"

                ]

            },

            "peer": {

                "expiry": "175200h",

                "usages": [

                    "signing",

                    "key encipherment",

                    "server auth",

                    "client auth"

                ]

            }

        }

    }

}

EOF
  • 创建etcd证书配置：/opt/certs/etcd-peer-csr.json

重点在hosts上，将所有可能的etcd服务器添加到host列表，不能使用网段，新增etcd服务器需要重新签发证书

cat>>/opt/certs/etcd-peer-csr.json<<EOF

{

    "CN": "k8s-etcd",

    "hosts": [

        "10.4.7.11",

        "10.4.7.12",

        "10.4.7.21",

        "10.4.7.22"

    ],

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "ST": "beijing",

            "L": "beijing",

            "O": "od",

            "OU": "ops"

        }

    ]

}

EOF
  • 签发证书
[root@hdss7-200 ~]# cd /opt/certs/

[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer

[root@hdss7-200 certs]# ll etcd-peer*

-rw-r--r-- 1 root root 1062 Jan  5 17:01 etcd-peer.csr

-rw-r--r-- 1 root root  363 Jan  5 16:59 etcd-peer-csr.json

-rw------- 1 root root 1675 Jan  5 17:01 etcd-peer-key.pem

-rw-r--r-- 1 root root 1428 Jan  5 17:01 etcd-peer.pem

3.1.2. 安装etcd

etcd地址：https://github.com/etcd-io/etcd/

实验使用版本: etcd-v3.1.20-linux-amd64.tar.gz

本次安装涉及：hdss7-12，hdss7-21，hdss7-22

  • 下载etcd
[root@hdss7-12 ~]# useradd -s /sbin/nologin -M etcd

[root@hdss7-12 ~]# cd /opt/src/

[root@hdss7-12 src]# wget https://github.com/etcd-io/etcd/releases/download/v3.1.20/etcd-v3.1.20-linux-amd64.tar.gz

[root@hdss7-12 src]# tar -xf etcd-v3.1.20-linux-amd64.tar.gz

[root@hdss7-12 src]# mv etcd-v3.1.20-linux-amd64 /opt/release/etcd-v3.1.20

[root@hdss7-12 src]# ln -s /opt/release/etcd-v3.1.20 /opt/apps/etcd

[root@hdss7-12 src]# ll /opt/apps/etcd

lrwxrwxrwx 1 root root 25 Jan  5 17:56 /opt/apps/etcd -> /opt/release/etcd-v3.1.20

[root@hdss7-12 src]# mkdir -p /opt/apps/etcd/certs /data/etcd /data/logs/etcd-server
  • 下发证书到各个etcd上
[root@hdss7-200 ~]# cd /opt/certs/

[root@hdss7-200 certs]# for i in 12 21 22;do scp ca.pem etcd-peer.pem etcd-peer-key.pem hdss7-${i}:/opt/apps/etcd/certs/ ;done


[root@hdss7-12 src]# md5sum /opt/apps/etcd/certs/*

8778d0c3411891af61a287e49a70c89a  /opt/apps/etcd/certs/ca.pem

7918783c2f6bf69e96edf03e67d04983  /opt/apps/etcd/certs/etcd-peer-key.pem

d4d849751a834c7727d42324fdedf92d  /opt/apps/etcd/certs/etcd-peer.pem
  • 创建启动脚本(部分参数每台机器不同)
[root@hdss7-12 ~]# vim /opt/apps/etcd/etcd-server-startup.sh

#!/bin/sh

# listen-peer-urls etcd节点之间通信端口

# listen-client-urls 客户端与etcd通信端口

# quota-backend-bytes 配额大小

# 需要修改的参数：name,listen-peer-urls,listen-client-urls,initial-advertise-peer-urls

WORK_DIR=$(dirname $(readlink -f $0))

[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/etcd/etcd --name etcd-server-7-12 \

    --data-dir /data/etcd/etcd-server \

    --listen-peer-urls https://10.4.7.12:2380 \

    --listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \

    --quota-backend-bytes 8000000000 \

    --initial-advertise-peer-urls https://10.4.7.12:2380 \

    --advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \

    --initial-cluster  etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \

    --ca-file ./certs/ca.pem \

    --cert-file ./certs/etcd-peer.pem \

    --key-file ./certs/etcd-peer-key.pem \

    --client-cert-auth  \

    --trusted-ca-file ./certs/ca.pem \

    --peer-ca-file ./certs/ca.pem \

    --peer-cert-file ./certs/etcd-peer.pem \

    --peer-key-file ./certs/etcd-peer-key.pem \

    --peer-client-cert-auth \

    --peer-trusted-ca-file ./certs/ca.pem \

    --log-output stdout


[root@hdss7-12 ~]# chmod u+x /opt/apps/etcd/etcd-server-startup.sh

[root@hdss7-12 ~]# chown -R etcd.etcd /opt/apps/etcd/ /data/etcd /data/logs/etcd-server

3.1.3. 启动etcd

因为这些进程都是要启动为后台进程，要么手动启动，要么采用后台进程管理工具，实验中使用后台管理工具

[root@hdss7-12 ~]# yum install -y supervisor

[root@hdss7-12 ~]# systemctl start supervisord ; systemctl enable supervisord

[root@hdss7-12 ~]# vim /etc/supervisord.d/etcd-server.ini

[program:etcd-server-7-12]

command=/opt/apps/etcd/etcd-server-startup.sh

numprocs=1

directory=/opt/apps/etcd

autostart=true

autorestart=true

startsecs=30

startretries=3

exitcodes=0,2

stopsignal=QUIT

stopwaitsecs=10

user=etcd

redirect_stderr=true

stdout_logfile=/data/logs/etcd-server/etcd.stdout.log

stdout_logfile_maxbytes=64MB

stdout_logfile_backups=5

stdout_capture_maxbytes=1MB

stdout_events_enabled=false

killasgroup=true

stopasgroup=true

[root@hdss7-12 ~]# mkdir /data/logs/etcd-server/ -p

[root@hdss7-12 ~]# supervisorctl update

etcd-server-7-12: added process group
  • etcd 进程状态查看
[root@hdss7-12 ~]# supervisorctl status  # supervisorctl 状态

etcd-server-7-12                 RUNNING   pid 22375, uptime 0:00:39

[root@hdss7-12 ~]# netstat -lntp|grep etcd

tcp        0      0 10.4.7.12:2379          0.0.0.0:*               LISTEN      22379/etcd

tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      22379/etcd

tcp        0      0 10.4.7.12:2380          0.0.0.0:*               LISTEN      22379/etcd

[root@hdss7-12 ~]# /opt/apps/etcd/etcdctl member list # 随着etcd重启，leader会变化

988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.22:2379 isLeader=false

5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.21:2379 isLeader=true

f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false

[root@hdss7-22 /opt/apps/etcd]# ./etcdctl cluster-health

member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379

member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379

member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379

cluster is healthy
  • etcd 启停方式
[root@hdss7-12 ~]# supervisorctl start etcd-server-7-12

[root@hdss7-12 ~]# supervisorctl stop etcd-server-7-12

[root@hdss7-12 ~]# supervisorctl restart etcd-server-7-12

[root@hdss7-12 ~]# supervisorctl status etcd-server-7-12

3.2. apiserver 安装

3.2.1. 下载kubernetes服务端

aipserver 涉及的服务器：hdss7-21，hdss7-22

下载 kubernetes 二进制版本包需要FQ工具

[root@hdss7-21 ~]# cd /opt/src

[root@hdss7-21 src]# wget https://dl.k8s.io/v1.15.2/kubernetes-server-linux-amd64.tar.gz

[root@hdss7-21 src]# tar -xf kubernetes-server-linux-amd64.tar.gz

[root@hdss7-21 src]# mv kubernetes /opt/release/kubernetes-v1.15.2

[root@hdss7-21 src]# ln -s /opt/release/kubernetes-v1.15.2 /opt/apps/kubernetes

[root@hdss7-21 src]# ll /opt/apps/kubernetes

lrwxrwxrwx 1 root root 31 Jan  6 12:59 /opt/apps/kubernetes -> /opt/release/kubernetes-v1.15.2

[root@hdss7-21 src]# cd /opt/apps/kubernetes

[root@hdss7-21 kubernetes]# rm -f kubernetes-src.tar.gz

[root@hdss7-21 kubernetes]# cd server/bin/

[root@hdss7-21 bin]# rm -f *.tar *_tag  # *.tar *_tag 镜像文件

[root@hdss7-21 bin]# ll

total 884636

-rwxr-xr-x 1 root root  43534816 Aug  5 18:01 apiextensions-apiserver

-rwxr-xr-x 1 root root 100548640 Aug  5 18:01 cloud-controller-manager

-rwxr-xr-x 1 root root 200648416 Aug  5 18:01 hyperkube

-rwxr-xr-x 1 root root  40182208 Aug  5 18:01 kubeadm

-rwxr-xr-x 1 root root 164501920 Aug  5 18:01 kube-apiserver

-rwxr-xr-x 1 root root 116397088 Aug  5 18:01 kube-controller-manager

-rwxr-xr-x 1 root root  42985504 Aug  5 18:01 kubectl

-rwxr-xr-x 1 root root 119616640 Aug  5 18:01 kubelet

-rwxr-xr-x 1 root root  36987488 Aug  5 18:01 kube-proxy

-rwxr-xr-x 1 root root  38786144 Aug  5 18:01 kube-scheduler

-rwxr-xr-x 1 root root   1648224 Aug  5 18:01 mounter

3.2.2. 签发证书

签发证书 涉及的服务器：hdss7-200

  • 签发client证书（apiserver和etcd通信证书）
[root@hdss7-200 ~]# cd /opt/certs/

[root@hdss7-200 certs]# cat>>/opt/certs/client-csr.json<<EOF

{

    "CN": "k8s-node",

    "hosts": [

    ],

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "ST": "beijing",

            "L": "beijing",

            "O": "od",

            "OU": "ops"

        }

    ]

}

EOF

[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json |cfssl-json -bare client

2020/01/06 13:42:47 [INFO] generate received request

2020/01/06 13:42:47 [INFO] received CSR

2020/01/06 13:42:47 [INFO] generating key: rsa-2048

2020/01/06 13:42:47 [INFO] encoded CSR

2020/01/06 13:42:47 [INFO] signed certificate with serial number 268276380983442021656020268926931973684313260543

2020/01/06 13:42:47 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for

websites. For more information see the Baseline Requirements for the Issuance and Management

of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);

specifically, section 10.2.3 ("Information Requirements").

[root@hdss7-200 certs]# ls client* -l

-rw-r--r-- 1 root root  993 Jan  6 13:42 client.csr

-rw-r--r-- 1 root root  280 Jan  6 13:42 client-csr.json

-rw------- 1 root root 1679 Jan  6 13:42 client-key.pem

-rw-r--r-- 1 root root 1363 Jan  6 13:42 client.pem
  • 签发server证书（apiserver和其它k8s组件通信使用）
# hosts中将所有可能作为apiserver的ip添加进去，VIP 10.4.7.10 也要加入

[root@hdss7-200 certs]# cat>>/opt/certs/apiserver-csr.json<<EOF

{

    "CN": "k8s-apiserver",

    "hosts": [

        "127.0.0.1",

        "192.168.0.1",

        "kubernetes.default",

        "kubernetes.default.svc",

        "kubernetes.default.svc.cluster",

        "kubernetes.default.svc.cluster.local",

        "10.4.7.10",

        "10.4.7.21",

        "10.4.7.22",

        "10.4.7.23"

    ],

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "ST": "beijing",

            "L": "beijing",

            "O": "od",

            "OU": "ops"

        }

    ]

}

EOF

[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json |cfssl-json -bare apiserver

2020/01/06 13:46:56 [INFO] generate received request

2020/01/06 13:46:56 [INFO] received CSR

2020/01/06 13:46:56 [INFO] generating key: rsa-2048

2020/01/06 13:46:56 [INFO] encoded CSR

2020/01/06 13:46:56 [INFO] signed certificate with serial number 573076691386375893093727554861295529219004473872

2020/01/06 13:46:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for

websites. For more information see the Baseline Requirements for the Issuance and Management

of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);

specifically, section 10.2.3 ("Information Requirements").

[root@hdss7-200 certs]# ls apiserver* -l

-rw-r--r-- 1 root root 1249 Jan  6 13:46 apiserver.csr

-rw-r--r-- 1 root root  566 Jan  6 13:45 apiserver-csr.json

-rw------- 1 root root 1675 Jan  6 13:46 apiserver-key.pem

-rw-r--r-- 1 root root 1598 Jan  6 13:46 apiserver.pem
  • 证书下发
[root@hdss7-200 certs]# for i in 21 22;do echo hdss7-$i;ssh hdss7-$i "mkdir /opt/apps/kubernetes/server/bin/certs";scp apiserver-key.pem apiserver.pem ca-key.pem ca.pem client-key.pem client.pem hdss7-$i:/opt/apps/kubernetes/server/bin/certs/;done

3.2.3. 配置apiserver日志审计

aipserver 涉及的服务器：hdss7-21，hdss7-22

[root@hdss7-21 bin]# mkdir /opt/apps/kubernetes/conf

# 打开文件后，设置 :set paste，避免自动缩进

[root@hdss7-21 bin]# \vi /opt/apps/kubernetes/conf/audit.yaml

apiVersion: audit.k8s.io/v1beta1 # This is required.

kind: Policy

# Don't generate audit events for all requests in RequestReceived stage.

omitStages:

  - "RequestReceived"

rules:

  # Log pod changes at RequestResponse level

  - level: RequestResponse

    resources:

    - group: ""

      # Resource "pods" doesn't match requests to any subresource of pods,

      # which is consistent with the RBAC policy.

      resources: ["pods"]

  # Log "pods/log", "pods/status" at Metadata level

  - level: Metadata

    resources:

    - group: ""

      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"

  - level: None

    resources:

    - group: ""

      resources: ["configmaps"]

      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services

  - level: None

    users: ["system:kube-proxy"]

    verbs: ["watch"]

    resources:

    - group: "" # core API group

      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.

  - level: None

    userGroups: ["system:authenticated"]

    nonResourceURLs:

    - "/api*" # Wildcard matching.

    - "/version"

  # Log the request body of configmap changes in kube-system.

  - level: Request

    resources:

    - group: "" # core API group

      resources: ["configmaps"]

    # This rule only applies to resources in the "kube-system" namespace.

    # The empty string "" can be used to select non-namespaced resources.

    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.

  - level: Metadata

    resources:

    - group: "" # core API group

      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.

  - level: Request

    resources:

    - group: "" # core API group

    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.

  - level: Metadata

    # Long-running requests like watches that fall under this rule will not

    # generate an audit event in RequestReceived.

    omitStages:

      - "RequestReceived"

3.2.4. 配置启动脚本

aipserver 涉及的服务器：hdss7-21，hdss7-22

  • 创建启动脚本
[root@hdss7-21 bin]# vim /opt/apps/kubernetes/server/bin/kube-apiserver-startup.sh

#!/bin/bash

WORK_DIR=$(dirname $(readlink -f $0))

[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/kubernetes/server/bin/kube-apiserver \

    --apiserver-count 2 \

    --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \

    --audit-policy-file ../../conf/audit.yaml \

    --authorization-mode RBAC \

    --client-ca-file ./certs/ca.pem \

    --requestheader-client-ca-file ./certs/ca.pem \

    --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \

    --etcd-cafile ./certs/ca.pem \

    --etcd-certfile ./certs/client.pem \

    --etcd-keyfile ./certs/client-key.pem \

    --etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \

    --service-account-key-file ./certs/ca-key.pem \

    --service-cluster-ip-range 192.168.0.0/16 \

    --service-node-port-range 3000-29999 \

    --target-ram-mb=1024 \

    --kubelet-client-certificate ./certs/client.pem \

    --kubelet-client-key ./certs/client-key.pem \

    --log-dir  /data/logs/kubernetes/kube-apiserver \

    --tls-cert-file ./certs/apiserver.pem \

    --tls-private-key-file ./certs/apiserver-key.pem \

    --v 2

    # 添加可执行权限

    chmod +x /opt/release/kubernetes-v1.15.2/server/bin/kube-apiserver-startup.sh
  • 配置supervisor启动配置
[root@hdss7-21 bin]# vim /etc/supervisord.d/kube-apiserver.ini

[program:kube-apiserver-7-21]

command=/opt/apps/kubernetes/server/bin/kube-apiserver-startup.sh

numprocs=1

directory=/opt/apps/kubernetes/server/bin

autostart=true

autorestart=true

startsecs=30

startretries=3

exitcodes=0,2

stopsignal=QUIT

stopwaitsecs=10

user=root

redirect_stderr=true

stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log

stdout_logfile_maxbytes=64MB

stdout_logfile_backups=5

stdout_capture_maxbytes=1MB

stdout_events_enabled=false

killasgroup=true

stopasgroup=true

[root@hdss7-21 bin]# mkdir /data/logs/kubernetes/kube-apiserver/ -p

[root@hdss7-21 bin]# supervisorctl update

[root@hdss7-21 bin]# supervisorctl status

etcd-server-7-21                 RUNNING   pid 23637, uptime 22:26:08

kube-apiserver-7-21              RUNNING   pid 32591, uptime 0:05:37
  • 启停apiserver
[root@hdss7-12 ~]# supervisorctl start kube-apiserver-7-21

[root@hdss7-12 ~]# supervisorctl stop kube-apiserver-7-21

[root@hdss7-12 ~]# supervisorctl restart kube-apiserver-7-21

[root@hdss7-12 ~]# supervisorctl status kube-apiserver-7-21
  • 查看进程
[root@hdss7-21 bin]# netstat -lntp|grep api

tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      32595/kube-apiserve

tcp6       0      0 :::6443                 :::*                    LISTEN      32595/kube-apiserve

[root@hdss7-21 bin]# ps uax|grep kube-apiserver|grep -v grep

root      32591  0.0  0.0 115296  1476 ?        S    20:17   0:00 /bin/bash /opt/apps/kubernetes/server/bin/kube-apiserver-startup.sh

root      32595  3.0  2.3 402720 184892 ?       Sl   20:17   0:16 /opt/apps/kubernetes/server/bin/kube-apiserver --apiserver-count 2 --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log --audit-policy-file ../../conf/audit.yaml --authorization-mode RBAC --client-ca-file ./certs/ca.pem --requestheader-client-ca-file ./certs/ca.pem --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --etcd-cafile ./certs/ca.pem --etcd-certfile ./certs/client.pem --etcd-keyfile ./certs/client-key.pem --etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 --service-account-key-file ./certs/ca-key.pem --service-cluster-ip-range 192.168.0.0/16 --service-node-port-range 3000-29999 --target-ram-mb=1024 --kubelet-client-certificate ./certs/client.pem --kubelet-client-key ./certs/client-key.pem --log-dir /data/logs/kubernetes/kube-apiserver --tls-cert-file ./certs/apiserver.pem --tls-private-key-file ./certs/apiserver-key.pem --v 2

3.3. 配置apiserver L4代理

3.3.1. nginx配置

L4 代理涉及的服务器：hdss7-11，hdss7-12

[root@hdss7-11 ~]# yum install -y nginx

[root@hdss7-11 ~]# vim /etc/nginx/nginx.conf

# 末尾加上以下内容，stream 只能加在 main 中

# 此处只是简单配置下nginx，实际生产中，建议进行更合理的配置

stream {

    log_format proxy '$time_local|$remote_addr|$upstream_addr|$protocol|$status|'

                     '$session_time|$upstream_connect_time|$bytes_sent|$bytes_received|'

                     '$upstream_bytes_sent|$upstream_bytes_received' ;

    upstream kube-apiserver {

        server 10.4.7.21:6443     max_fails=3 fail_timeout=30s;

        server 10.4.7.22:6443     max_fails=3 fail_timeout=30s;

    }

    server {

        listen 7443;

        proxy_connect_timeout 2s;

        proxy_timeout 900s;

        proxy_pass kube-apiserver;

        access_log /var/log/nginx/proxy.log proxy;

    }

}

[root@hdss7-11 ~]# systemctl start nginx; systemctl enable nginx

[root@hdss7-11 ~]# curl 127.0.0.1:7443  # 测试几次

Client sent an HTTP request to an HTTPS server.

[root@hdss7-11 ~]# cat /var/log/nginx/proxy.log

06/Jan/2020:21:00:27 +0800|127.0.0.1|10.4.7.21:6443|TCP|200|0.001|0.000|76|78|78|76

06/Jan/2020:21:05:03 +0800|127.0.0.1|10.4.7.22:6443|TCP|200|0.020|0.019|76|78|78|76

06/Jan/2020:21:05:04 +0800|127.0.0.1|10.4.7.21:6443|TCP|200|0.001|0.001|76|78|78|76

3.3.2. keepalived配置

aipserver L4 代理涉及的服务器：hdss7-11，hdss7-12

  • 安装keepalive
[root@hdss7-11 ~]# yum install -y keepalived

[root@hdss7-11 ~]# vim /etc/keepalived/check_port.sh # 配置检查脚本

#!/bin/bash

if [ $# -eq 1 ] && [[ $1 =~ ^[0-9]+ ]];then

    [ $(netstat -lntp|grep ":$1 " |wc -l) -eq 0 ] && echo "[ERROR] nginx may be not running!" && exit 1 || exit 0

else

    echo "[ERROR] need one port!"

    exit 1

fi

[root@hdss7-11 ~]# chmod +x /etc/keepalived/check_port.sh
  • 配置主节点：/etc/keepalived/keepalived.conf

主节点中，必须加上 nopreempt

因为一旦因为网络抖动导致VIP漂移，不能让它自动飘回来，必须要分析原因后手动迁移VIP到主节点！如主节点确认正常后，重启备节点的keepalive，让VIP飘到主节点.

keepalived 的日志输出配置此处省略，生产中需要进行处理。

vi /etc/keepalived/keepalived.conf 

! Configuration File for keepalived

global_defs {

   router_id 10.4.7.11

}

vrrp_script chk_nginx {

    script "/etc/keepalived/check_port.sh 7443"

    interval 2

    weight -20

}

vrrp_instance VI_1 {

    state MASTER

    interface ens32

    virtual_router_id 251

    priority 100

    advert_int 1

    mcast_src_ip 10.4.7.11

    nopreempt

    authentication {

        auth_type PASS

        auth_pass 11111111

    }

    track_script {

         chk_nginx

    }

    virtual_ipaddress {

        10.4.7.10

    }

}

  • 配置备节点：/etc/keepalived/keepalived.conf

    注意根据实际的网卡名称更改网卡名配置。

vi /etc/keepalived/keepalived.conf 

! Configuration File for keepalived

global_defs {

    router_id 10.4.7.12

}

vrrp_script chk_nginx {

    script "/etc/keepalived/check_port.sh 7443"

    interval 2

    weight -20

}

vrrp_instance VI_1 {

    state BACKUP

    interface ens32

    virtual_router_id 251

    mcast_src_ip 10.4.7.12

    priority 90

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 11111111

    }

    track_script {

        chk_nginx

    }

    virtual_ipaddress {

        10.4.7.10

    }

}
  • 启动keepalived
[root@hdss7-11 ~]# systemctl start keepalived ; systemctl enable keepalived

[root@hdss7-11 ~]# ip addr show ens32

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 00:0c:29:6d:b8:82 brd ff:ff:ff:ff:ff:ff

    inet 10.4.7.11/24 brd 10.4.7.255 scope global noprefixroute ens32

       valid_lft forever preferred_lft forever

    inet 10.4.7.10/32 scope global ens32

       valid_lft forever preferred_lft forever

......

3.4. controller-manager 安装

controller-manager 涉及的服务器：hdss7-21，hdss7-22

controller-manager 设置为只调用当前机器的 apiserver，走127.0.0.1网卡，因此不配制SSL证书

[root@hdss7-21 ~]# vim /opt/apps/kubernetes/server/bin/kube-controller-manager-startup.sh

#!/bin/sh

WORK_DIR=$(dirname $(readlink -f $0))

[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/kubernetes/server/bin/kube-controller-manager \

    --cluster-cidr 172.7.0.0/16 \

    --leader-elect true \

    --log-dir /data/logs/kubernetes/kube-controller-manager \

    --master http://127.0.0.1:8080 \

    --service-account-private-key-file ./certs/ca-key.pem \

    --service-cluster-ip-range 192.168.0.0/16 \

    --root-ca-file ./certs/ca.pem \

    --v 2

[root@hdss7-21 ~]# chmod u+x /opt/apps/kubernetes/server/bin/kube-controller-manager-startup.sh


[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-controller-manager.ini

[program:kube-controller-manager-7-21]

command=/opt/apps/kubernetes/server/bin/kube-controller-manager-startup.sh

numprocs=1

directory=/opt/apps/kubernetes/server/bin

autostart=true

autorestart=true

startsecs=30

startretries=3

exitcodes=0,2

stopsignal=QUIT

stopwaitsecs=10

user=root

redirect_stderr=true

stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log

stdout_logfile_maxbytes=64MB

stdout_logfile_backups=4

stdout_capture_maxbytes=1MB

stdout_events_enabled=false

killasgroup=true

stopasgroup=true


[root@hdss7-21 ~]# mkdir /data/logs/kubernetes/kube-controller-manager

[root@hdss7-21 ~]# supervisorctl update

kube-controller-manager-7-21: stopped

kube-controller-manager-7-21: updated process group

[root@hdss7-21 ~]# supervisorctl status

etcd-server-7-21                 RUNNING   pid 23637, uptime 1 day, 0:16:54

kube-apiserver-7-21              RUNNING   pid 32591, uptime 1:56:23

kube-controller-manager-7-21     RUNNING   pid 33357, uptime 0:00:38

3.5. kube-scheduler安装

kube-scheduler 涉及的服务器：hdss7-21，hdss7-22

kube-scheduler 设置为只调用当前机器的 apiserver，走127.0.0.1网卡，因此不配制SSL证书

[root@hdss7-21 ~]# vim /opt/apps/kubernetes/server/bin/kube-scheduler-startup.sh

#!/bin/sh

WORK_DIR=$(dirname $(readlink -f $0))

[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/kubernetes/server/bin/kube-scheduler \

    --leader-elect  \

    --log-dir /data/logs/kubernetes/kube-scheduler \

    --master http://127.0.0.1:8080 \

    --v 2

[root@hdss7-21 ~]# chmod u+x /opt/apps/kubernetes/server/bin/kube-scheduler-startup.sh

[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-scheduler


[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-scheduler.ini

[program:kube-scheduler-7-21]

command=/opt/apps/kubernetes/server/bin/kube-scheduler-startup.sh

numprocs=1

directory=/opt/apps/kubernetes/server/bin

autostart=true

autorestart=true

startsecs=30

startretries=3

exitcodes=0,2

stopsignal=QUIT

stopwaitsecs=10

user=root

redirect_stderr=true

stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log

stdout_logfile_maxbytes=64MB

stdout_logfile_backups=4

stdout_capture_maxbytes=1MB

stdout_events_enabled=false

killasgroup=true

stopasgroup=true


[root@hdss7-21 ~]# supervisorctl update

kube-scheduler-7-21: stopped

kube-scheduler-7-21: updated process group

[root@hdss7-21 ~]# supervisorctl status

etcd-server-7-21                 RUNNING   pid 23637, uptime 1 day, 0:26:53

kube-apiserver-7-21              RUNNING   pid 32591, uptime 2:06:22

kube-controller-manager-7-21     RUNNING   pid 33357, uptime 0:10:37

kube-scheduler-7-21              RUNNING   pid 33450, uptime 0:01:18

3.6. 检查主控节点状态

[root@hdss7-21 ~]# ln -s /opt/apps/kubernetes/server/bin/kubectl /usr/local/bin/

[root@hdss7-21 ~]# kubectl get cs

NAME                 STATUS    MESSAGE              ERROR

scheduler            Healthy   ok

controller-manager   Healthy   ok

etcd-1               Healthy   {"health": "true"}

etcd-0               Healthy   {"health": "true"}

etcd-2               Healthy   {"health": "true"}


[root@hdss7-22 ~]# ln -s /opt/apps/kubernetes/server/bin/kubectl /usr/local/bin/

[root@hdss7-22 ~]# kubectl get cs

NAME                 STATUS    MESSAGE              ERROR

controller-manager   Healthy   ok

scheduler            Healthy   ok

etcd-2               Healthy   {"health": "true"}

etcd-1               Healthy   {"health": "true"}

etcd-0               Healthy   {"health": "true"}

4. 运算节点部署

4.1. kubelet 部署

4.1.1. 签发证书

证书签发在 hdss7-200 操作

[root@hdss7-200 ~]# cd /opt/certs/

[root@hdss7-200 certs]# vim kubelet-csr.json # 将所有可能的kubelet机器IP添加到hosts中

{

    "CN": "k8s-kubelet",

    "hosts": [

    "127.0.0.1",

    "10.4.7.10",

    "10.4.7.21",

    "10.4.7.22",

    "10.4.7.23",

    "10.4.7.24",

    "10.4.7.25",

    "10.4.7.26",

    "10.4.7.27",

    "10.4.7.28"

    ],

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "ST": "beijing",

            "L": "beijing",

            "O": "od",

            "OU": "ops"

        }

    ]

}

[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet

2020/01/06 23:10:56 [INFO] generate received request

2020/01/06 23:10:56 [INFO] received CSR

2020/01/06 23:10:56 [INFO] generating key: rsa-2048

2020/01/06 23:10:56 [INFO] encoded CSR

2020/01/06 23:10:56 [INFO] signed certificate with serial number 61221942784856969738771370531559555767101820379

2020/01/06 23:10:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for

websites. For more information see the Baseline Requirements for the Issuance and Management

of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);

specifically, section 10.2.3 ("Information Requirements").

[root@hdss7-200 certs]# ls kubelet* -l

-rw-r--r-- 1 root root 1115 Jan  6 23:10 kubelet.csr

-rw-r--r-- 1 root root  452 Jan  6 23:10 kubelet-csr.json

-rw------- 1 root root 1675 Jan  6 23:10 kubelet-key.pem

-rw-r--r-- 1 root root 1468 Jan  6 23:10 kubelet.pem

[root@hdss7-200 certs]# for i in 21 22;do scp kubelet.pem kubelet-key.pem hdss7-$i:/opt/apps/kubernetes/server/bin/certs/;done

4.1.2. 创建kubelet配置

kubelet配置在 hdss7-21 hdss7-22 操作

  • set-cluster # 创建需要连接的集群信息，可以创建多个k8s集群信息
[root@hdss7-21 ~]# kubectl config set-cluster myk8s \

--certificate-authority=/opt/apps/kubernetes/server/bin/certs/ca.pem \

--embed-certs=true \

--server=https://10.4.7.10:7443 \

--kubeconfig=/opt/apps/kubernetes/conf/kubelet.kubeconfig
  • set-credentials # 创建用户账号，即用户登陆使用的客户端私有和证书，可以创建多个证书
[root@hdss7-21 ~]# kubectl config set-credentials k8s-node \

--client-certificate=/opt/apps/kubernetes/server/bin/certs/client.pem \

--client-key=/opt/apps/kubernetes/server/bin/certs/client-key.pem \

--embed-certs=true \

--kubeconfig=/opt/apps/kubernetes/conf/kubelet.kubeconfig
  • set-context # 设置context，即确定账号和集群对应关系
[root@hdss7-21 ~]# kubectl config set-context myk8s-context \

--cluster=myk8s \

--user=k8s-node \

--kubeconfig=/opt/apps/kubernetes/conf/kubelet.kubeconfig
  • use-context # 设置当前使用哪个context
[root@hdss7-21 ~]# kubectl config use-context myk8s-context --kubeconfig=/opt/apps/kubernetes/conf/kubelet.kubeconfig

4.1.3. 授权k8s-node用户

此步骤只需要在一台master节点执行

授权 k8s-node 用户绑定集群角色 system:node ，让 k8s-node 成为具备运算节点的权限。

[root@hdss7-21 ~]# vim k8s-node.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: k8s-node

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: system:node

subjects:

- apiGroup: rbac.authorization.k8s.io

  kind: User

  name: k8s-node

[root@hdss7-21 ~]# kubectl create -f k8s-node.yaml

clusterrolebinding.rbac.authorization.k8s.io/k8s-node created

[root@hdss7-21 ~]# kubectl get clusterrolebinding k8s-node

NAME       AGE

k8s-node   36s

4.1.4. 装备pause镜像

将pause镜像放入到harbor私有仓库中，仅在 hdss7-200 操作：

[root@hdss7-200 ~]# docker image pull kubernetes/pause

[root@hdss7-200 ~]# docker image tag kubernetes/pause:latest harbor.od.com/public/pause:latest

[root@hdss7-200 ~]# docker login -u admin harbor.od.com

[root@hdss7-200 ~]# docker image push harbor.od.com/public/pause:latest

4.1.5. 创建启动脚本

在node节点创建脚本并启动kubelet，涉及服务器： hdss7-21 hdss7-22

[root@hdss7-21 ~]# vim /opt/apps/kubernetes/server/bin/kubelet-startup.sh

#!/bin/sh

# 注意修改主机名

WORK_DIR=$(dirname $(readlink -f $0))

[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/kubernetes/server/bin/kubelet \

    --anonymous-auth=false \

    --cgroup-driver systemd \

    --cluster-dns 192.168.0.2 \

    --cluster-domain cluster.local \

    --runtime-cgroups=/systemd/system.slice \

    --kubelet-cgroups=/systemd/system.slice \

    --fail-swap-on="false" \

    --client-ca-file ./certs/ca.pem \

    --tls-cert-file ./certs/kubelet.pem \

    --tls-private-key-file ./certs/kubelet-key.pem \

    --hostname-override hdss7-21.host.com \

    --image-gc-high-threshold 20 \

    --image-gc-low-threshold 10 \

    --kubeconfig ../../conf/kubelet.kubeconfig \

    --log-dir /data/logs/kubernetes/kube-kubelet \

    --pod-infra-container-image harbor.od.com/public/pause:latest \

    --root-dir /data/kubelet

[root@hdss7-21 ~]# chmod u+x /opt/apps/kubernetes/server/bin/kubelet-startup.sh

[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet

[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-kubelet.ini

[program:kube-kubelet-7-21]

command=/opt/apps/kubernetes/server/bin/kubelet-startup.sh

numprocs=1

directory=/opt/apps/kubernetes/server/bin

autostart=true

autorestart=true

startsecs=30

startretries=3

exitcodes=0,2

stopsignal=QUIT

stopwaitsecs=10

user=root

redirect_stderr=true

stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log

stdout_logfile_maxbytes=64MB

stdout_logfile_backups=5

stdout_capture_maxbytes=1MB

stdout_events_enabled=false

killasgroup=true

stopasgroup=true


[root@hdss7-21 ~]# supervisorctl update

[root@hdss7-21 ~]# supervisorctl status

etcd-server-7-21                 RUNNING   pid 23637, uptime 1 day, 14:56:25

kube-apiserver-7-21              RUNNING   pid 32591, uptime 16:35:54

kube-controller-manager-7-21     RUNNING   pid 33357, uptime 14:40:09

kube-kubelet-7-21                RUNNING   pid 37232, uptime 0:01:08

kube-scheduler-7-21              RUNNING   pid 33450, uptime 14:30:50

[root@hdss7-21 ~]# kubectl get node

NAME                STATUS   ROLES    AGE     VERSION

hdss7-21.host.com   Ready    <none>   3m13s   v1.15.2

hdss7-22.host.com   Ready    <none>   3m13s   v1.15.2

4.1.6. 修改节点角色

使用 kubectl get nodes 获取的Node节点角色为空，可以按照以下方式修改,这个操作只是在节点上面打了个标签。

[root@hdss7-21 ~]# kubectl get node

NAME                STATUS   ROLES    AGE     VERSION

hdss7-21.host.com   Ready    <none>   3m13s   v1.15.2

hdss7-22.host.com   Ready    <none>   3m13s   v1.15.2

[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/node=

node/hdss7-21.host.com labeled

[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/master=

node/hdss7-21.host.com labeled

[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/master=

node/hdss7-22.host.com labeled

[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/node=

node/hdss7-22.host.com labeled

[root@hdss7-21 ~]# kubectl get node

NAME                STATUS   ROLES         AGE     VERSION

hdss7-21.host.com   Ready    master,node   7m44s   v1.15.2

hdss7-22.host.com   Ready    master,node   7m44s   v1.15.2

4.2. kube-proxy部署

4.2.1. 签发证书

证书签发在 hdss7-200 操作

[root@hdss7-200 ~]# cd /opt/certs/

# CN 其实是k8s中的角色

[root@hdss7-200 certs]# cat>>kube-proxy-csr.json<<EOF

{

    "CN": "system:kube-proxy",

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "ST": "beijing",

            "L": "beijing",

            "O": "od",

            "OU": "ops"

        }

    ]

}

EOF

[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json |cfssl-json -bare kube-proxy-client

2020/01/07 21:45:53 [INFO] generate received request

2020/01/07 21:45:53 [INFO] received CSR

2020/01/07 21:45:53 [INFO] generating key: rsa-2048

2020/01/07 21:45:53 [INFO] encoded CSR

2020/01/07 21:45:53 [INFO] signed certificate with serial number 620191685968917036075463174423999296907693104226

2020/01/07 21:45:53 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for

websites. For more information see the Baseline Requirements for the Issuance and Management

of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);

# 因为kube-proxy使用的用户是kube-proxy，不能使用client证书，必须要重新签发自己的证书

[root@hdss7-200 certs]# ls kube-proxy-c* -l

-rw-r--r-- 1 root root 1005 Jan  7 21:45 kube-proxy-client.csr

-rw------- 1 root root 1675 Jan  7 21:45 kube-proxy-client-key.pem

-rw-r--r-- 1 root root 1375 Jan  7 21:45 kube-proxy-client.pem

-rw-r--r-- 1 root root  267 Jan  7 21:45 kube-proxy-csr.json

[root@hdss7-200 certs]# scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-21:/opt/apps/kubernetes/server/bin/certs/                                                                         100% 1375   870.6KB/s   00:00

[root@hdss7-200 certs]# scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-22:/opt/apps/kubernetes/server/bin/certs/

for i in 21 22;do scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-$i:/opt/apps/kubernetes/server/bin/certs/;done

4.2.2. 创建kube-proxy配置

在所有node节点创建，涉及服务器：hdss7-21 ，hdss7-22

[root@hdss7-21 ~]# kubectl config set-cluster myk8s \

--certificate-authority=/opt/apps/kubernetes/server/bin/certs/ca.pem \

--embed-certs=true \

--server=https://10.4.7.10:7443 \

--kubeconfig=/opt/apps/kubernetes/conf/kube-proxy.kubeconfig

[root@hdss7-21 ~]# kubectl config set-credentials kube-proxy \

--client-certificate=/opt/apps/kubernetes/server/bin/certs/kube-proxy-client.pem \

--client-key=/opt/apps/kubernetes/server/bin/certs/kube-proxy-client-key.pem \

--embed-certs=true \

--kubeconfig=/opt/apps/kubernetes/conf/kube-proxy.kubeconfig

[root@hdss7-21 ~]# kubectl config set-context myk8s-context \

--cluster=myk8s \

--user=kube-proxy \

--kubeconfig=/opt/apps/kubernetes/conf/kube-proxy.kubeconfig

[root@hdss7-21 ~]# kubectl config use-context myk8s-context --kubeconfig=/opt/apps/kubernetes/conf/kube-proxy.kubeconfig

4.2.3. 加载ipvs模块

kube-proxy 共有3种流量调度模式，分别是 namespace，iptables，ipvs，其中ipvs性能最好。

[root@hdss7-21 ~]# for i in $(ls /usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs|grep -o "^[^.]*");do echo $i; /sbin/modinfo -F filename $i >/dev/null 2>&1 && /sbin/modprobe $i;done

[root@hdss7-21 ~]# lsmod | grep ip_vs  # 查看ipvs模块

4.2.4. 创建启动脚本

# 注意更改主机名

[root@hdss7-21 ~]# vim /opt/apps/kubernetes/server/bin/kube-proxy-startup.sh

#!/bin/sh

WORK_DIR=$(dirname $(readlink -f $0))

[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/kubernetes/server/bin/kube-proxy \

  --cluster-cidr 172.7.0.0/16 \

  --hostname-override hdss7-21.host.com \

  --proxy-mode=ipvs \

  --ipvs-scheduler=nq \

  --kubeconfig ../../conf/kube-proxy.kubeconfig

[root@hdss7-21 ~]# chmod u+x /opt/apps/kubernetes/server/bin/kube-proxy-startup.sh

[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-proxy

[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-proxy.ini

[program:kube-proxy-7-21]

command=/opt/apps/kubernetes/server/bin/kube-proxy-startup.sh

numprocs=1

directory=/opt/apps/kubernetes/server/bin

autostart=true

autorestart=true

startsecs=30

startretries=3

exitcodes=0,2

stopsignal=QUIT

stopwaitsecs=10

user=root

redirect_stderr=true

stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log

stdout_logfile_maxbytes=64MB

stdout_logfile_backups=5

stdout_capture_maxbytes=1MB

stdout_events_enabled=false

killasgroup=true

stopasgroup=true

[root@hdss7-21 ~]# supervisorctl update

4.2.5. 验证集群

[root@hdss7-21 ~]# supervisorctl status

etcd-server-7-21                 RUNNING   pid 23637, uptime 2 days, 0:27:18

kube-apiserver-7-21              RUNNING   pid 32591, uptime 1 day, 2:06:47

kube-controller-manager-7-21     RUNNING   pid 33357, uptime 1 day, 0:11:02

kube-kubelet-7-21                RUNNING   pid 37232, uptime 9:32:01

kube-proxy-7-21                  RUNNING   pid 47088, uptime 0:06:19

kube-scheduler-7-21              RUNNING   pid 33450, uptime 1 day, 0:01:43

[root@hdss7-21 ~]# yum install -y ipvsadm

[root@hdss7-21 ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  192.168.0.1:443 nq

  -> 10.4.7.21:6443               Masq    1      0          0

  -> 10.4.7.22:6443               Masq    1      0          0


cat >/root/nginx-ds.yaml <<'EOF'

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

  name: nginx-ds

spec:

  template:

    metadata:

      labels:

        app: nginx-ds

    spec:

      containers:

      - name: my-nginx

        image: harbor.od.com/public/nginx:v1.17.9

        ports:

        - containerPort: 80

EOF

[root@hdss7-21 ~]# kubectl apply -f nginx-ds.yaml

[root@hdss7-21 ~]# kubectl get pods -o wide

NAME             READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES

nginx-ds-cfc8d   1/1     Running   0          64s   172.7.22.2   hdss7-22.host.com   <none>           <none>

nginx-ds-vd4mk   1/1     Running   0          64s   172.7.21.2   hdss7-21.host.com   <none>           <none>


[root@hdss7-21 ~]# curl -I 172.7.21.2

HTTP/1.1 200 OK

Server: nginx/1.17.6

Date: Tue, 07 Jan 2020 14:28:46 GMT

Content-Type: text/html

Content-Length: 612

Last-Modified: Tue, 19 Nov 2019 12:50:08 GMT

Connection: keep-alive

ETag: "5dd3e500-264"

Accept-Ranges: bytes

[root@hdss7-21 ~]# curl -I 172.7.22.2  # 缺少网络插件，无法跨节点通信

二进制部署K8S-2集群部署的更多相关文章

  1. 这一篇 K8S(Kubernetes)集群部署 我觉得还可以！！！

    点赞再看,养成习惯,微信搜索[牧小农]关注我获取更多资讯,风里雨里,小农等你,很高兴能够成为你的朋友. 国内安装K8S的四种途径 Kubernetes 的安装其实并不复杂,因为Kubernetes 属 ...

  2. NetCore在Centos7上部署和Nginx集群部署访问

    NetCore在Linux上部署 工具:WMWare虚拟机,Wmware12,CentOS7ISO镜像,VS2017 1.安装虚拟机,过程略,网上一搜一大把 2.用VS2017建一个NetCore的W ...

  3. K8s 离线集群部署（二进制包无dashboard）

    https://www.cnblogs.com/cocowool/p/install_k8s_offline.html https://www.jianshu.com/p/073577bdec98 h ...

  4. 内网环境上部署k8s+docker集群：集群ftp的yum源配置

    接触docker已经有一年了,想把做的时候的一些知识分享给大家. 因为公司机房是内网环境无法连接外网,所以这里所有的部署都是基于内网环境进行的. 首先,需要通过ftp服务制作本地的yum源,可以从ht ...

  5. Kubernetes集群部署之三ETCD集群部署

    kuberntes 系统使用 etcd 存储所有数据,本文档介绍部署一个三节点高可用 etcd 集群的步骤,这三个节点复用 kubernetes 集群机器k8s-master.k8s-node-1.k ...

  6. docker 部署 HFish（集群部署）

    主节点部署: docker run -d --name hfish-master -p : -p : -p : -p : -p : -p : -p : -p : -p : -p : -p : -p : ...

  7. kubernetes kubeadm部署高可用集群

    k8s kubeadm部署高可用集群 kubeadm是官方推出的部署工具,旨在降低kubernetes使用门槛与提高集群部署的便捷性. 同时越来越多的官方文档,围绕kubernetes容器化部署为环境 ...

  8. 使用kubeadm进行单master(single master)和高可用(HA)kubernetes集群部署

    kubeadm部署k8s 使用kubeadm进行k8s的部署主要分为以下几个步骤: 环境预装: 主要安装docker.kubeadm等相关工具. 集群部署: 集群部署分为single master(单 ...

  9. 谈一谈Elasticsearch的集群部署

      Elasticsearch天生就支持分布式部署,通过集群部署可以提高系统的可用性.本文重点谈一谈Elasticsearch的集群节点相关问题,搞清楚这些是进行Elasticsearch集群部署和拓 ...

  10. 分布式监控工具Ganglia 介绍 与 集群部署.

    如果你目的很明确就是冲着标题来的,不爱看我唠叨,请直接进入第二个分割线之后的内容. 其实之前就是有做Swift监控平台的打算的,但是因为没什么硬性需求么,也不要紧的,就一直搁置了.最近实验室来了个大二 ...

随机推荐

  1. 【RocketMQ源码分析】深入消息存储（3）

    前文回顾 CommitLog篇 --[RocketMQ源码分析]深入消息存储(1) ConsumeQueue篇 --[RocketMQ源码分析]深入消息存储(2) 前面两篇已经说过了消息如何存储到Co ...

  2. WordPress的SEO优化技巧

    世界上大约有30%的网站都是由Wordpress搭建的,因为Wordpress自身构架清晰,代码规范,且网页评论直接书写在整个页面里,能够被搜索引擎检索到,因此对搜索引擎很友好.但有时候还是会出现只被 ...

  3. Dubbo 编解码那些事

    一.背景 笔者在一次维护基础公共组件的过程中,不小心修改了类的包路径.糟糕的是,这个类被各业务在facade中进行了引用.传递.幸运的是,同一个类,在提供者和消费者的包路径不一致,没有引起各业务报错. ...

  4. OOUML系列总结及终章回顾

    盼望着,盼望着,OO课程终于结束了,但是,此刻的我却感到一丝失落,甚至想着再来一单元岂不妙哉? 目录 总结本单元三次作业架构 四个单元中架构设计及OO方法理解的演进 四个单元中测试理解与实践的演进 课 ...

  5. 【CTF】WDCTF-2017 3-1 writeup

    题目来源:WDCTF-2017 题目链接:https://adworld.xctf.org.cn/task/answer?type=misc&number=1&grade=1& ...

  6. linux中mysql连接不上，服务启动失败等问题解决

    confluence问题解决方式 1)针对confluence访问页面报500与连接失败等问题 首先我们登陆部署confluence的服器 10.15.4.115 2)重启mysql服务,发现重启失败 ...

  7. 构建基于表单配置的 Jenkins 测试项目（接口、UI、APP、Jmeter）

    1. 第一个 hello world 项目 2. 构建自动触发的项目(接口测试) 1)新建测试项目(执行测试脚本) 2)新建 Maven 打包项目 3)手动执行构建 4)修改 Web 工程代码并 pu ...

  8. overflow和absolute之间的问题，transfrom可以解决

    CSS代码: .overflow { width: 191px; height: 191px; border: 2px solid #beceeb; overflow: hidden; } .over ...

  9. poj3648 2-sat

    题意:       一对夫妻结婚,请来了n对夫妻,所有人坐在同一张桌子上吃饭,这张桌子是长方形的可以坐两排人,有如下限制 (1) 新娘和新郎不做同一侧. (2) 每对来的夫妻也不能坐在同一侧.  (3 ...

  10. hdu4974 简单题

    题意:       一个人看比赛,这些比赛一共有n个人参与,每一场有两个人比,然后每一场之后这个人都会给比赛的这两个人打分,最多1最少0,比如看完了A,B两人比赛,他可能给这两个人分别的分数是00,1 ...