升级最新版Rancher 2.2.6

前言：之前采用离线方式部署好了 Rancher 2.2.4（https://www.cnblogs.com/weavepub/p/11053099.html），这次升级到最新版本 Rancher 2.2.6。

一、同步镜像

1.1 安装docker

新启动一台虚拟主机，安装docker（不要用生产环境，免得主机上的镜像太多）

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce -y
systemctl restart docker
systemctl enable docker
systemctl status docker

1.2 下载镜像列表文件

mkdir -p /opt/rancher/upgrade2.2.6 && cd /opt/rancher/upgrade2.2.6
https://github.com/rancher/rancher/releases/tag/v2.2.6/rancher-images.txt

1.3准备镜像脚本

1）镜像下载脚本

vim rancher-save-images.sh

内容如下：

#!/bin/bash
# 定义日志
workdir=`pwd`
log_file=${workdir}/sync_images_$(date +"%Y-%m-%d").log

logger()
{
    log=$
    cur_time='['$(date +"%Y-%m-%d %H:%M:%S")']'
    echo ${cur_time} ${log} | tee -a ${log_file}
}

list="rancher-images.txt"
#images="rancher-images.tar.gz"

POSITIONAL=()
while [[ $# -gt  ]]; do
    key="$1"
    case $key in
        -i|--images)
        images="$2"
        shift # past argument
        shift # past value
        ;;
        -l|--image-list)
        list="$2"
        shift # past argument
        shift # past value
        ;;
        -h|--help)
        help="true"
        shift
    ;;
    esac
done

usage () {
    echo "USAGE: $0 [--image-list rancher-images.txt] [--images rancher-images.tar.gz]"
    echo "  [-l|--images-list path] text file with list of images. 1 per line."
    echo "  [-l|--images path] tar.gz generated by docker save."
    echo "  [-h|--help] Usage message"
}

if [[ $help ]]; then
    usage
    exit
fi

set -e -x

mkdir -p rancher-images-$(date +"%Y-%m-%d")
cd rancher-images-$(date +"%Y-%m-%d")

for i in $(cat ${list});
do
    docker pull ${i}

    if [ $? -ne  ]; then
        logger "${i} pull failed."
    else
        logger "${i} pull successfully."
    fi

    docker save ${i} | gzip > $(echo $i | sed "s#/#-#g; s#:#-#g").tgz

    if [ $? -ne  ]; then
        logger "${i} save failed."
    else
        logger "${i} save successfully."
    fi
done

2）镜像上传到私有仓库脚本

vim rancher-push-images.sh

内容如下：

#!/bin/bash

## 镜像上传说明
# 需要先在镜像仓库中创建 rancher 项目
# 根据实际情况更改以下私有仓库地址

# 定义日志
workdir=`pwd`
log_file=${workdir}/sync_images_$(date +"%Y-%m-%d").log

logger()
{
    log=$
    cur_time='['$(date +"%Y-%m-%d %H:%M:%S")']'
    echo ${cur_time} ${log} | tee -a ${log_file}
}

images_hub() {

    while true; do
        read -p "输入镜像仓库地址(不加http/https): " registry
        read -p "输入镜像仓库用户名: " registry_user
        read -p "输入镜像仓库用户密码: " registry_password
        echo "您设置的仓库地址为: ${registry},用户名: ${registry_user},密码: xxx"
        read -p "是否确认(Y/N): " confirm

        if [ $confirm != Y ] && [ $confirm != y ] && [ $confirm == '' ]; then
            echo "输入不能为空，重新输入"
        else
            break
        fi
    done
}

images_hub

echo "镜像仓库 $(docker login -u ${registry_user} -p ${registry_password} ${registry})"

images=$(cat ./rancher-images.txt)
namespace=rancher

docker_push() {
    for imgs in $(echo ${images}); do
        n=$(echo ${imgs} | awk -F"/" '{print NF-1}')
        #如果镜像名中没有/，那么此镜像一定是library仓库的镜像；
        if [ ${n} -eq  ]; then
            img_tag=${imgs}
            #namespace=rancher
            #重命名镜像
            docker tag ${imgs} ${registry}/${namespace}/${img_tag}
            #删除原始镜像
            #docker rmi ${imgs}
            #上传镜像
            docker push ${registry}/${namespace}/${img_tag}
        #如果镜像名中有一个/，那么/左侧为项目名，右侧为镜像名和tag
        elif [ ${n} -eq  ]; then
            img_tag=$(echo ${imgs} | awk -F"/" '{print $2}')
            #namespace=$(echo ${imgs} | awk -F"/" '{print $1}')
            #重命名镜像
            docker tag ${imgs} ${registry}/${namespace}/${img_tag}
            #删除旧镜像
            #docker rmi ${imgs}
            #上传镜像
            docker push ${registry}/${namespace}/${img_tag}
        #如果镜像名中有两个/，
        elif [ ${n} -eq  ]; then
            img_tag=$(echo ${imgs} | awk -F"/" '{print $3}')
            #namespace=$(echo ${imgs} | awk -F"/" '{print $2}')
            #重命名镜像
            docker tag ${imgs} ${registry}/${namespace}/${img_tag}
            #删除旧镜像
            #docker rmi ${imgs}
            #上传镜像
            docker push ${registry}/${namespace}/${img_tag}
        else
            #标准镜像为四层结构，即：仓库地址/项目名/镜像名:tag,如不符合此标准，即为非有效镜像。
            echo "No available images"
        fi
    done
}

docker_push

1.4 同步镜像

1）下载镜像

chmod +x rancher-save-images.sh
./rancher-save-images.sh --image-list /opt/rancher/upgrade2.2.6/rancher-images.txt

2）上传镜像

chmod +x rancher-push-images.sh
./rancher-push-images.sh
reg.nexus.wmqe.com
admin
*******

二、升级Rancher

2.1 下载 Rancher 2.2.6 离线包

登入到当初部署 Rancher 时的 prod-rancher-server-01 主机

1）更新本地helm repo缓存

helm repo update

2）查看Rancher 最新版本

helm search rancher
NAME                      CHART VERSION    APP VERSION    DESCRIPTION
rancher-stable/rancher    2.2.            v2.2.6         Install Rancher Server to manage Kubernetes clusters acro...

3）获取Rancher Charts离线包

mkdir -p /opt/rancher/upgrade2.2.6 && cd /opt/rancher/upgrade2.2.6
helm fetch rancher-stable/rancher --version v2.2.6

2.2 升级 Rancher

1）升级Rancher

升级参数应该以安装时设置的参数为准。

tar zxvf rancher-2.2..tgz
kubeconfig=/root/.kube/config

helm --kubeconfig=$kubeconfig upgrade rancher ./rancher \
    --set hostname=rancher.wmqe.com \
    --set ingress.tls.source=secret \
    --set auditLog.level= \
    --set auditLog.destination=hostPath \
    --set auditLog.hostPath=/opt/rancher/auditlogs/ \
    --set auditLog.maxAge= \
    --set auditLog.maxBackups= \
    --set auditLog.maxSize= \
    --set rancherImage=reg.nexus.wmqe.com/rancher/rancher

2）查看pod状态

kubectl get pod -A
NAMESPACE       NAME                                      READY   STATUS      RESTARTS   AGE
cattle-system   cattle-cluster-agent-c87688dc5-xzlqg      1/1     Running     0          27m
cattle-system   cattle-node-agent-9tl5c                   1/1     Running     0          26m
cattle-system   cattle-node-agent-vrrmr                   1/1     Running     0          27m
cattle-system   cattle-node-agent-xrdkq                   1/1     Running     0          26m
cattle-system   rancher-5ddf8d4897-lmw5v                  2/2     Running     0          28m
cattle-system   rancher-5ddf8d4897-wnfqn                  2/2     Running     0          29m
cattle-system   rancher-5ddf8d4897-xcx5s                  2/2     Running     0          28m
ingress-nginx   default-http-backend-6d4b9c4cc7-5qh4c     /     Running               41d
ingress-nginx   nginx-ingress-controller-dll87            /     Running               41d
ingress-nginx   nginx-ingress-controller-fnwvb            /     Running               41d
ingress-nginx   nginx-ingress-controller-pzcxb            /     Running               41d
kube-system     canal-4qnsr                               /     Running               41d
kube-system     canal-b7p6v                               /     Running               41d
kube-system     canal-wv8p5                               /     Running               41d
kube-system     kube-dns-7fbcd74695-j7fx7                 /     Running               41d
kube-system     kube-dns-autoscaler-6b777db656-58xxz      /     Running               41d
kube-system     metrics-server-7c57d997f9-j7lt4           /     Running               41d
kube-system     rke-ingress-controller-deploy-job-nq7sq   /     Completed             41d
kube-system     rke-kube-dns-addon-deploy-job-5flrd       /     Completed             41d
kube-system     rke-metrics-addon-deploy-job-8tzcr        /     Completed             41d
kube-system     rke-network-plugin-deploy-job-vq4gh       /     Completed             41d
kube-system     tiller-deploy-768c796496-c4cd2            /     Running               41d

标红的都重新创建后，就可以成功访问Rancher主界面了。

三、清理节点

当升级时候，某一台节点若升级失败，则需清理该节点，然后重新加入，下面介绍清理节点的步骤：

1、在 rancher 中剔除主机

2、登入该主机删除所有容器

docker rm -f $(docker ps -qa)

3、删除所有容器卷

docker volume rm $(docker volume ls -q)

4、备份目录

mv /etc/kubernetes /etc/kubernetes-bak-$(date +"%Y%m%d%H%M")
mv /var/lib/etcd /var/lib/etcd-bak-$(date +"%Y%m%d%H%M")
mv /var/lib/rancher /var/lib/rancher-bak-$(date +"%Y%m%d%H%M")
mv /opt/rke /opt/rke-bak-$(date +"%Y%m%d%H%M")

5、删除残留路径

rm -rf /etc/ceph \
/etc/cni \
/opt/cni \
/run/secrets/kubernetes.io \
/run/calico \
/run/flannel \
/var/lib/calico \
/var/lib/cni \
/var/lib/kubelet \
/var/log/containers \
/var/log/pods \
/var/run/calico

6、重启docker

systemctl restart docker

参考：https://www.cnrancher.com/docs/rancher/v2.x/cn/install-prepare/remove-node/