1.openshift搭建

第1章 主机规划和所需文件

1.1 主机规划

IP地址	域名	用途
11.11.233.125	master01.song.test.cnpc	容器编排、etcd
11.11.233.126	master02.song.test.cnpc	容器编排、etcd
11.11.233.134	master03.song.test.cnpc	容器编排、etcd
11.11.233.127	node1.song.test.cnpc	Infra
11.11.233.128	node2.song.test.cnpc	Infra
11.11.233.129	node3.song.test.cnpc	Infra
11.11.233.130	node4.song.test.cnpc	容器运行
11.11.233.131	node5.song.test.cnpc	容器运行
11.11.233.132	node6.song.test.cnpc	容器运行
11.11.233.133	ha.song.test.cnpc registry.song.test.cnpc	Haproxy，registry

1.2 主机环境检测与确认

通过top，free，lsblk等命令检测各服务器的硬件配置是否符合规划

在registry主机上安装ansible 并执行一下play

1）网络配置检测

检测各服务器网络配置是否正确，包含ip地址，网络连通性，bond配置等。

注：bond的故障模拟测试在机房服务器配置网络过程中完成。

2）时区检测

使用date命令检测各服务器时区是否为CST。

运行ntpq -p或chronyc sources -v查看是否配置NTP。

3）主机名检测

检测各服务器的主机名是否符合规划。如果未在安装期间配置，则后续执行命令修改。

4）检测所有服务器libvirtd服务是否处于停止状态

# systemctl stop libvirtd

# systemctl disable libvirtd

# systemctl mask libvirtd

关闭服务之后重启服务器即可。

5）所有节点关闭firewalld

# systemctl stop firewalld

# systemctl disable firewalld

# systemctl mask firewalld

6）所有节点关闭selinux

# setenforce 0;

# sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config;

NetworkManager

master，node，haproxy节点的selinux不要关。默认为开启，不要改

NetworkManager默认开启，不要关

[master]

11.11.233.125   name=master01

11.11.233.126   name=master02

11.11.233.134   name=master03

[node]

11.11.233.127   name=node1

11.11.233.128   name=node2

11.11.233.129   name=node3

11.11.233.130   name=node4

11.11.233.131   name=node5

11.11.233.132   name=node6

[other]

11.11.233.133   name=ha

[test:children]

master

node

other

[test:vars]

ansible_ssh_user=sysadm

ansible_ssh_pass=Passc0de@tpcpjl

1.3 搭建yum仓库和docker仓库（ocp3.6）

OpenShift 3安装及运行依赖的RPM、Docker镜像及程序，需要在联网环境下预先下载。

需要下载的文件列表如下：

名称	备注
YUM源镜像	OpenShift安装所依赖的YUM Repo： l   rhel-7-server-extras-rpms-3.6 l   rhel-7-server-ose-3.6-rpms l   rhel-7-fast-datapath-rpms-3.6
Docker镜像	OpenShift运行所依赖的Docker镜像 l   jenkins-2-rhel7-latest.tar.gz l   logging-deployer-v3.6.tar.gz l   metrics-deployer-v3.6.tar.gz l   ose-haproxy-router-v3.6.173.0.96.tar.gz l   jenkins-slave-maven-rhel7-latest.tar.gz l   logging-elasticsearch-v3.6.tar.gz l   metrics-hawkular-openshift-agent-v3.6.tar.gz l   ose-pod-v3.6.173.0.96.tar.gz l   jenkins-slave-nodejs-rhel7-latest.tar.gz l   logging-fluentd-v3.6.tar.gz l   metrics-heapster-v3.6.tar.gz l   ose-sti-builder-v3.6.173.0.96.tar.gz l   logging-auth-proxy-v3.6.tar.gz l   logging-kibana-v3.6.tar.gz l   ose-deployer-v3.6.173.0.96.tar.gz l   registry-console-v3.6.tar.gz l   logging-curator-v3.6.tar.gz l   metrics-cassandra-v3.6.tar.gz l   ose-docker-registry-v3.6.173.0.96.tar.

[root@ha ~]# tree -L 3 /mnt/

/mnt/

├── registry

│   └── docker

│       └── registry

└── yum

├── rhel-7-fast-datapath-rpms

│   ├── Packages

│   └── repodata

├── rhel-7-server-ansible-2.4-rpms

│   ├── Packages

│   └── repodata

├── rhel-7-server-extras-rpms

│   ├── Packages

│   └── repodata

├── rhel-7-server-ose-3.6-rpms

│   ├── Packages

│   └── repodata

├── rhel-7-server-ose-3.7-rpms

│   ├── Packages

│   └── repodata

├── rhel-7-server-ose-3.8-rpms

│   ├── Packages

│   └── repodata

├── rhel-7-server-ose-3.9-rpms

│   ├── Packages

│   └── repodata

└── rhel-7-server-rpms

├── Packages

└── repodata

配置好httpd和 repo文件

[root@ha ~]# cat /etc/yum.repos.d/redhat7.3.repo

[server-ose-3.9-rpms]

baseurl = http://11.11.233.133/rhel-7-server-ose-3.9-rpms

name = Red Hat OpenShift Container Platform 3.9  RPMs

enabled = 0

gpgcheck = 0

[rhel-7-server-ose-3.6-rpms]

name = rhel-7-server-ose-3.6-rpms

baseurl = http://11.11.233.133/rhel-7-server-ose-3.6-rpms/

gpgcheck = 0

enabled = 1

[rhel-7-server-ose-3.8-rpms]

baseurl = http://11.11.233.133/rhel-7-server-ose-3.8-rpms

name = Red Hat OpenShift Container Platform 3.8  RPMs

enabled = 0

gpgcheck = 0

[rhel-7-server-ose-3.7-rpms]

baseurl = http://11.11.233.133/rhel-7-server-ose-3.7-rpms

name = Red Hat OpenShift Container Platform 3.7  RPMs

enabled = 0

gpgcheck = 0

[rhel-7-server-extras-rpms]

baseurl = http://11.11.233.133/rhel-7-server-extras-rpms

name = Red Hat rhel-7-server-extras-rpms  RPMs

enabled = 1

gpgcheck = 0

[rhel-7-fast-datapath-rpms]

baseurl = http://11.11.233.133/rhel-7-fast-datapath-rpms

name = Red Hat rhel-7-fast-datapath-rpms  RPMs

enabled = 1

gpgcheck = 0

[rhel-7-server-ansible-2.4-rpms]

baseurl = http://11.11.233.133/rhel-7-server-ansible-2.4-rpms

name = Red Hat rhel-7-server-ansible-2.4-rpms  RPMs

enabled = 1

gpgcheck = 0

[rhel-7-server-rpms]

baseurl = http://11.11.233.133/rhel-7-server-rpms

name = Red Hat rhel-7-server-rpms  RPMs

enabled = 1

gpgcheck = 0

[root@ha ~]# yum repolist

Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

repo id                                                                              repo name                                                                                           status

rhel-7-fast-datapath-rpms                                                            Red Hat rhel-7-fast-datapath-rpms  RPMs                                                                38

rhel-7-server-ansible-2.4-rpms                                                       Red Hat rhel-7-server-ansible-2.4-rpms  RPMs                                                           10

rhel-7-server-extras-rpms                                                            Red Hat rhel-7-server-extras-rpms  RPMs                                                               141

rhel-7-server-ose-3.6-rpms                                                           rhel-7-server-ose-3.6-rpms                                                                            483

rhel-7-server-rpms                                                                   Red Hat rhel-7-server-rpms  RPMs

仓库使用 docker-registry

[root@ha ~]# cat /etc/docker-distribution/registry/config.yml

version: 0.1

log:

fields:

service: registry

storage:

cache:

layerinfo: inmemory

filesystem:

rootdirectory: /mnt/registry

http:

addr: :5000

secret: 95d5b1erc2a905586e790f794514ea38

测试镜像拉取

v3.6: Pulling from registry.song.test.cnpc:5000/openshift3/logging-curator

9cadd93b16ff: Already exists

4aa565ad8b7a: Already exists

d131575534ed: Pull complete

Digest: sha256:9a0d7cf6532da31f08239cc25e74bad118a828b4dc3a67a8bf442ff6faba140f

Status: Downloaded newer image for registry.song.test.cnpc:5000/openshift3/logging-curator:v3.6

第2章 安装OpenShift预备

2.1 安装软件包并配置基础环境

l  在所有节点上安装OpenShift需要的软件包。命令如下：

yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion vim atomic-openshift-excluder atomic-openshift-docker-excluder unzip kexec sos psacct;

yum -y update;

atomic-openshift-excluder unexclude;

l  确认SELinux为permissive状态。命令如下：

setenforce 0;

sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config;

l  所有节点关闭firewalld。命令如下：

systemctl disable firewalld;

systemctl stop  firewalld;

2.2 配置免密登录

l  在Master节点上生成SSH所需之秘钥。命令如下，应答输入请直接输入回车。

ssh-keygen;

l  在Master节点上配置Master节点到所有节点的SSH主机互信。命令如下，请根据提示输入远程主机Root账户密码。

l  如果root登录关闭，需要开启，使用如下命令：sed -i 's/PermitRootLogin no/PermitRootLogin yes/' /etc/ssh/sshd_config

l  cat /etc/ssh/sshd_config

2.3 本地DNS服务器创建与配置

每个master和node

# 因为dnsmasq服务会和libvirt服务冲突，所以此处把它干掉

yum remove libvirt -y

ps -ef |grep dnsmasq |grep -v grep |awk '{print $2}' |xargs -i kill -9 {}

systemctl disable libvirtd

systemctl stop libvirtd

2.3.1 添加dnsmasq配置

每个master节点添加wildcard域名指向。命令如下：

cat > /etc/dnsmasq.d/openshift-cluster.conf <<EOF

local=/song.test.cnpc/

address=/.apps.song.test.cnpc/11.11.233.133

EOF

若router为高可用部署，此ip应该为ha主机的ip 11.11.233.133

启动dnsmasq服务

每个master节点启动dnsmasq服务。命令如下：

systemctl restart dnsmasq;

systemctl enable dnsmasq;

2.3.2 配置iptables

每个master和node节点修改iptables规则。命令如下：

cp /etc/sysconfig/iptables /etc/sysconfig/iptables.bak.$(date "+%Y%m%d%H%M%S");

sed -i '/.*--dport 22 -j ACCEPT.*/a\-A INPUT -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT' /etc/sysconfig/iptables;

sed -i '/.*--dport 22 -j ACCEPT.*/a\-A INPUT -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT' /etc/sysconfig/iptables;

systemctl restart iptables;

systemctl restart NetworkManager;

2.3.3 配置各节点域名解析

配置每个Node节点域名解析。命令如下：

cat > /etc/dnsmasq.d/openshift-cluster-node.conf <<EOF

server=11.11.233.125

server=11.11.233.126

server=11.11.233.134

EOF

此部署方式，若第一个节点down，dns轮询到第二个节点需等5秒，会导致应用通过dns访问中断

三个ip分别为master节点ip

systemctl restart dnsmasq;

systemctl enable dnsmasq;

2.3.4 测试DNS解析

在每个node节点执行

nslookup docker-registry-default.apps.jtdjnet.cnpc

 

2.4 安装配置docker

2.4.1 安装Docker

在所有master、node，registry上安装Docker。命令如下：

yum -y install docker;    #安装docker

systemctl enable docker;

cp /etc/sysconfig/docker /etc/sysconfig/docker.bak.$(date "+%Y%m%d%H%M%S")

sed  -i s/".*OPTIONS=.*"/"OPTIONS='--selinux-enabled --log-driver=journald --insecure-registry 172.30.0.0\/16  --insecure-registry registry.song.test.cnpc:5000'"/g /etc/sysconfig/docker;

sed -i 's/registry.access.redhat.com/registry.song.test.cnpc:5000/g' /etc/sysconfig/docker

echo "BLOCK_REGISTRY='--block-registry public --block-registry registry.access.redhat.com' ">>/etc/sysconfig/docker;

 

2.4.2 配置docker存储

POC和测试可跳过。生产环境必须配置

磁盘名称先fdisk -l 看下，有的环境可能不叫sdb，叫vdb等

cat<<EOF>/etc/sysconfig/docker-storage-setup

DEVS=/dev/sdb

VG=docker-vg

SETUP_LVM_THIN_POOL=yes

EOF

docker-storage-setup

设置完后查看配置更改是否成功

cat /etc/sysconfig/docker-storage

DOCKER_STORAGE_OPTIONS="--storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/docker-docker-pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true "

 

2.4.3 启动docker

systemctl restart docker;

docker info;

 

 

返回值要有registry.song.test.cnpc:5000和172.30.0.0

第3章 OpenShift 3安装

l  在registry节点安装openshift的安装脚本

yum -y install atomic-openshift-utils

l  登录Master01节点执行安装。命令如下：

cat > /etc/ansible/hosts <<EOF

# Create an OSEv3 group that contains the masters, nodes, and etcd groups

[OSEv3:children]

masters

nodes

etcd

lb

 

# Set variables common for all OSEv3 hosts

[OSEv3:vars]

ansible_ssh_user=root

• openshift_deployment_type=openshift-enterprise

 

# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider

• openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
• openshift_master_cluster_method=native
• openshift_master_cluster_hostname=master.song.test.cnpc.cnpc
• openshift_master_cluster_public_hostname=master.song.test.cnpc.cnpc

 

 

• openshift_docker_options="--selinux-enabled --insecure-registry 172.30.0.0/16 --log-driver json-file --log-opt max-size=50M --log-opt max-file=3 --insecure-registry registry.song.test.cnpc.cnpc:5000 --add-registry registry.song.test.cnpc.cnpc:5000"
• openshift_master_default_subdomain=apps.song.test.cnpc.cnpc

 

• os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'

 

• openshift_hosted_router_selector='router=router'
• openshift_hosted_router_replicas=2
• openshift_hosted_registry_selector='infra=infra'

 

• openshift_hosted_logging_deploy=true
• openshift_logging_image_prefix=registry.song.test.cnpc.cnpc:5000/openshift3/
• openshift_logging_image_version=v3.6
• openshift_logging_public_master_url=master.song.test.cnpc.cnpc

 

• openshift_hosted_metrics_deploy=true
• openshift_metrics_image_prefix=registry.song.test.cnpc.cnpc:5000/openshift3/
• openshift_metrics_image_version=v3.6
• openshift_hosted_metrics_public_url=https://hawkular-metrics.apps.song.test.cnpc.cnpc/hawkular/metrics

 

• openshift_cockpit_deployer_prefix=registry.song.test.cnpc.cnpc:5000/openshift3/
• openshift_cockpit_deployer_version=v3.6

 

• oreg_url=registry.song.test.cnpc.cnpc:5000/openshift3/ose-${component}:${version}
• openshift_examples_modify_imagestreams=true

 

• openshift_enable_service_catalog=false

 

• openshift_disable_check="disk_availability,docker_image_availability,memory_availability,docker_storage,package_version,package_availability"

 

# host group for masters

[masters]

djmast001.song.test.cnpc.cnpc  

djmast002.song.test.cnpc.cnpc  

djmast003.song.test.cnpc.cnpc

 

[lb]

djmlbt001.song.test.cnpc.cnpc

 

# host group for etcd

[etcd]

djmast001.song.test.cnpc.cnpc  

djmast002.song.test.cnpc.cnpc  

djmast003.song.test.cnpc.cnpc

 

# host group for nodes, includes region info

[nodes]

djmast001.song.test.cnpc.cnpc  

djmast002.song.test.cnpc.cnpc  

djmast003.song.test.cnpc.cnpc

djinft001.song.test.cnpc.cnpc  openshift_node_labels="{'region': 'infra', 'zone': 'default', 'router': 'router'}"

djinft002.song.test.cnpc.cnpc  openshift_node_labels="{'region': 'infra', 'zone': 'default', 'router': 'router'}"

djinft003.song.test.cnpc.cnpc  openshift_node_labels="{'region': 'infra', 'zone': 'default', 'infra': 'infra'}"

djnodt001.song.test.cnpc.cnpc  openshift_node_labels="{'region': 'primary', 'zone': 'zone1'}"

djnodt002.song.test.cnpc.cnpc  openshift_node_labels="{'region': 'primary', 'zone': 'zone2'}"

djnodt003.song.test.cnpc.cnpc  openshift_node_labels="{'region': 'primary', 'zone': 'zone3'}"

djnodt004.song.test.cnpc.cnpc  openshift_node_labels="{'region': 'primary', 'zone': 'zone4'}"

djnodt005.song.test.cnpc.cnpc  openshift_node_labels="{'region': 'primary', 'zone': 'zone5'}"

EOF

 

执行安装

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml;

 

备用卸载命令：

ansible-playbook  /usr/share/ansible/openshift-ansible/playbooks/adhoc/uninstall.yml;

备注：在安装的过程中会出现下面问题，Wait for API to become available，这是在调用API接口时找不到对应文件，就会一直尝试连接

l  重启sshd服务，命令如下：systemctl restart sshd