How To Build Kubernetes Platform (构建Kubernetes平台方案参考）

• Architecture

• Architecture Diagram

• Non-Prod Environment
• Prod Environment

• Cluster Networking
• Container RepositorySetup

• Minikube For Dev Env
• Kubeadm For Non-Prod Env
• Other IaaC For Prod Env

• Devops

• CI/CD
• Source Code Management
• PipeLine
• Monitoring

• Dashboard
• Metrics

• Security

• Pod Service Account
• Pod Security Policy

• Auth

• Authentication
• Authenrization
• Multi-Tenancy

• Application Architecture
• Microservice
• Servcie Mesh

• Training

• Knowledge

• Kubernets
• ETCD
• Containter
• Networking

• Orgnization & People

Architecture

Architecture Diagram

Non-Prod Environment

Prod Environment

Cluster Networking

Kubernetes supports for third-party netwroking the cluster via CNI plugin. for more infromation, please see Cluster Networking.

According to this chinese blog, Principles and Solutions of Kubernetes Networking from Yourongyun the three top of CNI providers for kubernetes cluster based on VMs are Project Calico, Flannel, Weave Net.

And also, here is comparison of variable networking solutions, https://github.com/xelatex/homepage/blob/master/source/_posts/Battlefield-Calico-Flannel-Weave-and-Docker-Overlay-Network.md

If just only considering performance, Project Calico should be perferred.

Container Repository

Two options as follows,

Option 1, Private Repository, such as Harbor.

Option 2, Repositories from Cloud Providers, such as AWS ECR.

Setup

We can refer to Picking the Right Solution in Kubernetes offical document to select a setup solution. Considering our goal of building a kuberletes platform based on VMs in our on-premises data center.

Minikube For Dev Env

Minikube can be engaed to create a kubernets development in a local local single machine.

For more information about Minikube, please read Running Kubernetes Locally via Minikube.

Kubeadm For Non-Prod Env

Using Kubeadm, we can build a kubernetes cluster for non-prod environment, which runs master key components as containers.

For more Kubeadm information, please see Using kubeadm to Create a Cluster.

Other IaaC For Prod Env

For creating a kubernetes cluster on VMs for Prod Env, We need to a infrastructure automation tool, such as Ansible, to have this done.

Here is a reference of Creating Kubernets Cluster via Ansible

Devops

CI/CD

Source Code Management

For small and agile web projects, such SaaS applications, considering adopting GitHub Flow. For more information, please refer to GitHub Flow.

For desk or client applications, such PC desk application, ISO/Android App, or being different time windows for delivery and release of applications, considering GitLab Flow. For more information please refer to Gitlab Flow.

PipeLine

Monitoring

Dashboard

Using Kubernetes Dashboard as Web-based UI for Kubernets clusters to manage the cluster itself along with its attendant resources.

For more information, please see Web-UI(Dashboard).

Metrics

Here is official suggested solution as follows. For more informaton, please see Tools for Monitoring Compute, Storage, and Network Resources

Grafana + Heapster / Prometheus + cAdvisor + InfluxDB

Heapster as a metircs aggregator and processor

InfluxDB time series database for storage

Grafana as a dashboarding and alerting solution

cAdvisor has been built in Kubelet, which collects host metrics like CPU, disk space, and memory utilization, in addition to container metrics.

And also, here is a practical example, How to Utilize the “Heapster + InfluxDB + Grafana” Stack in Kubernetes for Monitoring Pods.

Logging

ELK

APM

zipkin

pinpoint

Security

Pod Service Account

For more information, please see the User Guide to Service Accounts.

Pod Security Policy

For more information, please see Pod Security Policies

Auth

Authentication

For more information, please see Authenticating

Support SSO integration, such as SAML,AD OpenID, Auth2?

Authenrization

For more information, please see https://kubernetes.io/docs/admin/authorization/

ABAC/RBAC

Multi-Tenancy

Hypernetes

For more information, please see Hypernetes: Bringing Security and Multi-tenancy to Kubernetes

Application Architecture

Microservice

Here is a chinese blog about how to select open source tools for building a micorservice.

https://mp.weixin.qq.com/s/bsuveX-E6E2fKZ24mj03nQ

Servcie Mesh

Linkerd

Envoy

Istio

Training

Knowledge

Kubernets

Document

Kubernetes Handbook (Chinese Version)

ETCD

For more information, please see ETCD Document.

Containter

Docker

For moe information, please see https://docs.docker.com/.

CRI-O

For more information, please see http://cri-o.io/.

OCI

OCI is a container specification named Open Container Initiative, consisting of OCI Runtime Specification and OCI Image Format

Networking

ip/route/iptables/ipvs etc

Container Netwroking

OpenVswitch

CNI - Calico/Flannel

CNM bridge/host/none/(overlay)plugin

Orgnization & People