k8s之operator

背景

数字经济的兴起推动了云计算、物联网、大数据行业的快速蓬勃发展，对数据中心提出了更高的要求，同时，用户对于数据库运维自动化的需求越来越高，数据库即服务的需求越来越强烈。

随着k8s的普及以及云原生架构的兴起，越来越多的人希望把数据库这类有状态署服务也通过k8s进行编排。也就是对数据库进行容器化部署。

通过statefulset编排有状态应用具有很高的要求以及复杂性，很多官方或者一些社区(https://operatorhub.io/)对主流的的有状态应用将哪些复杂的、特有的操作步聚进行代码化，完成应用编排专用的生命周期管理。

Operator 是增强型的控制器(Controller)它扩展了KubernetesAPI的功能，并基于该扩展管理复杂应用程序。

K8s Operator

K8s Operator是一种用于特定应用的控制器，可扩展k8s API的功能，来代表k8s用户创建、配置和管理复杂应用的实例，它基于基本的k8s资源和控制器概念构建，但又涵盖了特定领域或应用的知识，用于实现所管理的应用生命周期的自动化。

常见的有:etcd-operator、prometheus-operator、mysql-operator等

CRD: 用户自定义资源

CR: 实现CRD的具体实例,依据CRD创建具体的对象。

Webhook: k8s的一种http回调，默认注册到kube-apiserver上，与apiserver进行绑定。主要作用资源的修改和验证。

Controller: CRD实现业务的核心组件，它控制当前CRD运行管理动作。持续监听集群状态变化，把跟自己有关的对象事件比如:create、delete、update放到工作队列中，并且会持续把当前资源状态变成用户定义的spec期望状态

无状态和有状态应用对比

无状态服务	有状态服务
数据方面	不会在本地存储持久化数据多个实例共享相同的持久化数据
结果方面	多个服务实例对同一用户请求的响应结果完全一致
关系方面	实例之间无依赖关系
影响方面	动态启停pod不会对其它的pod产生影响
示例方面	Nginx、tomcat、web应用
资源方面	Deployment、replicaSet等
创建方式	Deployment管理
缩容方式	随机

ELK operator

Elastic Cloud on Kubernetes (ECK)，是一款基于 Kubernetes Operator 模式的新型编排产品。

ECK 使用 Kubernetes Operator 模式构建而成，需要安装在您的 Kubernetes 集群内，其功能绝不仅限于简化 Kubernetes 上 Elasticsearch 和 Kibana 的部署工作这一项任务。ECK 专注于简化所有后期运行工作，例如：

• 管理和监测多个集群
• 轻松升级至新的堆栈版本
• 扩大或缩小集群容量
• 更改集群配置
• 动态调整本地存储的规模（包括 Elastic Local Volume（一款本地存储驱动器））
• 备份安排
  
  但 ECK 的功能却绝不仅限于 Kubernetes Operator。ECK 不仅能自动完成所有运行和集群管理任务，还专注于简化在 Kubernetes 上使用 Elasticsearch的完整体验。ECK 的愿景是为 Kubernetes 上的 Elastic 产品和解决方案提供 SaaS 般的体验。

安装ECK

https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html

• Install custom resource definitions(CRD)
  
  自定义资源类型CRD

root@k8s-master01:~# kubectl create -f https://download.elastic.co/downloads/eck/2.12.1/crds.yamlcustomresourcedefinition.apiextensions.k8s.io/agents.agent.k8s.elastic.co createdcustomresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticmapsservers.maps.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearchautoscalers.autoscaling.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/logstashes.logstash.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/stackconfigpolicies.stackconfigpolicy.k8s.elastic.co created

root@k8s-master01:~# kubectl api-resources|grep elastic
agents                            agent        agent.k8s.elastic.co/v1alpha1               true         Agent
apmservers                        apm          apm.k8s.elastic.co/v1                       true         ApmServer
elasticsearchautoscalers          esa          autoscaling.k8s.elastic.co/v1alpha1         true         ElasticsearchAutoscaler
beats                             beat         beat.k8s.elastic.co/v1beta1                 true         Beat
elasticsearches                   es           elasticsearch.k8s.elastic.co/v1             true         Elasticsearch
enterprisesearches                ent          enterprisesearch.k8s.elastic.co/v1          true         EnterpriseSearch
kibanas                           kb           kibana.k8s.elastic.co/v1                    true         Kibana
logstashes                        ls           logstash.k8s.elastic.co/v1alpha1            true         Logstash
elasticmapsservers                ems          maps.k8s.elastic.co/v1alpha1                true         ElasticMapsServer
stackconfigpolicies               scp          stackconfigpolicy.k8s.elastic.co/v1alpha1   true         StackConfigPolicy

• Install the operator with its RBAC rules

root@k8s-master01:~# kubectl apply -f https://download.elastic.co/downloads/eck/2.12.1/operator.yaml
namespace/elastic-system created
serviceaccount/elastic-operator created
secret/elastic-webhook-server-cert created
configmap/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator-view created
clusterrole.rbac.authorization.k8s.io/elastic-operator-edit created
clusterrolebinding.rbac.authorization.k8s.io/elastic-operator created
service/elastic-webhook-server created
statefulset.apps/elastic-operator created
validatingwebhookconfiguration.admissionregistration.k8s.io/elastic-webhook.k8s.elastic.co created

root@k8s-master01:~# kubectl get pod -n elastic-system
NAME                 READY   STATUS    RESTARTS   AGE
elastic-operator-0   1/1     Running   0          67s

root@k8s-master01:~# kubectl explain elasticsearch
GROUP:      elasticsearch.k8s.elastic.co
KIND:       Elasticsearch
VERSION:    v1

DESCRIPTION:
    Elasticsearch represents an Elasticsearch resource in a Kubernetes cluster.

FIELDS:
  apiVersion    <string>
    APIVersion defines the versioned schema of this representation of an object.
    Servers should convert recognized schemas to the latest internal value, and
    may reject unrecognized values. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

  kind  <string>
    Kind is a string value representing the REST resource this object
    represents. Servers may infer this from the endpoint the client submits
    requests to. Cannot be updated. In CamelCase. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

  metadata      <ObjectMeta>
    Standard object's metadata. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

  spec  <Object>
    ElasticsearchSpec holds the specification of an Elasticsearch cluster.

  status        <Object>
    ElasticsearchStatus represents the observed state of Elasticsearch.

root@k8s-master01:~# cat elasticsearch-myes-cluster.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: myes
  namespace: elastic-system
spec:
  version: 8.11.3
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
       #storageClassName: nfs-csi
        storageClassName: openebs-hostpath

root@k8s-master01:~# kubectl apply -f elasticsearch-myes-cluster.yaml
elasticsearch.elasticsearch.k8s.elastic.co/myes created
root@k8s-master01:~# kubectl get elasticsearch -n elastic-system
NAME   HEALTH    NODES   VERSION   PHASE             AGE
myes   unknown           8.11.3    ApplyingChanges   18s

root@k8s-master01:~# kubectl get pod -n elastic-system
NAME                 READY   STATUS    RESTARTS   AGE
elastic-operator-0   1/1     Running   0          144m
myes-es-default-0    1/1     Running   0          111m
myes-es-default-1    1/1     Running   0          111m
myes-es-default-2    1/1     Running   0          111m

root@k8s-master01:~# kubectl get svc -n elastic-system
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
elastic-webhook-server   ClusterIP   10.107.151.34   <none>        443/TCP    144m
myes-es-default          ClusterIP   None            <none>        9200/TCP   111m
myes-es-http             ClusterIP   10.99.67.165    <none>        9200/TCP   111m
myes-es-internal-http    ClusterIP   10.111.49.216   <none>        9200/TCP   111m
myes-es-transport        ClusterIP   None            <none>        9300/TCP   111m

root@k8s-master01:~# kubectl get secret -n elastic-system
NAME                                 TYPE     DATA   AGE
elastic-webhook-server-cert          Opaque   2      145m
myes-es-default-es-config            Opaque   1      111m
myes-es-default-es-transport-certs   Opaque   7      111m
myes-es-elastic-user                 Opaque   1      111m
myes-es-file-settings                Opaque   1      111m
myes-es-http-ca-internal             Opaque   2      111m
myes-es-http-certs-internal          Opaque   3      111m
myes-es-http-certs-public            Opaque   2      111m
myes-es-internal-users               Opaque   4      111m
myes-es-remote-ca                    Opaque   1      111m
myes-es-transport-ca-internal        Opaque   2      111m
myes-es-transport-certs-public       Opaque   1      111m
myes-es-xpack-file-realm             Opaque   4      111m

先获取到访问ElasticSearch的密码，该密码由部署过程自动生成，并保存在了相关名称空间下的Secrets中，该Secrets对象以集群名称为前缀，以“-es-elastic-user”为后缀。下面的命令将获取到的密码保存在名为PASSWORD的变量中。

root@k8s-master01:~# PASSWORD=$(kubectl get secret myes-es-elastic-user -n elastic-system -o go-template='{{.data.elastic | base64decode}}')
root@k8s-master01:~# echo $PASSWORD
K4281d4X9wX7XNw7MfD0u0dR

在集群上通过类似如下命令访问部署好的ElasticSearch集群。

kubectl run client-$RANDOM --image ikubernetes/admin-box:v1.2 -it --rm --restart=Never --command -- /bin/bash
curl -u "elastic:$PASSWORD" -k https://myes-es-http.elastic-system:9200
{
  "name" : "myes-es-default-1",
  "cluster_name" : "myes",
  "cluster_uuid" : "gCl81n1HS1q2W-dreAo31Q",
  "version" : {
    "number" : "8.11.3",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "64cf052f3b56b1fd4449f5454cb88aca7e739d9a",
    "build_date" : "2023-12-08T11:33:53.634979452Z",
    "build_snapshot" : false,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}