kubernetes部署Fluentd+Elasticsearch+kibana 日志收集系统

一、介绍

1. Fluentd 是一个开源收集事件和日志系统,用与各node节点日志数据的收集、处理等等。详细介绍移步-->官方地址：http://fluentd.org/

2. Elasticsearch 是一个开源的,基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。详细介绍移步-->官方地址：http://www.elasticsearch.org/overview/

3. Kibana 开源的用于数据可视化的web ui工具，可使用它对日志进行高效的搜索、可视化、分析等各种操作。详细介绍移步-->官方地址http://www.elasticsearch.org/overview/kibana/

二、流程

每个node节点上面的fluentd监控并收集该节点上面的系统日志，并将处理过后的日志信息发送给Elasticsearch，Elasticsearch汇总各个node节点的日志信息，最后结合Kibana 实现web ui界面的数据展示。

三、安装实现

1.确保k8s集群正常工作(当然这是必须的....)

2.fluentd.yaml文件编写，这里要实现每个节点都能有fluentd跑起来，只需要将kind设置为DaemonSet即可。

 apiVersion: extensions/v1beta1
 kind: DaemonSet
 metadata:
   name: fluentd-elasticsearch
   namespace: kube-system
   labels:
     k8s-app: fluentd-logging
 spec:
   template:
     metadata:
       labels:
         name: fluentd-elasticsearch
     spec:
       containers:
       - name: fluentd-elasticsearch
         image: gcr.io/google-containers/fluentd-elasticsearch:.
         resources:
           limits:
             memory: 200Mi
           requests:
             cpu: 100m
             memory: 200Mi
         volumeMounts:
         - name: varlog
           mountPath: /var/log
         - name: varlibdockercontainers
           mountPath: /var/lib/docker/containers
           readOnly: true
       terminationGracePeriodSeconds:
       volumes:
       - name: varlog
         hostPath:
           path: /var/log
       - name: varlibdockercontainers
         hostPath:
           path: /var/lib/docker/containers

3.elasticsearch-rc.yaml&elasticsearch-svc.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: elasticsearch-logging-v1
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  replicas:
  selector:
    k8s-app: elasticsearch-logging
    version: v1
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - image: gcr.io/google-containers/elasticsearch:v2..
        name: elasticsearch-logging
        resources:
          # need more cpu upon initialization, therefore burstable class
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort:
          name: db
          protocol: TCP
        - containerPort:
          name: transport
          protocol: TCP
        volumeMounts:
        - name: es-persistent-storage
          mountPath: /data
      volumes:
      - name: es-persistent-storage
        emptyDir: {}

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "Elasticsearch"
spec:
  ports:
  - port:
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch-logging

4.kibana-rc.yaml&kibana-svc.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
spec:
  replicas:
  selector:
    matchLabels:
      k8s-app: kibana-logging
  template:
    metadata:
      labels:
        k8s-app: kibana-logging
    spec:
      containers:
      - name: kibana-logging
        image: gcr.io/google-containers/kibana:v4..
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
          requests:
            cpu: 100m
        env:
          - name: "ELASTICSEARCH_URL"
            value: "http://elasticsearch-logging:9200"
          - name: "KIBANA_BASE_URL"
            value: "/api/v1/proxy/namespaces/kube-system/services/kibana-logging"
        ports:
        - containerPort:
          name: ui
          protocol: TCP

apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "Kibana"
spec:
  ports:
  - port:
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging

5.kubectl create -f ****** ，这里就自己发挥吧。

镜像推荐使用最新的iamge,多去github/kubernetes看看 里面有详细的说明