使用 opendistro for elasticsearch 做为graylog的后端存储

graylog 是一个很不错的日志分析、收集、报警平台，包好了丰富的插件，同时内部的架构设计很不错
input 组件很多，使用stream、pipeline可以方便的进行数据处理，可以同时3.0 对于sidcar 的支持更好了，内部强大的
dashboard 以及查询能力，可以方便的进行常见系统的性能分析。
以下测试下opendistro for elasticsearch 与graylog 的集成，同时测试下功能的兼容性

环境准备

• docker-compose 文件

version: '3'
services:
  mongodb:
    image: mongo:3
    networks:
      - odfe-net
  elasticsearch:
    image: amazon/opendistro-for-elasticsearch:0.8.0
    container_name: elasticsearch
    environment:
      - opendistro_security.ssl.http.enabled=false
      - cluster.name=odfe-cluster
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  odfe-node2:
    image: amazon/opendistro-for-elasticsearch:0.8.0
    container_name: odfe-node2
    environment:
      - opendistro_security.ssl.http.enabled=false
      - cluster.name=odfe-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.zen.ping.unicast.hosts=elasticsearch
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data2:/usr/share/elasticsearch/data
    networks:
      - odfe-net
  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:0.8.0
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
    networks:
      - odfe-net
  graylog:
    image: graylog/graylog:3.0
    environment:
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
      - GRAYLOG_ELASTICSEARCH_HOSTS=http://admin:admin@elasticsearch:9200 #  连接方式使用basic auth
    links:
      - mongodb:mongo
    networks:
      - odfe-net
    depends_on:
      - mongodb
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
volumes:
  odfe-data1:
  odfe-data2:

networks:
  odfe-net:

• 启动

docker-compose up -d

测试&&查询

• 登录

open http://localhost:9000
账户  admin admin 

• 添加一个input

  使用场景的GELF HTTP input

graylog 3.0 一些新的input 组件

• push 数据

curl -X POST -H 'Content-Type: application/json' -d '{ "version": "1.1", "host": "dalongdemo.org", "short_message": "A short message app demo", "level": 5, "_some_info": "foo" }' 'http://localhost:12201/gelf'

• 界面

• 一个简单的dashboard

• sql 查询

  graylog 默认创建的索引是以grraylog 开头的,以下为使用sql 查询

GET _opendistro/_sql
{
  "query": "select * from graylog_0"
}

效果

说明

通过简单的测试，我们发现兼容还是不错的，我们可以互相利用，搞一个灵活便捷的日志监控系统

参考资料

https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/
http://docs.graylog.org/en/3.0/pages/sending_data.html#gelf-via-http
http://docs.graylog.org/en/3.0/pages/sidecar.html
https://github.com/rongfengliang/opendistro-graylog-docker-compose