ELK集群模式部署

架构拓扑图为：

准备工作：

下载资源包：

Elasticsearch： wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz  # 这一步用docker启动，可以不用下载。

Kibana: wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz

Logstash:wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz

Filebeat:wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz

安装：

根据拓扑图把对应的服务安装在对应的服务器。

tar xvf kibana-6.2.3-linux-x86_64.tar.gz -C /usr/local/

tar xvf logstash-6.2.4.tar.gz -C /usr/local/

tar -xvf filebeat-6.2.4-linux-x86_64.tar.gz -C /usr/local/

部署启动:

Elasticsearch-master(192.168.11.1)：

需自行提前安装好docker 和 docker-compose。

vim docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
    container_name: elasticsearch
    environment:
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "network.publish_host=192.168.11.1"
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "192.168.11.1:9200:9200"
      - "192.168.11.1:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d

Elasticsearch-node2.data(192.168.11.2)：

需自行提前安装好docker 和 docker-compose。

vim docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
    container_name: elasticsearch
    environment:
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "network.publish_host=192.168.11.2"
      - "discovery.zen.ping.unicast.hosts=192.168.11.1"
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "192.168.11.2:9200:9200"
      - "192.168.11.2:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d

Elasticsearch-node3.data(192.168.11.3)：

需自行提前安装好docker 和 docker-compose。

vim docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
    container_name: elasticsearch
    environment:
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "network.publish_host=192.168.11.3"
      - "discovery.zen.ping.unicast.hosts=192.168.11.1"
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "192.168.11.3:9200:9200"
      - "192.168.11.3:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d

查看集群节点：curl -XGET 'http://192.168.11.1:9200/_cat/nodes?pretty'

 

kibana(192.168.11.4)：

cd /usr/local/kibana-6.2.3-linux-x86_64

cp ./config/kibana.yml ./config/kibana.yml.bak

echo '' > ./config/kibana.yml

vim ./config/kibana.yml

server.port: 5601
server.host: "192.168.11.4"
elasticsearch.url: "http://192.168.11.1:9200"

启动：./bin/kibana

扩展包（可装可不装）：

kibana 报警扩展:

安装 ./bin/kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.2.3-2/sentinl-v6.2.3.zip

重启 kibana

logstash(192.168.11.5)

cd /usr/local/logstash-6.2.4

vim nginx.conf

input {
beats {
port => 5066
host => "192.168.11.5"
codec => "json"
}
}
filter {

mutate {

gsub => ["message", "\\x", "\\\x"]

}

json {

source => "message"

}

}
output {
    elasticsearch {
        action => "index"
        hosts => "192.168.11.1:9200"
        index => "nginx-json-%{+YYYY.MM.dd}"
   }
}

mkdir -p /data/logstash-data/nginx

启动：./bin/logstash -f ./nginx.conf --path.data=/data/logstash-data/nginx &

扩展

grok调试地址：http://grokdebug.herokuapp.com/

如需调试请修改output成以下，方便debug

output {
        stdout {
        codec    => rubydebug
    }
}

filebeat(192.168.11.6)

cd /usr/local/filebeat-6.2.4-linux-x86_64

vim nginx.yml

filebeat.prospectors:
- input_type: log
  document_type: jsonlog
  paths:
    - /var/log/nginx/access.log
output:
  logstash:
    hosts: ["192.168.11.5:5066"]

启动：./filebeat -c ./nginx.yml &

nginx 日志的输出格式为：

log_format json '{"@timestamp":"$time_iso8601",'
               '"remote_addr":"$remote_addr",'
               '"request":"$request",'
               '"status":$status,'
               '"body_bytes":$body_bytes_sent,'
               '"user_agent":"$http_user_agent",'
               '"resp_time":"$upstream_response_time",'
               '"req_time":$request_time,'
               '"host":"$host",'
               '"@version":"1",'
               '"http_x_forwarded_for":"$http_x_forwarded_for",'
               '"upstream_addr":"$upstream_addr",'
               '"req_body":"$request_body"}';

参考地址：

https://www.elastic.co/guide/index.html

http://docs.docker.com/compose/

https://www.elastic.co/guide/en/logstash/current/index.html

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docker.html

https://www.elastic.co/guide/en/kibana/current/install.html