ELK Stack 5.2.2 安装文档

简介：

ELK Stack 安装文档，这次都使用最新版本(5.2.2)、RPM 包的方式搭建 ELK Stack。

下载地址:

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.2.rpm
https://artifacts.elastic.co/downloads/logstash/logstash-5.2.2.rpm
https://artifacts.elastic.co/downloads/kibana/kibana-5.2.2-x86_64.rpm

jre: http://javadl.oracle.com/webapps/download/AutoDL?BundleId=216423
jdk: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

# 如果只是需要 java 环境，那么安装 jre 即可，如还需编译 java 包就需要安装 jdk 了。
# 运行 Elasticsearch jre 即可

一、安装

shell > rpm -e elasticsearch
Stopping elasticsearch service... OK
warning: /etc/sysconfig/elasticsearch saved as /etc/sysconfig/elasticsearch.rpmsave
warning: /etc/init.d/elasticsearch saved as /etc/init.d/elasticsearch.rpmsave
warning: /etc/elasticsearch/elasticsearch.yml saved as /etc/elasticsearch/elasticsearch.yml.rpmsave
Deleting log directory... OK
Deleting plugins directory... OK

shell > rpm -e kibana
Stopping kibana service... OK
warning: /opt/kibana/config/kibana.yml saved as /opt/kibana/config/kibana.yml.rpmsave

# 我之前使用 2.4.1 版本，也是通过 rpm 安装的，需要先卸载。

shell > rm -rf /etc/sysconfig/elasticsearch.rpmsave
shell > rm -rf /etc/init.d/elasticsearch.rpmsave
shell > rm -rf /etc/elasticsearch/elasticsearch.yml.rpmsave
shell > rm -rf /opt/kibana/config/kibana.yml.rpmsave

# 可以看到卸载的时候，这些文件没有被删除，强迫症的我是不允许这些文件存在的。

shell > cd /usr/local/src; rpm -ivh elasticsearch-5.2..rpm logstash-5.2..rpm kibana-5.2.-x86_64.rpm

# 由于是测试一下新版本，所以都装在了一台服务器上。

二、配置

1、Elasticsearch

shell > grep -vP '^#|^$' /etc/elasticsearch/elasticsearch.yml
# 集群名称
cluster.name: elk
# 节点名称
node.name: node-
# 数据路径
path.data: /data/elast/data
# 日志路径
path.logs: /data/elast/logs
# 启动时锁住内存，防止数据被交换到 SWAP
bootstrap.memory_lock: true
# 监听地址
network.host: 0.0.0.0
# 与其余节点通信地址
network.publish_host: 10.127.174.217
# 开启 HTTP 协议
http.port:
# 解决启动报错
bootstrap.system_call_filter: false

shell > mkdir -p /data/elast/{data,logs}
shell > chown -R elasticsearch.elasticsearch /data/elast

# 创建数据、日志目录

2、Logstash

shell > vim /etc/logstash/conf.d/for_elk.conf
# 输入插件，这里从 redis 中读取数据
input {
    redis {
        data_type => "list"
        key => "for_elk"
        host => "10.217.79.61"
        port =>
        threads =>
    }
}
# 过滤插件，按需切割日志、加减字段等
filter {
    mutate {
        split => ["message", "|"]
        add_field => {"clientip" => "%{message[0]}"}
        add_field => {"localtime" => "%{message[1]}"}
        add_field => {"api" => "%{message[2]}"}
        add_field => {"request_all" => "%{message[3]}"}
        add_field => {"http_code" => "%{message[4]}"}
        add_field => {"request_body" => "%{message[6]}"}
        add_field => {"request_time" => "%{message[11]}"}
    }

    date {
        match => ["localtime", "dd/MMM/yyyy:HH:mm:ss Z"]
    }

    geoip {
        source => "clientip"
        fields => ["city_name", "latitude", "longitude"]
    }

    kv {
        source => "request_body"
        field_split => "&"
        remove_field => "host"
        remove_field => "path"
        remove_field => "message"
        remove_field => "request_all"
        remove_field => "request_body"
    }

    mutate {
        convert => [
            "id", "integer",
            "cid", "integer",
            "tid", "integer",
            "vid", "integer",
            "version", "float",
            "http_code", "integer",
            "request_time", "float"
        ]
    }
}
# 输出插件
output {
    elasticsearch {
        hosts => ["10.127.174.217:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
        template_overwrite => true
    }
    # stdout {
    #     codec => rubydebug
    # }
}

# 可以测试能否从 redis 拿到数据，然后在做 filter，最后测试能否写入 elasticsearch

3、Kibana

shell > /etc/kibana/kibana.yml

# Kibana 其实不用修改，暂时采用默认配置即可

三、启动

1、Elasticsearch

shell > /etc/init.d/elasticsearch start

2、Logstash

shell > /usr/share/logstash/bin/logstash --path.settings /etc/logstash > /dev/null &

3、Kibana

shell > /etc/init.d/kibana start

四、访问

# http://x.x.x.x:5601 即可，根据 index 建立索引，嗯 确实比 K4 漂亮

五、插件安装

1、Elasticsearch head (从 5.0 起，该插件以一个单独的服务运行)

shell > cd /usr/local

shell > git clone git://github.com/mobz/elasticsearch-head.git

shell > cd elasticsearch-head

shell > npm install

shell > vim Gruntfile.js

                connect: {
                        server: {
                                options: {
                                        hostname: '0.0.0.0',
                                        port: ,
                                        base: '.',
                                        keepalive: true
                                }
                        }
                }

# 默认只监听 127.0.0.1，所以要加上 hostname: '0.0.0.0'

shell > ./node_modules/grunt/bin/grunt server > /dev/null &

shell > vim /etc/elasticsearch/elasticsearch.yml

# head plugin
http.cors.enabled: true
http.cors.allow-origin: "*"

# elasticsearch 5.x 需要设置该参数，否则无法 head 无法连接 es
# 你可能注意到 es 集群状态为 yellow，不要慌...
# 那是因为副本不可用，因为只有一个 es 节点，而副本不能在本机，不碍事 ！

2、IK Analysis for Elasticsearch

shell > wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.5.0/binaries/apache-maven-3.5.0-bin.tar.gz
shell > tar zxf apache-maven-3.5.-bin.tar.gz -C /usr/local
shell > echo -e '\nexport JAVA_HOME=/usr/java/default' >> /etc/profile && source /etc/profile

shell > wget https://github.com/medcl/elasticsearch-analysis-ik/archive/v5.2.2.zip
shell > unzip v5.2.2.zip
shell > cd elasticsearch-analysis-ik-5.2.
shell > /usr/local/apache-maven-3.5./bin/mvn package
shell > unzip target/releases/elasticsearch-analysis-ik-5.2..zip -d /usr/share/elasticsearch/plugins/ik
shell > /usr/share/elasticsearch/bin/elasticsearch-plugin list
ik
shell > /etc/init.d/elasticsearch restart

附件：

1、Elasticsearch 启动报错

> bootstrap.memory_lock: true 参数导致

memory locking requested for elasticsearch process but memory is not locked

解决方法:

shell > vim /etc/security/limits.conf

# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

> CentOS 6.x 不支持 CONFIG_SECCOMP 导致

[--01T12::,][WARN ][o.e.b.JNANatives         ] unable to install syscall filter:
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
        at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:) ~[elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:) ~[elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.cli.Command.main(Command.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:) [elasticsearch-5.2..jar:5.2.]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:) [elasticsearch-5.2..jar:5.2.]

bootstrap checks failed
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

解决方法：

shell > vim /etc/elasticsearch/elasticsearch.yml

bootstrap.system_call_filter: falses

> /etc/security/limits.d/90-nproc.conf 默认参数过低导致启动失败

[--06T14::,][ERROR][o.e.b.Bootstrap          ] [node01] node validation exception
bootstrap checks failed
max number of threads [] for user [elasticsearch] is too low, increase to at least []

解决方法：

shell > vim /etc/security/limits.d/-nproc.conf

*          soft    nproc
root       soft    nproc     unlimited

# 将原  改为