ELK应用之一:ELK平台搭建部署

一、获取安装包:

EKL官网:https://www.elastic.co/products

二、安装Elasticsearch

2.1、解压安装包，在安装目录创建data和logs目录，并在配置文件中指定路径:

elasticsearch.yml:

path.data: /data/local/elasticsearch/data
path.logs: /data/local/elasticsearch/logs

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

network.host: 0.0.0.0

http.port: 9200
http.enabled: true

http.cors.enabled: true
http.cors.allow-origin: "*"

elasticsearch默认不允许root用户运行，创建elasticsearch用户，并将elasticsearch安装目录拥有着修改为elasticsearch。

修改几个文件:

2.2、/etc/security/limits.conf:

*               soft    nproc           2048
*               hard    nproc           4096
*               soft    nofile          65536
*               hard    nofile          131072

2.3、/etc/sysctl.conf，添加如下:

vm.max_map_count = 655360

2.4、/etc/security/limits.d/20-nproc.conf:

*          soft    nproc     4096

发生的错误:

ERROR: [1] bootstrap checks failed

禁止bootstrap检测:

修改elasticsearch.yml，添加如下:

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

以上修改后，若还提示错误，需要重启系统。

启动elasticsearch：

切换到elasticsearch用户，在安装目录/bin下直接运行:

./elasticsearch &

可以看到启动日志:

[2017-08-03T15:09:45,481][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [aggs-matrix-stats]
[2017-08-03T15:09:45,481][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [ingest-common]
[2017-08-03T15:09:45,481][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [lang-expression]
[2017-08-03T15:09:45,481][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [lang-groovy]
[2017-08-03T15:09:45,481][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [lang-mustache]
[2017-08-03T15:09:45,482][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [lang-painless]
[2017-08-03T15:09:45,482][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [parent-join]
[2017-08-03T15:09:45,482][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [percolator]
[2017-08-03T15:09:45,482][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [reindex]
[2017-08-03T15:09:45,482][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [transport-netty3]
[2017-08-03T15:09:45,482][INFO ][o.e.p.PluginsService     ] [QaS1DC7] loaded module [transport-netty4]
[2017-08-03T15:09:45,483][INFO ][o.e.p.PluginsService     ] [QaS1DC7] no plugins loaded
[2017-08-03T15:09:47,328][INFO ][o.e.d.DiscoveryModule    ] [QaS1DC7] using discovery type [zen]
[2017-08-03T15:09:47,878][INFO ][o.e.n.Node               ] initialized
[2017-08-03T15:09:47,879][INFO ][o.e.n.Node               ] [QaS1DC7] starting ...
[2017-08-03T15:09:48,051][INFO ][o.e.t.TransportService   ] [QaS1DC7] publish_address {192.168.1.23:9300}, bound_addresses {[::]:9300}
[2017-08-03T15:09:48,061][INFO ][o.e.b.BootstrapChecks    ] [QaS1DC7] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-08-03T15:09:51,113][INFO ][o.e.c.s.ClusterService   ] [QaS1DC7] new_master {QaS1DC7}{QaS1DC7FQy6uyuZv4RCoEw}{9u0bZc88TCKmbcT1tOwdlw}{192.168.1.23}{192.168.1.23:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-08-03T15:09:51,132][INFO ][o.e.h.n.Netty4HttpServerTransport] [QaS1DC7] publish_address {192.168.1.23:9200}, bound_addresses {[::]:9200}
[2017-08-03T15:09:51,132][INFO ][o.e.n.Node               ] [QaS1DC7] started
[2017-08-03T15:09:51,146][INFO ][o.e.g.GatewayService     ] [QaS1DC7] recovered [0] indices into cluster_state

2.5、安装elasticsearch-head：

elasticsearch-head是干啥的？

答:elasticsearch-head是一个可视化的管理elasticsearch集群的工具。

获取软件包:

git clone git://github.com/mobz/elasticsearch-head.git

到elasticsearch-head目录下：

npm install

注：前提需要安装nodjs。

tar zxvf node-v6.10.3.tar.gz
cd node
./configure
make
make install

编辑Gruntfile.js文件，connect修改如下:

connect: {
                        server: {
                                options: {
                                        hostname: '＊',
                                        port: 9100,
                                        base: '.',
                                        keepalive: true
                                }
                        }
                }

启动head:

/elasticsearch-head/node_modules/grunt/bin/grunt server 

打开浏览器，输入IP:9100就可以看到Elasticsearch的状态了：

三、安装Logstash

logstash的安装也比较简单，下载安装包后，直接解压就可以了。

测试logstash:

在解压后的config目录下创建用于测试的配置文件:

#vim test.conf

input {
    stdin {}
}
output {
    stdout {
        codec => rubydebug {}
    }
}

配置文件的说明:

文件中定义了输入和输出两部分，输入为标准输入，输出格式为codec => rubydebug {}

启动logstash:

./bin/logstash -f config/test.conf

-f参数，指定配置文件。

终端中开启交互模式，输入hello后，会被标准输出:

hello
{
    "@timestamp" => 2017-07-14T05:32:04.765Z,
      "@version" => "",
          "host" => "localhost",
       "message" => "hello"
}

配置logstash的输入和输出:

logstash支持多种输入类型

1、从普通日志文件输入:

input {
  file {
    path => "/var/log/messages"
    type => "syslog"
  }

input输入为文件时，可以将所有文件放到列表中:

path => [ "/var/log/messages", "/var/log/*.log" ]

2、从beats库输入:

beats库是一系列采集数据的插件，可以替代logstash，候问介绍。

input {
     beats {
        port => ""     ＃beats库已运行于5044端口
    }
}

3、其他输入：

logstash还支持TCP/IP、Syslog等输入，这里不详细介绍。

四、安装Kibana

kibana的安装同logstash，解压安装包后，直接运行bin下的kibana就可以启动。

配置文件:

config/kibana.yml:

erver.port: 5601     #监听的端口

server.host: "0.0.0.0"   #允许其他远程客户端访问

elasticsearch.url: "http://localhost:9200"    #连接的Elasticsearch地址

启动kibana后，就可以通过页面访问:

Beats库的介绍: