filebeat+logstash+Elasticsearch+kibana实现nginx日志图形化展示

 

by:授客  QQ:1033553122

 

测试环境

Win7 64

CentOS-7-x86_64-DVD-1503-01.iso(kibana安装环境)

CentOS 6.5-x86_64(其它软件安装环境)

nginx-1.10.0

filebeat-5.5.2-linux-x86_64.tar.gz

下载地址:

https://pan.baidu.com/s/1dEBkIuH

https://www.elastic.co/downloads/beats/filebeat#ga-release

https://www.elastic.co/start

kibana-5.5.0-linux-x86_64.tar.gz

下载地址:

https://pan.baidu.com/s/1dEBkIuH

https://www.elastic.co/start

logstash-5.5.2.tar.gz

下载地址:

https://pan.baidu.com/s/1dEBkIuH

https://www.elastic.co/downloads/logstash

elasticsearch-5.5.2

下载地址:

https://pan.baidu.com/s/1dEBkIuH

https://www.elastic.co/downloads/elasticsearch#preview-release

安装Nginx

Nginx日志配置

http {

include       mime.types;

default_type  application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" $status $request_time $upstream_response_time $request_length $bytes_sent $body_bytes_sent $gzip_ratio $connection_requests "$http_referer" "$http_user_agent" "$http_x_forwarded_for"';

access_log  logs/access.log  main;

运行nginx

安装java

参考文章:

http://blog.sina.com.cn/s/blog_13cc013b50102w01m.html#_Toc438402186

[root@bogon ~]# java -version

java version "1.8.0_65"

64-Bit Server VM (build 25.65-b01, mixed mode)

注意:logstash要求Java 8,不支持java9

https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

安装logstash

# tar -xzvf logstash-5.5.2.tar.gz

# ls

logstash-5.5.2  logstash-5.5.2.tar.gz

# mkdir -p /usr/local/logstash

# mv logstash-5.5.2 /usr/local/logstash/

 

配置logstash

# vim /usr/local/logstash/logstash-5.5.2/logstash.conf

input { stdin {} }

output {

elasticsearch { hosts => ["192.168.1.101:9200"] }

stdout { codec => rubydebug }

}

说明:

input { stdin {} }  表示从标准输入中接收数据

192.168.1.101:9200  分别代表Elasticsearch搜索访问ip和监听端口

stdout { codec => rubydebug }  表示输出到控制台

参考链接:

https://www.elastic.co/guide/en/logstash/current/config-examples.html

运行logstash

# cd /usr/local/logstash/logstash-5.5.2/

# bin/logstash -f logstash.conf

……(略)

The stdin plugin is now waiting for input:

[2017-07-14T03:40:50,373][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

hello world

{

"@timestamp" => 2017-07-13T19:59:53.848Z,

"@version" => "1",

"host" => "0.0.0.0",

"message" => "hello world"

}

说明:启动后,输入上述带背景色内容 hello world,待控制台输出带黄色背景色内容后,在Elasticsearch中执行搜索,如下

GET /logstash-2017.07.13/_search

如上图,能搜索到输入数据,说明成功了

停止运行logstash

按CTRL + D键

参考链接:

https://www.elastic.co/guide/en/logstash/current/first-event.html

安装Elasticsearch

安装kibana

# mkdir -p /usr/local/kibana

# tar -xvzf kibana-5.5.0-linux-x86_64.tar.gz

# mv kibana-5.5.0-linux-x86_64 /usr/local/kibana/

参考链接:

https://www.elastic.co/guide/en/kibana/current/targz.html

配置kibana

# cd /usr/local/kibana/kibana-5.5.0-linux-x86_64/config/

# vim kibana.yml

server.host: "192.168.1.104"

elasticsearch.url: "http://192.168.1.101:9200"

参考链接:

https://www.elastic.co/guide/en/kibana/current/settings.html

运行kibana

# cd /usr/local/kibana/kibana-5.5.0-linux-x86_64/

# ./bin/kibana

log   [23:51:04.051] [info][status][plugin:kibana@5.5.0] Status changed from uninitialized to green - Ready

log   [23:51:04.510] [info][status][plugin:elasticsearch@5.5.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log   [23:51:04.594] [info][status][plugin:console@5.5.0] Status changed from uninitialized to green - Ready

log   [23:51:04.617] [warning] You're running Kibana 5.5.0 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v5.5.2 @ 192.168.1.101:9200 (192.168.1.101)

log   [23:51:04.674] [info][status][plugin:metrics@5.5.0] Status changed from uninitialized to green - Ready

log   [23:51:04.706] [info][status][plugin:elasticsearch@5.5.0] Status changed from yellow to green - Kibana index ready

log   [23:51:06.992] [info][status][plugin:timelion@5.5.0] Status changed from uninitialized to green - Ready

log   [23:51:07.032] [info][listening] Server running at http://192.168.1.104:5601

log   [23:51:07.037] [info][status][ui settings] Status changed from uninitialized to green - Ready

验证

浏览器中访问:http://192.168.1.104:5601/status

结果发现打不开

解决方法:停止防火墙

# systemctl stop firewalld.service

再次访问

参考链接:

https://www.elastic.co/guide/en/kibana/current/access.html

配置索引模式(index pattern

要使用Kibana至少需要配置一个索引模式(index pattern)。索引模式用于确认执行搜索和分析的Elasticsearch索引。

窗体顶端

Index name or pattern

配置索引名称,或者索引模式。索引模式允许使用通配符 * 。比如 logstash-*

Time Filter field name

设置时间过滤器,方便在Discover页面中按时间筛选数据

窗体底端

Management -> Index Patterns -> Create Index Pattern,重新设置

参考链接:

https://www.elastic.co/guide/en/kibana/current/connect-to-elasticsearch.html

安装filebeat

# tar -xvzf filebeat-5.5.2-linux-x86_64.tar.gz

# mkdir -p /usr/local/filebeat

# mv filebeat-5.5.2-linux-x86_64 /usr/local/filebeat/

配置

# vim /usr/local/filebeat/filebeat-5.5.2-linux-x86_64/filebeat.yml

配置日志文件路径

如上,可以指定具体的文件名,

- /usr/local/ngnix/logs/access.log

- /usr/local/ngnix/logs/error.log

也可以使用通配符,表示/usr/local/ngnix/logs/目录下,所有.log文件

- /usr/local/ngnix/logs/*.log

配置logstash输出

 

注意:hosts: 后面必须接一个空格,否则会报错

测试配置是否正确

# cd /usr/local/filebeat/filebeat-5.5.2-linux-x86_64/

# ./filebeat -configtest -e

2017/08/17 23:55:32.651228 beat.go:285: INFO Home path: [/usr/local/filebeat/filebeat-5.5.2-linux-x86_64] Config path: [/usr/local/filebeat/filebeat-5.5.2-linux-x86_64] Data path: [/usr/local/filebeat/filebeat-5.5.2-linux-x86_64/data] Logs path: [/usr/local/filebeat/filebeat-5.5.2-linux-x86_64/logs]

2017/08/17 23:55:32.651335 beat.go:186: INFO Setup Beat: filebeat; Version: 5.5.2

2017/08/17 23:55:32.651564 logstash.go:90: INFO Max Retries set to: 3

2017/08/17 23:55:32.652006 outputs.go:108: INFO Activated logstash as output plugin.

2017/08/17 23:55:32.652250 metrics.go:23: INFO Metrics logging every 30s

2017/08/17 23:55:32.662026 publish.go:295: INFO Publisher name: bogon

2017/08/17 23:55:32.698907 async.go:63: INFO Flush Interval set to: 1s

2017/08/17 23:55:32.699214 async.go:64: INFO Max Bulk Size set to: 2048

Config OK

运行filebeat

# ./filebeat -e -c filebeat.yml -d "publish"

参考链接:

https://www.elastic.co/guide/en/beats/filebeat/5.5/filebeat-starting.html

https://www.elastic.co/guide/en/beats/filebeat/5.5/config-filebeat-logstash.html

修改logstash配置

[root@bogon logstash-5.5.2]# vim logstash.conf

input {

beats {

port => "9400"

}

}

filter{

grok {

match => {"message" => "%{IP:remote_addr} - %{USER:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} %{DATA:http_version}" %{NUMBER:status:int} %{NUMBER:request_time:float} %{NUMBER:upstream_response_time:float} %{NUMBER:request_length:int} %{NUMBER:bytes_sent:int} %{NUMBER:body_bytes_sent:int} %{DATA:gzip_ratio:float} %{NUMBER:connection_requests:int} "%{DATA:http_referer}" %{QUOTEDSTRING:http_user_agent} %{DATA:http_x_forwarded_for}"}

}

}

output {

elasticsearch { hosts => ["192.168.1.101:9200"] }

stdout { codec => rubydebug }

}

~

~

message对应的日志样例如下:

"192.168.1.101 - - [15/Sep/2017:01:04:51 +0800] "GET /zentaopms/www/theme/default/zh-cn.default.css?v=8.0 HTTP/1.1" 304 0.006 0.006 652 141 0 - 1 "http://192.168.1.102:8080/zentaopms/www/index.php?m=user&f=login&referer=L3plbnRhb3Btcy93d3cvaW5kZXgucGhw" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:55.0) Gecko/20100101 Firefox/55.0" "-"",

测试配置是否正确

# cd /usr/local/logstash/logstash-5.5.2/

# bin/logstash -f logstash.conf --config.test_and_exit

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.

Sending Logstash's logs to /usr/local/logstash/logstash-5.5.2/logs which is now configured via log4j2.properties

Configuration OK

[2017-08-31T00:14:15,049][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

运行logstash

说明:如果以--config.reload.automatic方式运行,已经在运行了,修改配置后,会自动重新加载配置,不需要重新运行logstash

# bin/logstash -f logstash.conf --config.reload.automatic

17/08/18 00:53:20.024649 output.go:109: DBG  output worker: publish 323 events

2017/08/18 00:53:20.075676 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 192.168.1.103:9400: getsockopt: no route to host

2017/08/18 00:53:21.109983 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 192.168.1.103:9400: getsockopt: no route to host

2017/08/18 00:53:23.270575 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 192.168.1.103:9400: getsockopt: no route to host

2017/08/18 00:53:27.467576 single.go:140: ERR Connecting error publishing

……

解决方法:防火墙开放端口

#  firewall-cmd --permanent --zone=public --add-port=9400/tcp

success

# firewall-cmd --reload

success

[root@bogon logstash-5.5.2]# bin/logstash -f logstash.conf --config.test_and_exit

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.

Sending Logstash's logs to /usr/local/logstash/logstash-5.5.2/logs which is now configured via log4j2.properties

Configuration OK

[2017-09-03T17:56:46,275][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

# bin/logstash -f logstash.conf --config.reload.automatic

 

字段:

参考链接:

https://github.com/elastic/logstash/blob/v1.1.9/patterns/grok-patterns

参考链接:

https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html

Nginx filebeat+logstash+Elasticsearch+kibana实现nginx日志图形化展示的更多相关文章

  1. filebeat -> logstash -> elasticsearch -> kibana ELK 日志收集搭建

    Filebeat 安装参考 http://blog.csdn.net/kk185800961/article/details/54579376 elasticsearch 安装参考http://blo ...

  2. logstash+elasticsearch+kibana快速搭建日志平台

    使用logstash+elasticsearch+kibana快速搭建日志平台   日志的分析和监控在系统开发中占非常重要的地位,系统越复杂,日志的分析和监控就越重要,常见的需求有: 根据关键字查询日 ...

  3. 使用logstash+elasticsearch+kibana快速搭建日志平台

    日志的分析和监控在系统开发中占非常重要的地位,系统越复杂,日志的分析和监控就越重要,常见的需求有: * 根据关键字查询日志详情 * 监控系统的运行状况 * 统计分析,比如接口的调用次数.执行时间.成功 ...

  4. 【转载】使用logstash+elasticsearch+kibana快速搭建日志平台

    原文链接:http://www.cnblogs.com/buzzlight/p/logstash_elasticsearch_kibana_log.html 日志的分析和监控在系统开发中占非常重要的地 ...

  5. Logstash+ElasticSearch+Kibana处理nginx访问日志(转)

    ELK似乎是当前最为流行的日志收集-存储-分析的全套解决方案. 去年年初, 公司里已经在用, 当时自己还山寨了一个统计系统(postgresql-echarts, 日志无结构化, json形式存储到p ...

  6. Filebeat+Logstash+ElasticSearch+Kibana搭建Apache访问日志解析平台

    对于ELK还不太熟悉的同学可以参考我前面的两篇文章ElasticSearch + Logstash + Kibana 搭建笔记.Log stash学习笔记(一),本文搭建了一套专门访问Apache的访 ...

  7. filebeat + logstash + elasticsearch + granfa

    filebeat + logstash + elasticsearch + granfa https://www.cnblogs.com/wenchengxiaopenyou/p/9034213.ht ...

  8. 安装logstash,elasticsearch,kibana三件套

    logstash,elasticsearch,kibana三件套 elk是指logstash,elasticsearch,kibana三件套,这三件套可以组成日志分析和监控工具 注意: 关于安装文档, ...

  9. 安装logstash,elasticsearch,kibana三件套(转)

    logstash,elasticsearch,kibana三件套 elk是指logstash,elasticsearch,kibana三件套,这三件套可以组成日志分析和监控工具 注意: 关于安装文档, ...

随机推荐

  1. iview导航菜单updateOpened和updateActiveName的使用

    先看官方文档: iview导航菜单 这里主要遇到的问题有两个: 1. 点击回到首页(B按钮)时需要取消选中当前选中的菜单项(全部不选中),这里用到的是 updateActiveName方法 2. 点击 ...

  2. Git基本命令 -- 历史

    历史. 收先需要了解一下git log命令, 使用git的帮助看看: git help log: 执行该命令后, 我的win10弹出来一个html页面, 里面是git log命令的帮助: 首先看看gi ...

  3. yum install --downloadonly 下载依赖包研究

    在CentOS中可以使用yum自动安装软件,在离线环境中却行不通. Linux localhost 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 U ...

  4. hashMap的hashCode() 和equal()的使用

    hashMap的hashCode() 和equa()的使用 在java的集合中,判断两个对象是否相等的规则是: ,判断两个对象的hashCode是否相等 如果不相等,认为两个对象也不相等,完毕 如果相 ...

  5. [NewLife.XCode]数据层缓存(网站性能翻10倍)

    NewLife.XCode是一个有10多年历史的开源数据中间件,支持nfx/netcore,由新生命团队(2002~2019)开发完成并维护至今,以下简称XCode. 整个系列教程会大量结合示例代码和 ...

  6. linux http服务源码编译安装详解

    相信大家大多都听过linux 的编译安装,但它到底是怎么把源代码变为自己电脑里可以应用的软件哪?今天,小编就以httpd 为例详细讲解一下. 什么是编译安装——编译:将源代码变为机器可执行的代码文件. ...

  7. Perl文件句柄相关常量变量

    文件句柄相关变量 对应的官方手册:http://perldoc.perl.org/perlvar.html#Variables-related-to-filehandles 默认情况下: $/:输入行 ...

  8. 有关于MVC SignalR的问题

    刚拜读 @Learning hard 的 [Asp.net 开发系列之SignalR篇]专题一:Asp.net SignalR快速入门 跟着博文一步步操作,这是新人的学习方式 然而笔者的开发环境和 @ ...

  9. 如何定义一个有效的OWIN Startup Class

    命名约定 Katana在程序集内的程序集名称空间下查找一个叫做Startup的类, 通过属性指定 [assembly: OwinStartup(typeof(OwinConsoleApp.Startu ...

  10. Kafka安装与配置(windows)

    作者:灬花儿灬 出处:http://www.cnblogs.com/flower1990/ 本文版权归作者和博客园共有,欢迎转载,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文连接,否则 ...