Nginx filebeat+logstash+Elasticsearch+kibana实现nginx日志图形化展示
filebeat+logstash+Elasticsearch+kibana实现nginx日志图形化展示
by:授客 QQ:1033553122
测试环境
Win7 64
CentOS-7-x86_64-DVD-1503-01.iso(kibana安装环境)
CentOS 6.5-x86_64(其它软件安装环境)
nginx-1.10.0
filebeat-5.5.2-linux-x86_64.tar.gz
下载地址:
https://pan.baidu.com/s/1dEBkIuH
https://www.elastic.co/downloads/beats/filebeat#ga-release
kibana-5.5.0-linux-x86_64.tar.gz
下载地址:
https://pan.baidu.com/s/1dEBkIuH
logstash-5.5.2.tar.gz
下载地址:
https://pan.baidu.com/s/1dEBkIuH
https://www.elastic.co/downloads/logstash
elasticsearch-5.5.2
下载地址:
https://pan.baidu.com/s/1dEBkIuH
https://www.elastic.co/downloads/elasticsearch#preview-release
安装Nginx
略
Nginx日志配置
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $request_time $upstream_response_time $request_length $bytes_sent $body_bytes_sent $gzip_ratio $connection_requests "$http_referer" "$http_user_agent" "$http_x_forwarded_for"';
access_log logs/access.log main;
运行nginx
安装java
参考文章:
http://blog.sina.com.cn/s/blog_13cc013b50102w01m.html#_Toc438402186
[root@bogon ~]# java -version
java version "1.8.0_65"
64-Bit Server VM (build 25.65-b01, mixed mode)
注意:logstash要求Java 8,不支持java9
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
安装logstash
# tar -xzvf logstash-5.5.2.tar.gz
# ls
logstash-5.5.2 logstash-5.5.2.tar.gz
# mkdir -p /usr/local/logstash
# mv logstash-5.5.2 /usr/local/logstash/
配置logstash
# vim /usr/local/logstash/logstash-5.5.2/logstash.conf
input { stdin {} }
output {
elasticsearch { hosts => ["192.168.1.101:9200"] }
stdout { codec => rubydebug }
}
说明:
input { stdin {} } 表示从标准输入中接收数据
192.168.1.101:9200 分别代表Elasticsearch搜索访问ip和监听端口
stdout { codec => rubydebug } 表示输出到控制台
参考链接:
https://www.elastic.co/guide/en/logstash/current/config-examples.html
运行logstash
# cd /usr/local/logstash/logstash-5.5.2/
# bin/logstash -f logstash.conf
……(略)
The stdin plugin is now waiting for input:
[2017-07-14T03:40:50,373][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
hello world
{
"@timestamp" => 2017-07-13T19:59:53.848Z,
"@version" => "1",
"host" => "0.0.0.0",
"message" => "hello world"
}
说明:启动后,输入上述带背景色内容 hello world,待控制台输出带黄色背景色内容后,在Elasticsearch中执行搜索,如下
GET /logstash-2017.07.13/_search
如上图,能搜索到输入数据,说明成功了
停止运行logstash
按CTRL + D键
参考链接:
https://www.elastic.co/guide/en/logstash/current/first-event.html
安装Elasticsearch
略
安装kibana
# mkdir -p /usr/local/kibana
# tar -xvzf kibana-5.5.0-linux-x86_64.tar.gz
# mv kibana-5.5.0-linux-x86_64 /usr/local/kibana/
参考链接:
https://www.elastic.co/guide/en/kibana/current/targz.html
配置kibana
# cd /usr/local/kibana/kibana-5.5.0-linux-x86_64/config/
# vim kibana.yml
server.host: "192.168.1.104"
elasticsearch.url: "http://192.168.1.101:9200"
参考链接:
https://www.elastic.co/guide/en/kibana/current/settings.html
运行kibana
# cd /usr/local/kibana/kibana-5.5.0-linux-x86_64/
# ./bin/kibana
log [23:51:04.051] [info][status][plugin:kibana@5.5.0] Status changed from uninitialized to green - Ready
log [23:51:04.510] [info][status][plugin:elasticsearch@5.5.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [23:51:04.594] [info][status][plugin:console@5.5.0] Status changed from uninitialized to green - Ready
log [23:51:04.617] [warning] You're running Kibana 5.5.0 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v5.5.2 @ 192.168.1.101:9200 (192.168.1.101)
log [23:51:04.674] [info][status][plugin:metrics@5.5.0] Status changed from uninitialized to green - Ready
log [23:51:04.706] [info][status][plugin:elasticsearch@5.5.0] Status changed from yellow to green - Kibana index ready
log [23:51:06.992] [info][status][plugin:timelion@5.5.0] Status changed from uninitialized to green - Ready
log [23:51:07.032] [info][listening] Server running at http://192.168.1.104:5601
log [23:51:07.037] [info][status][ui settings] Status changed from uninitialized to green - Ready
验证
浏览器中访问:http://192.168.1.104:5601/status
结果发现打不开
解决方法:停止防火墙
# systemctl stop firewalld.service
再次访问
参考链接:
https://www.elastic.co/guide/en/kibana/current/access.html
配置索引模式(index pattern)
要使用Kibana至少需要配置一个索引模式(index pattern)。索引模式用于确认执行搜索和分析的Elasticsearch索引。
窗体顶端
Index name or pattern
配置索引名称,或者索引模式。索引模式允许使用通配符 * 。比如 logstash-*
Time Filter field name
设置时间过滤器,方便在Discover页面中按时间筛选数据
窗体底端
Management -> Index Patterns -> Create Index Pattern,重新设置
参考链接:
https://www.elastic.co/guide/en/kibana/current/connect-to-elasticsearch.html
安装filebeat
# tar -xvzf filebeat-5.5.2-linux-x86_64.tar.gz
# mkdir -p /usr/local/filebeat
# mv filebeat-5.5.2-linux-x86_64 /usr/local/filebeat/
配置
# vim /usr/local/filebeat/filebeat-5.5.2-linux-x86_64/filebeat.yml
配置日志文件路径
如上,可以指定具体的文件名,
- /usr/local/ngnix/logs/access.log
- /usr/local/ngnix/logs/error.log
也可以使用通配符,表示/usr/local/ngnix/logs/目录下,所有.log文件
- /usr/local/ngnix/logs/*.log
配置logstash输出
注意:hosts: 后面必须接一个空格,否则会报错
测试配置是否正确
# cd /usr/local/filebeat/filebeat-5.5.2-linux-x86_64/
# ./filebeat -configtest -e
2017/08/17 23:55:32.651228 beat.go:285: INFO Home path: [/usr/local/filebeat/filebeat-5.5.2-linux-x86_64] Config path: [/usr/local/filebeat/filebeat-5.5.2-linux-x86_64] Data path: [/usr/local/filebeat/filebeat-5.5.2-linux-x86_64/data] Logs path: [/usr/local/filebeat/filebeat-5.5.2-linux-x86_64/logs]
2017/08/17 23:55:32.651335 beat.go:186: INFO Setup Beat: filebeat; Version: 5.5.2
2017/08/17 23:55:32.651564 logstash.go:90: INFO Max Retries set to: 3
2017/08/17 23:55:32.652006 outputs.go:108: INFO Activated logstash as output plugin.
2017/08/17 23:55:32.652250 metrics.go:23: INFO Metrics logging every 30s
2017/08/17 23:55:32.662026 publish.go:295: INFO Publisher name: bogon
2017/08/17 23:55:32.698907 async.go:63: INFO Flush Interval set to: 1s
2017/08/17 23:55:32.699214 async.go:64: INFO Max Bulk Size set to: 2048
Config OK
运行filebeat
# ./filebeat -e -c filebeat.yml -d "publish"
参考链接:
https://www.elastic.co/guide/en/beats/filebeat/5.5/filebeat-starting.html
https://www.elastic.co/guide/en/beats/filebeat/5.5/config-filebeat-logstash.html
修改logstash配置
[root@bogon logstash-5.5.2]# vim logstash.conf
input {
beats {
port => "9400"
}
}
filter{
grok {
match => {"message" => "%{IP:remote_addr} - %{USER:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} %{DATA:http_version}" %{NUMBER:status:int} %{NUMBER:request_time:float} %{NUMBER:upstream_response_time:float} %{NUMBER:request_length:int} %{NUMBER:bytes_sent:int} %{NUMBER:body_bytes_sent:int} %{DATA:gzip_ratio:float} %{NUMBER:connection_requests:int} "%{DATA:http_referer}" %{QUOTEDSTRING:http_user_agent} %{DATA:http_x_forwarded_for}"}
}
}
output {
elasticsearch { hosts => ["192.168.1.101:9200"] }
stdout { codec => rubydebug }
}
~
~
message对应的日志样例如下:
"192.168.1.101 - - [15/Sep/2017:01:04:51 +0800] "GET /zentaopms/www/theme/default/zh-cn.default.css?v=8.0 HTTP/1.1" 304 0.006 0.006 652 141 0 - 1 "http://192.168.1.102:8080/zentaopms/www/index.php?m=user&f=login&referer=L3plbnRhb3Btcy93d3cvaW5kZXgucGhw" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:55.0) Gecko/20100101 Firefox/55.0" "-"",
测试配置是否正确
# cd /usr/local/logstash/logstash-5.5.2/
# bin/logstash -f logstash.conf --config.test_and_exit
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /usr/local/logstash/logstash-5.5.2/logs which is now configured via log4j2.properties
Configuration OK
[2017-08-31T00:14:15,049][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
运行logstash
说明:如果以--config.reload.automatic方式运行,已经在运行了,修改配置后,会自动重新加载配置,不需要重新运行logstash
# bin/logstash -f logstash.conf --config.reload.automatic
17/08/18 00:53:20.024649 output.go:109: DBG output worker: publish 323 events
2017/08/18 00:53:20.075676 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 192.168.1.103:9400: getsockopt: no route to host
2017/08/18 00:53:21.109983 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 192.168.1.103:9400: getsockopt: no route to host
2017/08/18 00:53:23.270575 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 192.168.1.103:9400: getsockopt: no route to host
2017/08/18 00:53:27.467576 single.go:140: ERR Connecting error publishing
……
解决方法:防火墙开放端口
# firewall-cmd --permanent --zone=public --add-port=9400/tcp
success
# firewall-cmd --reload
success
[root@bogon logstash-5.5.2]# bin/logstash -f logstash.conf --config.test_and_exit
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /usr/local/logstash/logstash-5.5.2/logs which is now configured via log4j2.properties
Configuration OK
[2017-09-03T17:56:46,275][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
# bin/logstash -f logstash.conf --config.reload.automatic
字段:
参考链接:
https://github.com/elastic/logstash/blob/v1.1.9/patterns/grok-patterns
参考链接:
https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html
Nginx filebeat+logstash+Elasticsearch+kibana实现nginx日志图形化展示的更多相关文章
- filebeat -> logstash -> elasticsearch -> kibana ELK 日志收集搭建
Filebeat 安装参考 http://blog.csdn.net/kk185800961/article/details/54579376 elasticsearch 安装参考http://blo ...
- logstash+elasticsearch+kibana快速搭建日志平台
使用logstash+elasticsearch+kibana快速搭建日志平台 日志的分析和监控在系统开发中占非常重要的地位,系统越复杂,日志的分析和监控就越重要,常见的需求有: 根据关键字查询日 ...
- 使用logstash+elasticsearch+kibana快速搭建日志平台
日志的分析和监控在系统开发中占非常重要的地位,系统越复杂,日志的分析和监控就越重要,常见的需求有: * 根据关键字查询日志详情 * 监控系统的运行状况 * 统计分析,比如接口的调用次数.执行时间.成功 ...
- 【转载】使用logstash+elasticsearch+kibana快速搭建日志平台
原文链接:http://www.cnblogs.com/buzzlight/p/logstash_elasticsearch_kibana_log.html 日志的分析和监控在系统开发中占非常重要的地 ...
- Logstash+ElasticSearch+Kibana处理nginx访问日志(转)
ELK似乎是当前最为流行的日志收集-存储-分析的全套解决方案. 去年年初, 公司里已经在用, 当时自己还山寨了一个统计系统(postgresql-echarts, 日志无结构化, json形式存储到p ...
- Filebeat+Logstash+ElasticSearch+Kibana搭建Apache访问日志解析平台
对于ELK还不太熟悉的同学可以参考我前面的两篇文章ElasticSearch + Logstash + Kibana 搭建笔记.Log stash学习笔记(一),本文搭建了一套专门访问Apache的访 ...
- filebeat + logstash + elasticsearch + granfa
filebeat + logstash + elasticsearch + granfa https://www.cnblogs.com/wenchengxiaopenyou/p/9034213.ht ...
- 安装logstash,elasticsearch,kibana三件套
logstash,elasticsearch,kibana三件套 elk是指logstash,elasticsearch,kibana三件套,这三件套可以组成日志分析和监控工具 注意: 关于安装文档, ...
- 安装logstash,elasticsearch,kibana三件套(转)
logstash,elasticsearch,kibana三件套 elk是指logstash,elasticsearch,kibana三件套,这三件套可以组成日志分析和监控工具 注意: 关于安装文档, ...
随机推荐
- iview导航菜单updateOpened和updateActiveName的使用
先看官方文档: iview导航菜单 这里主要遇到的问题有两个: 1. 点击回到首页(B按钮)时需要取消选中当前选中的菜单项(全部不选中),这里用到的是 updateActiveName方法 2. 点击 ...
- Git基本命令 -- 历史
历史. 收先需要了解一下git log命令, 使用git的帮助看看: git help log: 执行该命令后, 我的win10弹出来一个html页面, 里面是git log命令的帮助: 首先看看gi ...
- yum install --downloadonly 下载依赖包研究
在CentOS中可以使用yum自动安装软件,在离线环境中却行不通. Linux localhost 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 U ...
- hashMap的hashCode() 和equal()的使用
hashMap的hashCode() 和equa()的使用 在java的集合中,判断两个对象是否相等的规则是: ,判断两个对象的hashCode是否相等 如果不相等,认为两个对象也不相等,完毕 如果相 ...
- [NewLife.XCode]数据层缓存(网站性能翻10倍)
NewLife.XCode是一个有10多年历史的开源数据中间件,支持nfx/netcore,由新生命团队(2002~2019)开发完成并维护至今,以下简称XCode. 整个系列教程会大量结合示例代码和 ...
- linux http服务源码编译安装详解
相信大家大多都听过linux 的编译安装,但它到底是怎么把源代码变为自己电脑里可以应用的软件哪?今天,小编就以httpd 为例详细讲解一下. 什么是编译安装——编译:将源代码变为机器可执行的代码文件. ...
- Perl文件句柄相关常量变量
文件句柄相关变量 对应的官方手册:http://perldoc.perl.org/perlvar.html#Variables-related-to-filehandles 默认情况下: $/:输入行 ...
- 有关于MVC SignalR的问题
刚拜读 @Learning hard 的 [Asp.net 开发系列之SignalR篇]专题一:Asp.net SignalR快速入门 跟着博文一步步操作,这是新人的学习方式 然而笔者的开发环境和 @ ...
- 如何定义一个有效的OWIN Startup Class
命名约定 Katana在程序集内的程序集名称空间下查找一个叫做Startup的类, 通过属性指定 [assembly: OwinStartup(typeof(OwinConsoleApp.Startu ...
- Kafka安装与配置(windows)
作者:灬花儿灬 出处:http://www.cnblogs.com/flower1990/ 本文版权归作者和博客园共有,欢迎转载,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文连接,否则 ...