1.导入数据
批量导入数据并查看
1.1 导入数据
1) 使用POST方式批量导入数据,数据格式为json,url 编码使用data-binary导入含有index配置的json文件
]# scp /var/ftp/elk/*.gz 192.168.1.66:/root/
kibana ~]# gzip -d logs.jsonl.gz
kibana ~]# gzip -d accounts.json.gz
kibana ~]# gzip -d shakespeare.json.gz
--data-binary @shakespeare.json
--data-binary @accounts.json
//索引是xixi,类型是haha,必须导入索引和类型,没有索引,要加上
--data-binary @logs.jsonl
2)使用GET查询结果
"docs":[
{
"_index":"shakespeare",
"_type:":"act",
"_id":0
},
{
"_index":"shakespeare",
"_type:":"line",
"_id":0
},
{
"_index":"xixi",
"_type:":"haha",
"_id":25
}
]
}'
1.2 使用kibana查看数据是否导入成功
1)数据导入以后查看logs是否导入成功,如图所示:
2)kibana导入数据,如图-2所示:
3)成功创建会有logstash-*
4)导入成功之后选择Discover,如图-4所示:
注意: 这里没有数据的原因是导入日志的时间段不对,默认配置是最近15分钟,在这可以修改一下时间来显示
5)kibana修改时间,选择Lsat 15 miuntes--选择Absolute--选择时间2015-5-15到2015-5-22,返回,查看结果
6)除了柱状图,Kibana还支持很多种展示方式,选择visualize--Forms a new Search
查看前六名访问网页的国家:
buckets--Split Slices --Terms--Fieled--geo.rsc--Count-Size(6)
查看访问网页的人都用什么操作系统的机器
buckets--小按钮折叠上一个--Split Slices --Terms--Fieled--machine.os--Count-Size(前5)
2. 综合练习
练习插件
安装一台Apache服务并配置
使用filebeat收集Apache服务器的日志
使用grok处理filebeat发送过来的日志
存入elasticsearch
2.1 安装logstash
1)配置主机名,ip和yum源,配置/etc/hosts(请把se1-se5和kibana主机配置和logstash一样的/etc/hosts)
logstash ~]# vim /etc/hosts
192.168.1.61 se1
192.168.1.62 se2
192.168.1.63 se3
192.168.1.64 se4
192.168.1.65 se5
192.168.1.66 kibana
192.168.1.67 logstash
2)安装java-1.8.0-openjdk和logstash
logstash ~]# yum -y install java-1.8.0-openjdk
logstash ~]# yum -y install logstash
logstash ~]# java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
logstash ~]# touch /etc/logstash/logstash.conf
logstash ~]# /opt/logstash/bin/logstash --version
logstash 2.3.4
logstash ~]# /opt/logstash/bin/logstash-plugin list //查看插件
...
logstash-input-stdin //标准输入插件
logstash-output-stdout //标准输出插件
...
logstash ~]# vim /etc/logstash/logstash.conf
input{
stdin{
}
}
filter{
}
output{
stdout{
}
}
//启动并测试
logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
aa
2019-03-01T08:36:25.227Z logstash aa
bb
2019-03-01T08:36:29.282Z logstash bb
备注:若不会写配置文件可以找帮助,插件文档的位置:
3)codec类插件
logstash ~]# vim /etc/logstash/logstash.conf
input{
stdin{
codec => "json" //输入设置为编码json
}
}
filter{
}
output{
stdout{
codec => "rubydebug" //输出设置为rubydebug
}
}
logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{"a":1}(手打)
{
"a" => 1,
"@version" => "1",
"@timestamp" => "2019-03-01T08:39:23.978Z",
"host" => "logstash"
}
4)file模块插件
logstash ~]# vim /etc/logstash/logstash.conf
input{
file {
path => [ "/tmp/a.log", "/var/tmp/b.log" ]
sincedb_path => "/var/lib/logstash/sincedb" //记录读取文件的位置
start_position => "beginning" //配置第一次读取文件从什么地方开始
type => "testlog" //类型名称
}
}
filter{
}
output{
stdout{
codec => "rubydebug"
}
}
logstash ~]# touch /tmp/a.log
logstash ~]# touch /var/tmp/b.log
logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
(已经在检测中)
另开一个终端:写入数据
logstash ~]# echo a1 > /tmp/a.log
logstash ~]# echo b1 > /var/tmp/b.log
显示数据:
Settings: Default pipeline workers: 2
Pipeline main started
{
"message" => "a1",
"@version" => "1",
"@timestamp" => "2019-03-01T08:43:38.633Z",
"path" => "/tmp/a.log",
"host" => "logstash",
"type" => "testlog"
}
{
"message" => "b1",
"@version" => "1",
"@timestamp" => "2019-03-01T08:43:52.671Z",
"path" => "/var/tmp/b.log",
"host" => "logstash",
"type" => "testlog"
}
5)tcp、udp模块插件
logstash ~]# vim /etc/logstash/logstash.conf
input{
file {
path => [ "/tmp/a.log", "/var/tmp/b.log" ]
sincedb_path => "/var/lib/logstash/sincedb"
start_position => "beginning"
type => "testlog"
}
tcp {
host => "0.0.0.0"
port => "8888"
type => "tcplog"
}
udp {
host => "0.0.0.0"
port => "9999"
type => "udplog"
}
}
filter{
}
output{
stdout{
codec => "rubydebug"
}
}
logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//启动,监控中
另开一个终端查看,可以看到端口
logstash tmp]# netstat -antup | grep 8888
tcp6 0 0 :::8888 :::* LISTEN 22191/java
logstash tmp]# netstat -antup | grep 9999
udp6 0 0 :::9999 :::* 22191/java
在另一台主机上写一个脚本,发送数据,使启动的logstash可以接收到数据
se5 ~]# vim tcp.sh
function sendmsg(){
if [[ "$1" == "tcp" ]];then
exec 9<>/dev/tcp/192.168.1.67/8888
else
exec 9<>/dev/udp/192.168.1.67/9999
fi
echo "$2" >&9
exec 9<&-
}
se5 ~]# . tcp.sh //重新载入一下
se5 ~]# sendmsg udp "is tcp test"
se5 ~]# sendmsg udp "is tcp ss"
logstash主机查看结果
Settings: Default pipeline workers: 2
Pipeline main started
{
"message" => "is tcp test\n",
"@version" => "1",
"@timestamp" => "2019-03-01T08:50:39.921Z",
"type" => "udplog",
"host" => "192.168.1.65"
}
{
"message" => "is tcp ss\n",
"@version" => "1",
"@timestamp" => "2019-03-01T08:50:55.608Z",
"type" => "udplog",
"host" => "192.168.1.65"
}
6)syslog插件练习
logstash ~]# systemctl list-unit-files | grep syslog
rsyslog.service enabled
syslog.socket static
logstash ~]# vim /etc/logstash/logstash.conf
input{
file {
path => [ "/tmp/a.log", "/var/tmp/b.log" ]
sincedb_path => "/var/lib/logstash/sincedb"
start_position => "beginning"
type => "testlog"
}
tcp {
host => "0.0.0.0"
port => "8888"
type => "tcplog"
}
udp {
host => "0.0.0.0"
port => "9999"
type => "udplog"
}
syslog {
port => "514"
type => "syslog"
}
}
filter{
}
output{
stdout{
codec => "rubydebug"
}
}
logstash]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
开启检测
另一个终端查看是否检测到514
logstash ~]# netstat -antup | grep 514
cp6 0 0 :::514 :::* LISTEN 1278/java
udp6 0 0 :::514 :::* 1278/java
另一台主机上面操作,本地写的日志本地可以查看
se5 ~]# vim /etc/rsyslog.conf
local0.info /var/log/mylog //自己添加这一行
se5 ~]# systemctl restart rsyslog //重启rsyslog
se5 ~]# ll /var/log/mylog //提示没有那个文件或目录
se5 ~]# logger -p local0.info -t nsd "elk" //写日志
se5 ~]# ll /var/log/mylog //再次查看,有文件
-rw------- 1 root root 29 3月 1 17:05 /var/log/mylog
se5 ~]# tail /var/log/mylog //可以查看到写的日志
Mar 1 17:05:02 se5 nsd: elk
se5 ~]# tail /var/log/messages
//可以查看到写的日志,因为配置文件里有写以.info结尾的可以收到
...
Mar 1 17:05:02 se5 nsd: elk
把本地的日志发送给远程1.67
se5 ~]# vim /etc/rsyslog.conf
local0.info @192.168.1.67:514 (添加这行)
//写一个@或两个@@都可以,一个@代表udp,两个@@代表tcp
se5 ~]# systemctl restart rsyslog
se5 ~]# logger -p local0.info -t nds "001 elk"
logstash检测到写的日志
{
"message" => "001 elk",
"@version" => "1",
"@timestamp" => "2019-03-01T09:11:12.000Z",
"type" => "syslog",
"host" => "192.168.1.65",
"priority" => 134,
"timestamp" => "Mar 1 17:11:12",
"logsource" => "se5",
"program" => "nds",
"severity" => 6,
"facility" => 16,
"facility_label" => "local0",
"severity_label" => "Informational"
}
rsyslog.conf配置向远程发送数据,远程登陆1.65的时侯,把登陆日志的信息(/var/log/secure)转发给logstash即1.67这台机器
se5 ~]# vim /etc/rsyslog.conf
57 authpriv.* @@192.168.1.67:514
//57行的/var/log/secure改为@@192.168.1.67:514
se5 ~]# systemctl restart rsyslog
logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//找一台主机登录1.65,logstash主机会有数据
Settings: Default pipeline workers: 2
Pipeline main started
{
"message" => "Accepted publickey for root from 192.168.1.254 port 40954 ssh2: RSA SHA256:+bNE5YtlrU+NbKSDeszQXuk2C8Dc0si6BMbKVybe4Ig\n",
"@version" => "1",
"@timestamp" => "2019-03-01T09:15:31.000Z",
"type" => "syslog",
"host" => "192.168.1.65",
"priority" => 86,
"timestamp" => "Mar 1 17:15:31",
"logsource" => "se5",
"program" => "sshd",
"pid" => "968",
"severity" => 6,
"facility" => 10,
"facility_label" => "security/authorization",
"severity_label" => "Informational"
}
{
"message" => "pam_unix(sshd:session): session opened for user root by (uid=0)\n",
"@version" => "1",
"@timestamp" => "2019-03-01T09:15:31.000Z",
"type" => "syslog",
"host" => "192.168.1.65",
"priority" => 86,
"timestamp" => "Mar 1 17:15:31",
"logsource" => "se5",
"program" => "sshd",
"pid" => "968",
"severity" => 6,
"facility" => 10,
"facility_label" => "security/authorization",
"severity_label" => "Informational"
}
7)filter grok插件
grok插件:
解析各种非结构化的日志数据插件
grok使用正则表达式把非结构化的数据结构化
在分组匹配,正则表达式需要根据具体数据结构编写
虽然编写困难,但适用性极广
logstash ~]# vim /etc/logstash/logstash.conf
input{
stdin{ codec => "json" }
file {
path => [ "/tmp/a.log", "/var/tmp/b.log" ]
sincedb_path => "/var/lib/logstash/sincedb"
start_position => "beginning"
type => "testlog"
}
tcp {
host => "0.0.0.0"
port => "8888"
type => "tcplog"
}
udp {
host => "0.0.0.0"
port => "9999"
type => "udplog"
}
syslog {
port => "514"
type => "syslog"
}
}
filter{
grok{
match => ["message", "(?<key>reg)"]
}
}
output{
stdout{
codec => "rubydebug"
}
}
logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
开启检测
se5 ~]# yum -y install httpd
se5 ~]# systemctl restart httpd
se5 ~]# vim /var/log/httpd/access_log
192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] "GET / HTTP/1.1" 403 4897 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
新开窗口,复制/var/log/httpd/access_log的日志到logstash下的/tmp/a.log
logstash ~]# vim /tmp/a.log
192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] "GET / HTTP/1.1" 403 4897 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
//出现message的日志,但是没有解析是什么意思
{
"message" => ".168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET / HTTP/1.1\" 403 4897 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
"@version" => "1",
"@timestamp" => "2019-03-01T09:38:24.428Z",
"path" => "/tmp/a.log",
"host" => "logstash",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
]
}
若要解决没有解析的问题,同样的方法把日志复制到/tmp/a.log,logstash.conf配置文件里面修改grok
查找正则宏路径
logstash ~]# cd /opt/logstash/vendor/bundle/
bundle]# vim grok-patterns //查找COMBINEDAPACHELOG
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
logstash ~]# vim /etc/logstash/logstash.conf
...
filter{
grok{
match => ["message", "%{COMBINEDAPACHELOG}"]
}
}
...
解析出的结果
logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
"message" => "192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET /noindex/css/open-sans.css HTTP/1.1\" 200 5081 \"
http://192.168.1.65/\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
"@version" => "1",
"@timestamp" => "2018-09-15T10:55:57.743Z",
"path" => "/tmp/a.log",
"host" => "logstash",
"type" => "testlog",
"clientip" => "192.168.1.254",
"ident" => "-",
"auth" => "-",
"timestamp" => "15/Sep/2018:18:25:46 +0800",
"verb" => "GET",
"request" => "/noindex/css/open-sans.css",
"httpversion" => "1.1",
"response" => "200",
"bytes" => "5081",
"agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\""
}
2.2 安装Apache服务,用filebeat收集Apache服务器的日志,存入elasticsearch
1)在之前安装了Apache的主机上面安装filebeat
se5 ~]# yum -y install filebeat
se5 ~]# vim /etc/filebeat/filebeat.yml
paths:
- /var/log/httpd/access_log //日志的路径,短横线加空格代表yml格式
document_type: apachelog //文档类型
elasticsearch: //加上注释
hosts: ["localhost:9200"] //加上注释
logstash: //去掉注释
hosts: ["192.168.1.67:5044"] //去掉注释,logstash那台主机的ip
se5 ~]# systemctl start filebeat
logstash ~]# vim /etc/logstash/logstash.conf
input{
stdin{ codec => "json" }
beats{
port => 5044
}
file {
path => [ "/tmp/a.log", "/var/tmp/b.log" ]
sincedb_path => "/dev/null"
start_position => "beginning"
type => "testlog"
}
tcp {
host => "0.0.0.0"
port => "8888"
type => "tcplog"
}
udp {
host => "0.0.0.0"
port => "9999"
type => "udplog"
}
syslog {
port => "514"
type => "syslog"
}
}
filter{
if [type] == "apachelog"{
grok{
match => ["message", "%{COMBINEDAPACHELOG}"]
}}
}
output{
stdout{ codec => "rubydebug" }
if [type] == "filelog"{
elasticsearch {
hosts => ["192.168.1.61:9200", "192.168.1.62:9200"]
index => "filelog"
flush_size => 2000
idle_flush_time => 10
}}
}
logstash]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
打开另一终端查看5044是否成功启动
logstash ~]# netstat -antup | grep 5044
se5 ~]# firefox 192.168.1.65 //ip为安装filebeat的那台机器
回到原来的终端,有数据
2)修改logstash.conf文件
logstash]# vim /etc/logstash/logstash.conf
...
output{
stdout{ codec => "rubydebug" }
if [type] == "apachelog"{
elasticsearch {
hosts => ["192.168.1.61:9200", "192.168.1.62:9200"]
index => "apachelog"
flush_size => 2000
idle_flush_time => 10
}}
}
浏览器访问Elasticsearch,有apachelog
##########################
知识点整理:
01:Kibana 数据展示
02:Logstash 数据采集、加工处理即传输的工具
所有类型的数据集中处理
不同模式和模式数据的正常化
自定义日志格式的迅速扩展
为自定义数据源轻松添加插件
input-->filter-->output
03:Logstash配置扩展插件
codec,input file,sincedb_path(日志读取起始位置),syslog(获取系统日志),filter grok 解析各种非结构化的日志数据插件
www.jinbuguo.com(帮助文档汉化)
04:filebeat(c语言编写,安装在每台web服务器上)将web日志发送给logstash(的beats模块)
#########################
- Coding and Paper Letter(五十八)
资源整理. 1 Coding: 1.支持TMS.WMTS标准瓦片下载,支持百度地图瓦片.高德地图瓦片.腾讯地图瓦片.天地图.ArcServer Rest瓦片.ArcServer本地缓存切片.geose ...
- Linux性能优化实战学习笔记:第五十八讲
一.上节回顾 专栏更新至今,咱们专栏最后一部分——综合案例模块也要告一段落了.很高兴看到你没有掉队,仍然在积极学习思考.实践操作,并热情地分享你在实际环境中,遇到过的各种性能问题的分析思路以及优化方法 ...
- javaweb学习总结十八(软件密码学、配置tomcat的https连接器以及tomcat管理平台)
一:软件密码学 1:对称加密 对称加密是最快速.最简单的一种加密方式,加密(encryption)与解密(decryption)用的是同样的密钥(secret key).对称加密有很多种算法,由于它效 ...
- 第五十八篇、iOS 微信聊天发送小视频的秘密
对于播放视频,大家应该一开始就想到比较方便快捷使用简单的MPMoviePlayerController类,确实用这个苹果官方为我们包装好了的 API 确实有很多事情都不用我们烦心,我们可以很快的做出一 ...
- Java开发笔记(五十八)简单接口及其实现
前面介绍了抽象方法及抽象类的用法,看似解决了不确定行为的方法定义,既然叫唤动作允许声明为抽象方法,那么飞翔.游泳也能声明为抽象方法,并且鸡类涵盖的物种不够多,最好把这些行为动作扩展到鸟类这个群体,于是 ...
- Java开发笔记(五十九)Java8之后的扩展接口
前面介绍了接口的基本用法,有心的朋友可能注意到这么一句话“在Java8以前,接口内部的所有方法都必须是抽象方法”,如此说来,在Java8之后,接口的内部方法也可能不是抽象方法了吗?之所以Java8对接 ...
- 初学Android,创建,启动,停止Service(五十八)
Service跟Windows系统里的服务概念差不多,都在后台执行,它跟Activity的最大区别就是,它是无界面的 开发Service与开发Activity的步骤类似 1.定义一个继承Service ...
- linux杂谈(十八):DNSserver的配置(一)
1.DNSserver简单介绍 域名系统(英文:Domain Name System,縮寫:DNS)是因特网的一项服务. 它作为将域名和IP地址相互映射的一个分布式数据库,可以使人更方便的訪问互联网. ...
- 十八、oracle查看、扩展表空间及linux服务器硬盘容量大小查看
/*备注:表空间是数据库的逻辑组成部分从物理上将:数据库数据存放在数据文件中从逻辑上将:数据库则是存放在表空间中表空间由一个或是多个数据文件组成*/ --1.查看用户下面的所有的表SELECT * F ...
随机推荐
- Spring Cloud 中注册中心Eureka客户端配置
注册中心配置客户端(注册一个虚拟的商品服务) 一.新建项目: 1.创建一个SpirngBoot应用,增加服务注册和发现依赖 2.模拟商品信息,存储在内存中 3.开发商品列表接口 ...
- (转) 从0移植uboot(五) _实现串口输出
ref : https://www.cnblogs.com/xiaojiang1025/p/6500520.html 串口作为一种非常简单的通信方式,才是嵌入式系统调试的王道,通过设置串口输出,我们可 ...
- Task执行多次
项目中,曾经出现过启动时数据库连接数瞬间增大,当时并没有注意该问题. 后期,由于Task任务多次执行,才着手查看这个问题,经排查,由于tomcat中webapp配置多次,导致webapp被扫描多次(配 ...
- (一)easyUI之第一个demo
一.下载 官网下载 : http://www.jeasyui.net/download/ 同时并下载官方中文API文档. 解压后的目录结构: 二.第一个demo 1 新建工程并导入包 2 ...
- (九) spring 使用自定义限定符注解
案例一 定义接口 CD.java package interfacepackage; public interface CD { void play(); } 定义接口 player .java p ...
- (六)maven之常用插件
一.maven的插件 maven官方插件:http://maven.apache.org/plugins/index.html 二.常用插件之:javadoc 作用:生成代码文档 2.1 编写代码,并 ...
- python练习:函数3
习题: 用lambda和filter完成下面功能:输出一个列表,列表里面包括:1-100内的所有偶数.(提示:可以用filter,lambda) [ x for x in range(1,101) i ...
- 三、eureka服务端获取服务列表
所有文章 https://www.cnblogs.com/lay2017/p/11908715.html 正文 eureka服务端维护了一个服务信息的列表,服务端节点之间相互复制服务信息.而作为eur ...
- 两个重叠的div做前后翻转
当需要做一个翻转卡片式的div时候,需要两个div的大小等大例如: 画出两个等大的div后,将他们重叠 图中的两个div做了重叠,做重叠时候用的属性是 position: absolute; 并且需要 ...
- 使用百度echarts仿雪球分时图(一)
第一次写技术博客,有不足的地方希望大家指证出来,我再加以改正,谢谢大家. 之前一直没有找到一个合适的分时图项目,所以决定自己动手撸一个.接触的图表框架不多,在网上看到不少人推荐使用echarts,看了 ...