Logstash——multiline 插件，匹配多行日志
本文内容
- 测试数据
- 字段属性
- 按多行解析运行时日志
- 把多行日志解析到字段
- 参考资料
在处理日志时，除了访问日志外，还要处理运行时日志，该日志大都用程序写的，比如 log4j。运行时日志跟访问日志最大的不同是，运行时日志是多行，也就是说，连续的多行才能表达一个意思。
本文主要说明，如何用 multiline 出来运行日志。
如果能按多行处理，那么把他们拆分到字段就很容易了。
迁移到：http://www.bdata-cap.com/newsinfo/1712113.html
测试数据
[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS asc
[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS desc
[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS asc
[16-04-12 03:40:07 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
测试是在7秒内发生的（当然是假数据）。可以看到，第二、五、六秒的日志是多行的，有条SQL语句。其他是单行的。
字段属性
对 multiline 插件来说，有三个设置比较重要：negate、pattern 和 what。
negate
类型是 boolean
默认为
false否定正则表达式（如果没有匹配的话）。
pattern
必须设置
类型为 string
没有默认值
要匹配的正则表达式。
what
必须设置
可以为
previous 或next没有默认值
如果正则表达式匹配了，那么该事件是属于下一个或是前一个事件？
按多行解析运行时日志
示例1：若配置文件如下所示，
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=> multiline {
pattern => "^\["
negate => true
what => "previous"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
output{
stdout{
codec=>rubydebug
}
}
说明：匹配以“[”开头的行，如果不是，那肯定是属于前一行的。
解析结果如下所示，能解析出6个JSON：
{
"@timestamp" => "2016-06-01T04:37:43.147Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.152Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.152Z",
"message" => "[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.155Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.157Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.159Z",
"message" => "[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
解析时，最后一行日志，不会解析。只有当再追加一条日志时，才会解析最后一条日志。
示例2：若将配置文件修改为，
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=>multiline {
pattern => "^\["
negate => true
what => "next"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
output{
stdout{
codec=>rubydebug
}
}
解析结果为，能解析出7个JSON：
{
"@timestamp" => "2016-06-01T04:40:43.232Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.237Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.238Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc\n[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.239Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.244Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.245Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc\n[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.249Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc\n[16-04-12 03:40:07 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
示例3：若将配置文件修改为，
codec=>multiline {
pattern => "^\["
negate => false
what => "previous"
}
则解析结果为：
{
"@timestamp" => "2016-06-01T05:38:50.853Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.856Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.858Z",
"message" => "order by product_category.ORDERS asc\n[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.860Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.861Z",
"message" => "order by product_category.ORDERS desc\n[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.863Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
把多行日志解析到字段
配置文件如下所示：
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=>multiline {
pattern => "^\["
negate => true
what => "previous"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
filter {
grok {
match=>["message","\[%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}\] %{GREEDYDATA:msg}"]
}
}
output{
stdout{
codec=>rubydebug
}
}
解析后结果：
{
"@timestamp" => "2016-06-01T06:33:26.426Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:01",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.485Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:02",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc"
}
{
"@timestamp" => "2016-06-01T06:33:26.491Z",
"message" => "[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:03",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.492Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:04",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.494Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:05",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc"
}
{
"@timestamp" => "2016-06-01T06:33:26.495Z",
"message" => "[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:06",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc"
}
参考资料
Logstash——multiline 插件，匹配多行日志的更多相关文章
- logstash之multiline插件，匹配多行日志
在外理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如log4j.运行时日志跟访问日志最大的不同是,运行时日志是多行,也就是说,连续的多行才能表达一个意思. 在filter中,加 ...
- Logstash——multiline 插件，匹配多行日志
本文内容 测试数据 字段属性 按多行解析运行时日志 把多行日志解析到字段 参考资料 在处理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如 log4j.运行时日志跟访问日志最大的 ...
- Logstash-安装logstash-filter-multiline插件(解决logstash匹配多行日志)
ELK-logstash在搬运日志的时候会出现多行日志,普通的搬运会造成保存到ES中日志一条一条的保存,很丑,而且不方便读取,logstash-filter-multiline可以解决该问题. 接下来 ...
- logstash匹配多行日志
在工作中,遇到一个问题就是日志的处理,首选的方案就是ELFK(filebeat+logstash+es+kibana) 因为之前使用过logstash采集日志的时候,非常的消耗系统的资源,所以这里我选 ...
- Python正则处理多行日志一例
正则表达式基础知识请参阅<正则表达式基础知识>,本文使用正则表达式来匹配多行日志并从中解析出相应的信息. 假设现在有这样的SQL日志: SELECT * FROM open_app WHE ...
- Python正则处理多行日志一例(可配置化)
正则表达式基础知识请参阅<正则表达式基础知识>,本文使用正则表达式来匹配多行日志并从中解析出相应的信息. 假设现在有这样的SQL日志: SELECT * FROM open_app WHE ...
- 写给大忙人的ELK最新版6.2.4学习笔记-Logstash和Filebeat解析(java异常堆栈下多行日志配置支持)
接前一篇CentOS 7下最新版(6.2.4)ELK+Filebeat+Log4j日志集成环境搭建完整指南,继续对ELK. logstash官方最新文档https://www.elastic.co/g ...
- ELK学习笔记之Logstash和Filebeat解析对java异常堆栈下多行日志配置支持
0x00 概述 logstash官方最新文档.假设有几十台服务器,每台服务器要监控系统日志syslog.tomcat日志.nginx日志.mysql日志等等,监控OOM.内存低下进程被kill.ngi ...
- logstash 安装插件multiline
一.安装multiline 在使用elk 传输记录 java 日志时,如下 一个java的报错 在elk中会按每一行 产生多条记录,不方便查阅 这里修改配置文件 使用 multiline 插件 ...
随机推荐
- php 中文正则
utf8编码中文 preg_match("/^[\x{4e00}-\x{9fa5}]+$/u") 而不是 "/^[\x4e00-\x9fa5]+$/u"
- 常用Sql语句书写
一.增:有2种方法 1.使用insert插入单行数据: 语法:insert [into] <表名> [列名] values <列值> 例:insert into Strdent ...
- 1606: [Usaco2008 Dec]Hay For Sale 购买干草
Description 约翰遭受了重大的损失:蟑螂吃掉了他所有的干草,留下一群饥饿的牛．他乘着容量为C(1≤C≤50000)个单位的马车,去顿因家买一些干草． 顿因有H(1≤H≤5000)包 ...
- MVC_Ajax请求
MVC_Ajax请求MVC中的AJAX操作原理还是基于Jquery的封装操作.但是吧没有那么恐怖.Ajax.BeginForm:使用Ajax.BeginForm方法会生成一个form表单,最后以Aja ...
- XproerIM产品使用手册
1. 产品介绍 版权所有:(c)2009-2016 荆门泽优软件有限公司 保留所有权利. 产品官网:http://www.ncmem.com/apps/xproerim/index.asp 安装 ...
- ＃笔记＃JavaScript进阶篇二
#常用函数对象属性介绍2 getAttribute()方法—— 通过元素节点的属性名称获取属性的值. 语法: elementNode.getAttribute(name) 说明: 1. name:要想 ...
- Java小应用程序Applet，画布上新建按钮和文本
<pre name="code" class="java">package com.hx; import java.applet.*; import ...
- [原] XAF ListView显示隐藏Footer菜单
using System; using DevExpress.ExpressApp; using DevExpress.ExpressApp.Win.Editors; using DevExpress ...
- IQueryable和IQueryProvider初尝
前言 相信大家对Entity Framework一定不陌生,我相信其中Linq To Sql是其最大的亮点之一,但是我们一直使用到现在却不曾明白内部是如何实现的,今天我们就简单的介绍IQueryabl ...
- WPF快速入门系列(6)——WPF资源和样式
一.引言 WPF资源系统可以用来保存一些公有对象和样式,从而实现重用这些对象和样式的作用.而WPF样式是重用元素的格式的重要手段,可以理解样式就如CSS一样,尽管我们可以在每个控件中定义格式,但是如果 ...