Logstash——multiline 插件，匹配多行日志
本文内容
- 测试数据
- 字段属性
- 按多行解析运行时日志
- 把多行日志解析到字段
- 参考资料
在处理日志时，除了访问日志外，还要处理运行时日志，该日志大都用程序写的，比如 log4j。运行时日志跟访问日志最大的不同是，运行时日志是多行，也就是说，连续的多行才能表达一个意思。
本文主要说明，如何用 multiline 出来运行日志。
如果能按多行处理，那么把他们拆分到字段就很容易了。
迁移到：http://www.bdata-cap.com/newsinfo/1712113.html
测试数据
[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS asc
[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS desc
[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS asc
[16-04-12 03:40:07 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
测试是在7秒内发生的（当然是假数据）。可以看到，第二、五、六秒的日志是多行的，有条SQL语句。其他是单行的。
字段属性
对 multiline 插件来说，有三个设置比较重要：negate、pattern 和 what。
negate
类型是 boolean
默认为
false否定正则表达式（如果没有匹配的话）。
pattern
必须设置
类型为 string
没有默认值
要匹配的正则表达式。
what
必须设置
可以为
previous 或next没有默认值
如果正则表达式匹配了，那么该事件是属于下一个或是前一个事件？
按多行解析运行时日志
示例1：若配置文件如下所示，
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=> multiline {
pattern => "^\["
negate => true
what => "previous"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
output{
stdout{
codec=>rubydebug
}
}
说明：匹配以“[”开头的行，如果不是，那肯定是属于前一行的。
解析结果如下所示，能解析出6个JSON：
{
"@timestamp" => "2016-06-01T04:37:43.147Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.152Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.152Z",
"message" => "[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.155Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.157Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.159Z",
"message" => "[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
解析时，最后一行日志，不会解析。只有当再追加一条日志时，才会解析最后一条日志。
示例2：若将配置文件修改为，
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=>multiline {
pattern => "^\["
negate => true
what => "next"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
output{
stdout{
codec=>rubydebug
}
}
解析结果为，能解析出7个JSON：
{
"@timestamp" => "2016-06-01T04:40:43.232Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.237Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.238Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc\n[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.239Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.244Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.245Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc\n[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.249Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc\n[16-04-12 03:40:07 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
示例3：若将配置文件修改为，
codec=>multiline {
pattern => "^\["
negate => false
what => "previous"
}
则解析结果为：
{
"@timestamp" => "2016-06-01T05:38:50.853Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.856Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.858Z",
"message" => "order by product_category.ORDERS asc\n[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.860Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.861Z",
"message" => "order by product_category.ORDERS desc\n[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.863Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
把多行日志解析到字段
配置文件如下所示：
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=>multiline {
pattern => "^\["
negate => true
what => "previous"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
filter {
grok {
match=>["message","\[%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}\] %{GREEDYDATA:msg}"]
}
}
output{
stdout{
codec=>rubydebug
}
}
解析后结果：
{
"@timestamp" => "2016-06-01T06:33:26.426Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:01",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.485Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:02",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc"
}
{
"@timestamp" => "2016-06-01T06:33:26.491Z",
"message" => "[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:03",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.492Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:04",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.494Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:05",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc"
}
{
"@timestamp" => "2016-06-01T06:33:26.495Z",
"message" => "[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:06",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc"
}
参考资料
Logstash——multiline 插件，匹配多行日志的更多相关文章
- logstash之multiline插件，匹配多行日志
在外理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如log4j.运行时日志跟访问日志最大的不同是,运行时日志是多行,也就是说,连续的多行才能表达一个意思. 在filter中,加 ...
- Logstash——multiline 插件，匹配多行日志
本文内容 测试数据 字段属性 按多行解析运行时日志 把多行日志解析到字段 参考资料 在处理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如 log4j.运行时日志跟访问日志最大的 ...
- Logstash-安装logstash-filter-multiline插件(解决logstash匹配多行日志)
ELK-logstash在搬运日志的时候会出现多行日志,普通的搬运会造成保存到ES中日志一条一条的保存,很丑,而且不方便读取,logstash-filter-multiline可以解决该问题. 接下来 ...
- logstash匹配多行日志
在工作中,遇到一个问题就是日志的处理,首选的方案就是ELFK(filebeat+logstash+es+kibana) 因为之前使用过logstash采集日志的时候,非常的消耗系统的资源,所以这里我选 ...
- Python正则处理多行日志一例
正则表达式基础知识请参阅<正则表达式基础知识>,本文使用正则表达式来匹配多行日志并从中解析出相应的信息. 假设现在有这样的SQL日志: SELECT * FROM open_app WHE ...
- Python正则处理多行日志一例(可配置化)
正则表达式基础知识请参阅<正则表达式基础知识>,本文使用正则表达式来匹配多行日志并从中解析出相应的信息. 假设现在有这样的SQL日志: SELECT * FROM open_app WHE ...
- 写给大忙人的ELK最新版6.2.4学习笔记-Logstash和Filebeat解析(java异常堆栈下多行日志配置支持)
接前一篇CentOS 7下最新版(6.2.4)ELK+Filebeat+Log4j日志集成环境搭建完整指南,继续对ELK. logstash官方最新文档https://www.elastic.co/g ...
- ELK学习笔记之Logstash和Filebeat解析对java异常堆栈下多行日志配置支持
0x00 概述 logstash官方最新文档.假设有几十台服务器,每台服务器要监控系统日志syslog.tomcat日志.nginx日志.mysql日志等等,监控OOM.内存低下进程被kill.ngi ...
- logstash 安装插件multiline
一.安装multiline 在使用elk 传输记录 java 日志时,如下 一个java的报错 在elk中会按每一行 产生多条记录,不方便查阅 这里修改配置文件 使用 multiline 插件 ...
随机推荐
- Eclipse的安装和java环境变量的设置
首先准备工作是要下载好Eclipse和java JDK. 必须要注意的是,Eclipse和java JDK必须下载同一位数的版本,即64位同为64位,32位同为32位.否则在安装完成运行Eclipse ...
- 开始学CI
未来一段时间的学习计划 1.codeIgniter 2.angular JS 深入 3.react 4.python 边工作边学习,保持进步
- Subsets 子集系列问题 leetcode
子集系列问题: Coding 问题中有时会出现这样的问题:给定一个集合,求出这个集合所有的子集(所谓子集,就是包含原集合中的一部分元素的集合). 或者求出满足一定要求的子集,比如子集中元素总和为定值, ...
- React Native填坑之旅--动画
动画是提高用户体验不可缺少的一个元素.恰如其分的动画可以让用户更明确的感知当前的操作是什么. 无疑在使用React Native开发应用的时候也需要动画.这就需要知道RN都给我们提供了那些动画,和每个 ...
- 给“.Net工资低”争论一个了结吧！
昨天我写了一篇<工资低的.Net程序员,活该你工资低>,底下的支持.争吵.骂娘的评论依旧像之前几篇园友的博客一样繁荣.公说公有理,婆说婆有理,这样争吵下去永远没有尽头.数据没有情绪,是公正 ...
- （ 转）基于.NET平台常用的框架整理
自从学习.NET以来,优雅的编程风格,极度简单的可扩展性,足够强大开发工具,极小的学习曲线,让我对这个平台产生了浓厚的兴趣,在工作和学习中也积累了一些开源的组件,就目前想到的先整理于此,如果再想到,就 ...
- ASP.NET MVC学习之Ajax（完结）
一.前言 通过上面的一番学习,大家一定收获不少.但是总归会有一个结束的时候,但是这个结束也意味着新的开始. 如果你是从事ASP.NET开发,并且也使用了第三方控件,那么一定会觉得ASP.NET开发aj ...
- Dynamic CRM 2013学习笔记（十三）附件上传 / 上传附件
上传附件可能是CRM里比较常用的一个需求了,本文将介绍如何在CRM里实现附件的上传.显示及下载.包括以下几个步骤: 附件上传的web页面 附件显示及下载的附件实体 调用上传web页面的JS文件 实体上 ...
- 学习WPF——了解WPF中的XAML
XAML的简单说明 XAML是用于实例化.NET对象的标记语言,主要用于构建WPF的用户界面 XAML中的每一个元素都映射为.NET类的一个实例,例如<Button>映射为WPF的Butt ...
- 写给自己看的Linux运维基础(三) - Mono
如果使用ubuntu,可使用apg-get安装完mono,xsp,mod_mono,我的yum源并没有找到mono可安装,网上也没找到CentOS的源,最后选择下载编译安装. Stackoverflo ...