Logstash——multiline 插件，匹配多行日志
本文内容
- 测试数据
- 字段属性
- 按多行解析运行时日志
- 把多行日志解析到字段
- 参考资料
在处理日志时，除了访问日志外，还要处理运行时日志，该日志大都用程序写的，比如 log4j。运行时日志跟访问日志最大的不同是，运行时日志是多行，也就是说，连续的多行才能表达一个意思。
本文主要说明，如何用 multiline 出来运行日志。
如果能按多行处理，那么把他们拆分到字段就很容易了。
迁移到：http://www.bdata-cap.com/newsinfo/1712113.html
测试数据
[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS asc
[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS desc
[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS asc
[16-04-12 03:40:07 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
测试是在7秒内发生的（当然是假数据）。可以看到，第二、五、六秒的日志是多行的，有条SQL语句。其他是单行的。
字段属性
对 multiline 插件来说，有三个设置比较重要：negate、pattern 和 what。
negate
类型是 boolean
默认为
false否定正则表达式（如果没有匹配的话）。
pattern
必须设置
类型为 string
没有默认值
要匹配的正则表达式。
what
必须设置
可以为
previous 或next没有默认值
如果正则表达式匹配了，那么该事件是属于下一个或是前一个事件？
按多行解析运行时日志
示例1：若配置文件如下所示，
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=> multiline {
pattern => "^\["
negate => true
what => "previous"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
output{
stdout{
codec=>rubydebug
}
}
说明：匹配以“[”开头的行，如果不是，那肯定是属于前一行的。
解析结果如下所示，能解析出6个JSON：
{
"@timestamp" => "2016-06-01T04:37:43.147Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.152Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.152Z",
"message" => "[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.155Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.157Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.159Z",
"message" => "[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
解析时，最后一行日志，不会解析。只有当再追加一条日志时，才会解析最后一条日志。
示例2：若将配置文件修改为，
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=>multiline {
pattern => "^\["
negate => true
what => "next"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
output{
stdout{
codec=>rubydebug
}
}
解析结果为，能解析出7个JSON：
{
"@timestamp" => "2016-06-01T04:40:43.232Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.237Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.238Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc\n[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.239Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.244Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.245Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc\n[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.249Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc\n[16-04-12 03:40:07 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
示例3：若将配置文件修改为，
codec=>multiline {
pattern => "^\["
negate => false
what => "previous"
}
则解析结果为：
{
"@timestamp" => "2016-06-01T05:38:50.853Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.856Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.858Z",
"message" => "order by product_category.ORDERS asc\n[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.860Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.861Z",
"message" => "order by product_category.ORDERS desc\n[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.863Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
把多行日志解析到字段
配置文件如下所示：
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=>multiline {
pattern => "^\["
negate => true
what => "previous"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
filter {
grok {
match=>["message","\[%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}\] %{GREEDYDATA:msg}"]
}
}
output{
stdout{
codec=>rubydebug
}
}
解析后结果：
{
"@timestamp" => "2016-06-01T06:33:26.426Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:01",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.485Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:02",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc"
}
{
"@timestamp" => "2016-06-01T06:33:26.491Z",
"message" => "[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:03",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.492Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:04",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.494Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:05",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc"
}
{
"@timestamp" => "2016-06-01T06:33:26.495Z",
"message" => "[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:06",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc"
}
参考资料
Logstash——multiline 插件，匹配多行日志的更多相关文章
- logstash之multiline插件，匹配多行日志
在外理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如log4j.运行时日志跟访问日志最大的不同是,运行时日志是多行,也就是说,连续的多行才能表达一个意思. 在filter中,加 ...
- Logstash——multiline 插件，匹配多行日志
本文内容 测试数据 字段属性 按多行解析运行时日志 把多行日志解析到字段 参考资料 在处理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如 log4j.运行时日志跟访问日志最大的 ...
- Logstash-安装logstash-filter-multiline插件(解决logstash匹配多行日志)
ELK-logstash在搬运日志的时候会出现多行日志,普通的搬运会造成保存到ES中日志一条一条的保存,很丑,而且不方便读取,logstash-filter-multiline可以解决该问题. 接下来 ...
- logstash匹配多行日志
在工作中,遇到一个问题就是日志的处理,首选的方案就是ELFK(filebeat+logstash+es+kibana) 因为之前使用过logstash采集日志的时候,非常的消耗系统的资源,所以这里我选 ...
- Python正则处理多行日志一例
正则表达式基础知识请参阅<正则表达式基础知识>,本文使用正则表达式来匹配多行日志并从中解析出相应的信息. 假设现在有这样的SQL日志: SELECT * FROM open_app WHE ...
- Python正则处理多行日志一例(可配置化)
正则表达式基础知识请参阅<正则表达式基础知识>,本文使用正则表达式来匹配多行日志并从中解析出相应的信息. 假设现在有这样的SQL日志: SELECT * FROM open_app WHE ...
- 写给大忙人的ELK最新版6.2.4学习笔记-Logstash和Filebeat解析(java异常堆栈下多行日志配置支持)
接前一篇CentOS 7下最新版(6.2.4)ELK+Filebeat+Log4j日志集成环境搭建完整指南,继续对ELK. logstash官方最新文档https://www.elastic.co/g ...
- ELK学习笔记之Logstash和Filebeat解析对java异常堆栈下多行日志配置支持
0x00 概述 logstash官方最新文档.假设有几十台服务器,每台服务器要监控系统日志syslog.tomcat日志.nginx日志.mysql日志等等,监控OOM.内存低下进程被kill.ngi ...
- logstash 安装插件multiline
一.安装multiline 在使用elk 传输记录 java 日志时,如下 一个java的报错 在elk中会按每一行 产生多条记录,不方便查阅 这里修改配置文件 使用 multiline 插件 ...
随机推荐
- 5.3 RDD编程---数据读写
一.文件数据读写 1.本地文件系统的数据读写 可以采用多种方式创建Pair RDD,其中一种主要方式是使用map()函数来实现 惰性机制,即使输入了错误的语句spark-shell也不会马上报错. ( ...
- 学习9：MongoDB知识
MongoDB学习笔记 1 基本介绍 基本概念 MongoDB**是一种面向文档的数据库管理系统,由C++语言编写的,是一个基于分布式文件存储的开源数据库系统.2007年10月,MongoDB由10g ...
- Java中用import导入类和用Class方法加载类有什么区别？
import仅仅包含导入操作,并不包含将字节码文件加载进内存这一动作,将字节码文件加载进内存是后续的实例化操作完成的. 例如通过import导入了一堆包和类,但是后续什么都没用(没用实例化),那么导入 ...
- python 一些程序规范（跨目录调用文件）
文档内容学习于 http://www.cnblogs.com/xiaozhiqi/ 参考: https://www.cnblogs.com/monologuesmw/p/9490093.html 软件 ...
- maven配置阿里云国内仓库
<mirror> <id>alimaven</id> <name>aliyun maven</name> <url>http:/ ...
- day27_8.6 网络编程7层协议
一.软件开发架构 在所有软件中有两种结构模式 1.c/s架构(client/server) c代表的是客户端 s代表的是服务端 2.b/s架构(browser/server) b代表的是浏览器 s代表 ...
- 继上篇博客对安卓爬虫以及TextView更新的问题解释
只能在主线程更新ui,或者用handler传输.爬虫是在另一个异步线程进行,所以爬出来的数据对控件赋值就比较麻烦.这次使用了线程数据传到runnable将数据传到handler进行赋值. 结果图: 源 ...
- UrlRouting原理笔记
UrlRouting路由流程: 添加路由:可以通过调用MapRoute().MapPageRoute(),它们内部都是创建Route对象,最终添加到RouteCollection中. 还可以使用[Ro ...
- 洛谷 P1209 [USACO1.3]修理牛棚 Barn Repair题解
题目 打死我也没想到是贪心 虽然是lkx写了贪心题解让我去点赞我才写的这道题 神仙思路 首先排好序 假设我们现在只有一块木板 我们做一个差分数组 对这个差分数组排序之后 一次断开最长的区间 m-1次之 ...
- sqlserver数据库使用navicat生成数据字典
https://blog.csdn.net/Honnyee/article/details/86156832 SELECT 表名 then d.name else '' end, 表说明 then i ...