Logstash——multiline 插件，匹配多行日志
本文内容
- 测试数据
- 字段属性
- 按多行解析运行时日志
- 把多行日志解析到字段
- 参考资料
在处理日志时，除了访问日志外，还要处理运行时日志，该日志大都用程序写的，比如 log4j。运行时日志跟访问日志最大的不同是，运行时日志是多行，也就是说，连续的多行才能表达一个意思。
本文主要说明，如何用 multiline 出来运行日志。
如果能按多行处理，那么把他们拆分到字段就很容易了。
迁移到：http://www.bdata-cap.com/newsinfo/1712113.html
测试数据
[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS asc
[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS desc
[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS asc
[16-04-12 03:40:07 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
测试是在7秒内发生的（当然是假数据）。可以看到，第二、五、六秒的日志是多行的，有条SQL语句。其他是单行的。
字段属性
对 multiline 插件来说，有三个设置比较重要：negate、pattern 和 what。
negate
类型是 boolean
默认为
false否定正则表达式（如果没有匹配的话）。
pattern
必须设置
类型为 string
没有默认值
要匹配的正则表达式。
what
必须设置
可以为
previous 或next没有默认值
如果正则表达式匹配了，那么该事件是属于下一个或是前一个事件？
按多行解析运行时日志
示例1：若配置文件如下所示，
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=> multiline {
pattern => "^\["
negate => true
what => "previous"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
output{
stdout{
codec=>rubydebug
}
}
说明：匹配以“[”开头的行，如果不是，那肯定是属于前一行的。
解析结果如下所示，能解析出6个JSON：
{
"@timestamp" => "2016-06-01T04:37:43.147Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.152Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.152Z",
"message" => "[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.155Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.157Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:37:43.159Z",
"message" => "[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
解析时，最后一行日志，不会解析。只有当再追加一条日志时，才会解析最后一条日志。
示例2：若将配置文件修改为，
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=>multiline {
pattern => "^\["
negate => true
what => "next"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
output{
stdout{
codec=>rubydebug
}
}
解析结果为，能解析出7个JSON：
{
"@timestamp" => "2016-06-01T04:40:43.232Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.237Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.238Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc\n[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.239Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.244Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.245Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc\n[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T04:40:43.249Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc\n[16-04-12 03:40:07 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
示例3：若将配置文件修改为，
codec=>multiline {
pattern => "^\["
negate => false
what => "previous"
}
则解析结果为：
{
"@timestamp" => "2016-06-01T05:38:50.853Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.856Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.858Z",
"message" => "order by product_category.ORDERS asc\n[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.860Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.861Z",
"message" => "order by product_category.ORDERS desc\n[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
{
"@timestamp" => "2016-06-01T05:38:50.863Z",
"message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog"
}
把多行日志解析到字段
配置文件如下所示：
input {
file{
path=>"/usr/local/elk/logstash/logs/c.out"
type=>"runtimelog"
codec=>multiline {
pattern => "^\["
negate => true
what => "previous"
}
start_position=>"beginning"
sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
ignore_older=>0
}
}
filter {
grok {
match=>["message","\[%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}\] %{GREEDYDATA:msg}"]
}
}
output{
stdout{
codec=>rubydebug
}
}
解析后结果：
{
"@timestamp" => "2016-06-01T06:33:26.426Z",
"message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:01",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.485Z",
"message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:02",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc"
}
{
"@timestamp" => "2016-06-01T06:33:26.491Z",
"message" => "[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:03",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.492Z",
"message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
"@version" => "1",
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:04",
"level" => "DEBUG",
"msg" => "model.MappingNode:- ['/store/shopclass'] matched over."
}
{
"@timestamp" => "2016-06-01T06:33:26.494Z",
"message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:05",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc"
}
{
"@timestamp" => "2016-06-01T06:33:26.495Z",
"message" => "[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "/usr/local/elk/logstash/logs/c.out",
"host" => "vcyber",
"type" => "runtimelog",
"timestamp" => "16-04-12 03:40:06",
"level" => "DEBUG",
"msg" => "impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc"
}
参考资料
Logstash——multiline 插件，匹配多行日志的更多相关文章
- logstash之multiline插件，匹配多行日志
在外理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如log4j.运行时日志跟访问日志最大的不同是,运行时日志是多行,也就是说,连续的多行才能表达一个意思. 在filter中,加 ...
- Logstash——multiline 插件，匹配多行日志
本文内容 测试数据 字段属性 按多行解析运行时日志 把多行日志解析到字段 参考资料 在处理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如 log4j.运行时日志跟访问日志最大的 ...
- Logstash-安装logstash-filter-multiline插件(解决logstash匹配多行日志)
ELK-logstash在搬运日志的时候会出现多行日志,普通的搬运会造成保存到ES中日志一条一条的保存,很丑,而且不方便读取,logstash-filter-multiline可以解决该问题. 接下来 ...
- logstash匹配多行日志
在工作中,遇到一个问题就是日志的处理,首选的方案就是ELFK(filebeat+logstash+es+kibana) 因为之前使用过logstash采集日志的时候,非常的消耗系统的资源,所以这里我选 ...
- Python正则处理多行日志一例
正则表达式基础知识请参阅<正则表达式基础知识>,本文使用正则表达式来匹配多行日志并从中解析出相应的信息. 假设现在有这样的SQL日志: SELECT * FROM open_app WHE ...
- Python正则处理多行日志一例(可配置化)
正则表达式基础知识请参阅<正则表达式基础知识>,本文使用正则表达式来匹配多行日志并从中解析出相应的信息. 假设现在有这样的SQL日志: SELECT * FROM open_app WHE ...
- 写给大忙人的ELK最新版6.2.4学习笔记-Logstash和Filebeat解析(java异常堆栈下多行日志配置支持)
接前一篇CentOS 7下最新版(6.2.4)ELK+Filebeat+Log4j日志集成环境搭建完整指南,继续对ELK. logstash官方最新文档https://www.elastic.co/g ...
- ELK学习笔记之Logstash和Filebeat解析对java异常堆栈下多行日志配置支持
0x00 概述 logstash官方最新文档.假设有几十台服务器,每台服务器要监控系统日志syslog.tomcat日志.nginx日志.mysql日志等等,监控OOM.内存低下进程被kill.ngi ...
- logstash 安装插件multiline
一.安装multiline 在使用elk 传输记录 java 日志时,如下 一个java的报错 在elk中会按每一行 产生多条记录,不方便查阅 这里修改配置文件 使用 multiline 插件 ...
随机推荐
- LeetCode OJ-- Wildcard Matching **@
https://oj.leetcode.com/problems/wildcard-matching/ 模拟通配符的匹配 做法非常好 class Solution { public: bool isM ...
- Learning OpenCV
1. 读取图片 opencv/highgui.h 2. 读取视频 opencv/cv.h opencv/highgui.h 3. 高斯平滑滤波 4. 灰度单通道与边缘检测 5. 摄像头打开 void ...
- HTML回顾
<frameset>和<body>是同一级的,已经在html5中被弃用 结合----> 效果 注意::::span标签不自动换行****
- Log4j 配置数据库连接池（将日志信息保存到数据库）
org.apache.log4j.jdbc.JDBCAppender 是利用传统的 JDBC 连接方法,这种方式连接数据库效率低下,为了解决这个问题,现在自定义一个 Log4j 的 Appender, ...
- 使用 sqlcmd 运行 Transact-SQL 脚本文件
在数据恢复时遇到问题:bat文件批处理的指导文档 https://msdn.microsoft.com/zh-cn/library/ms170572%28v=sql.120%29.aspx
- MMM互助金融/理财源码
1.1.1MMM互助金融配比系统源码销售 (3mphp.com/mmm-office.com) 联系QQ: 3375551869,全套源码,包含: 1 源码:安装.开发文档 2 数据库:含演示数据,自 ...
- PC-PC-单片机(Arduino)通信实例
请仔细理解相关参数,如端口设置.IP设置.COM口设置......等等.....不要盲目COPY.....这涉及2台电脑和一个单片机,其中一台电脑作为服务器并与单片机相连,负责通过网络与客户端通信(s ...
- [ACM_模拟] POJ 1094 Sorting It All Out (拓扑排序+Floyd算法 判断关系是否矛盾或统一）
Description An ascending sorted sequence of distinct values is one in which some form of a less-than ...
- [matlab] MATLAB 界面编程 傻瓜教程
>_<:在 MATLAB 的命令窗口(Command Window)中运行 guide 命令,来打开 GUIDE 界面,如下: >_<:然后,选择空模板(Blang GUI), ...
- 一句话在网页右上角加一个精致下拉框：forkme on github
随着我国科技水平不断发展,玩Github的童鞋越来越多了,按照惯例,开源项目会有一个示例网站,而网站的右上角,通常会有一个forkme on github,这说明你可以去Github查看.下载项目源码 ...