elk logstash Managing Multiline Events

1.Java程序的日志特征，logstash 正为此准备好了 codec/multiline 插件！

有时候应用程序会抛异常，就存在着如何合并多行信息的问题，我这里做的配置就是如果当前行是以‘空格’，‘字母‘ 和 ‘-’开头的，那么就直接合并到上

[2019-06-26 11:59:42,758[ INFO ThreadPool 任务运行队列 thread:2](cn.com.to1.common.dac.SupportDAC:575) prepared的查询语句是：SELECT COUNT(DISTINCTr.user_id) FROM TB_DEPARTMENT_INFO d, tb_qy_user_department_ref r where r.department_id = d.id and d.org_id = :orgId AND (d.dept_full_name LIKE :deptFullNameLike or d.dept_full_name = :deptFullName)
[2019-06-26 11:59:42,763[ INFO ThreadPool 任务运行队列 thread:8](cn.com.to1.common.dac.SupportDAC:575) prepared的查询语句是：insert into TB_QY_USER_DEPARTMENT_REF(ID,USER_ID,DEPARTMENT_ID,SORT,ORG_ID) values(:id,:userId,:departmentId,:sort,:orgId)
2019-06-26 11:59:42,764[ERROR DefaultQuartzScheduler_Worker-10](cn.com.to1.component.util.EmailUtil:225) EmailUtil sendWarnEmail error
javax.mail.AuthenticationFailedException
        at javax.mail.Service.connect(Service.java:319)
        at javax.mail.Service.connect(Service.java:169)
        at javax.mail.Service.connect(Service.java:118)
        at javax.mail.Transport.send0(Transport.java:188)
        at javax.mail.Transport.send(Transport.java:118)
        at cn.com.do1.component.util.EmailUtil.send(EmailUtil.java:191)
        at cn.com.do1.component.util.EmailUtil.sendWarnEmail(EmailUtil.java:217)
        at cn.com.do1.component.util.ThreadPoolUtils.warn(ThreadPoolUtils.java:278)
        at cn.com.do1.component.util.ThreadPoolUtils.execute(ThreadPoolUtils.java:127)
        at cn.com.do1.component.runtask.thread.RunTaskJob.runTask(RunTaskJob.java:63)
        at cn.com.do1.component.runtask.thread.RunTaskJob.execute(RunTaskJob.java:52)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
[2019-06-26 11:59:42,764[ INFO ThreadPool 任务运行队列 thread:8](cn.com.do1.common.dac.SupportDAC:575) prepared的查询语句是：insert into TB_QY_USER_INFO(ID,USER_ID,ORG_ID,PERSON_NAME,PINYIN,SEX,UPDATE_TIME,USER_STATUS,HEAD_PIC,CREATE_TIME,IS_CONCERNED,WX_USER_ID,CORP_ID) values(:id,:userId,:orgId,:personName,:pinyin,:sex,:updateTime,:userStatus,:headPic,:createTime,:isConcerned,:wxUserId,:corpId)

2.logstash的配置关键

合并多行数据(Multiline)

demo

[Aug/08/08 14:54:03] hello world
[Aug/08/09 14:54:04] hello logstash
    hello best practice
    hello raochenlin
[Aug/08/10 14:54:05] the end

logstash的关键配置

[root@VM_0_92_centos opt]# cat /usr/local/logstash/config/exception.conf
input {
  file {
    path => "/opt/50910626.log"
    codec => multiline {
      pattern => "^\["
      negate => true
      what => "previous"
    }
    start_position => "beginning"
  }
}
output {
  elasticsearch { hosts => ["10.0.0.92:9200"]
                  index => "linemuti"
  }
  stdout { codec => rubydebug }
}
说明：匹配以“[”开头的行，如果不是，那肯定是属于前一行的。

参考：https://doc.yonyoucloud.com/doc/logstash-best-practice-cn/codec/multiline.html