logstash 安装插件multiline

一、安装multiline

在使用elk 传输记录 java 日志时，如下

一个java的报错

在elk中会按每一行 产生多条记录，不方便查阅

这里修改配置文件 使用  multiline   插件 即可实现多行合一的 输出模式

修改配置文件

# vi  /etc/logstash/conf.d/logstash.conf  

input {
 file {
    path => "/w_logs/error.log.2018-06-05"
    type => "test"
 }
}

filter {
   multiline {
            pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}"
            negate => true
            what => "previous"
        }

   grok {
       match => [ "message", "%{NOTSPACE:day} %{NOTSPACE:datetime}  %{NOTSPACE:level} %{GREEDYDATA:msginfo} " ]
   }
}

output {
 if [type] == "test" {
        elasticsearch {
            hosts => ["10.10.15.95:9200"]
            index => "12.83-test"
        }
  }
}

修改完 重启logstash
报错：

[ERROR] -- ::59.834 [Ruby--Thread-: /usr/share/logstash/vendor/bundle/jruby/2.3./gems/stud-0.0./lib/stud/task.rb:] registry - Tried to load a plugin's code, but failed.
 {:exception=>#<LoadError: no such file to load -- logstash/filters/multiline>, :path=>"logstash/filters/multiline", :type=>"filter", :name=>"multiline"}

[ERROR] -- ::59.838 [Ruby--Thread-: /usr/share/logstash/vendor/bundle/jruby/2.3./gems/stud-0.0./lib/stud/task.rb:] agent - 
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any filter plugin named 'multiline'. Are you sure this is correct? Trying to load the multiline filter plugin resulted in this error: no such file to load -- logstash/filters/multiline", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:192:in `lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:in `lookup'", "/usr/share/logstash/logstash-core/lib/logs

提示缺少 插件 filters/multiline

我们看看logstash都安装了哪些插件

# /usr/share/logstash/bin/logstash-plugin list
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-jdbc_static
logstash-filter-jdbc_streaming
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-xml
logstash-input-beats
logstash-input-dead_letter_queue
logstash-input-elasticsearch
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-jdbc
logstash-input-kafka
logstash-input-pipe
logstash-input-rabbitmq
logstash-input-redis
logstash-input-s3
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-file
logstash-output-graphite
logstash-output-http
logstash-output-kafka
logstash-output-lumberjack
logstash-output-nagios
logstash-output-null
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-patterns-core

有一个logstash-codec-multiline
并没有我们需要的  logstash-filter-multiline

我们来安装这个插件，先看一下  logstash-plugin 的用法

Usage:
    bin/logstash-plugin [OPTIONS] SUBCOMMAND [ARG] ...

Parameters:
    SUBCOMMAND                    subcommand
    [ARG] ...                     subcommand arguments

Subcommands:
    list                          List all installed Logstash plugins
    install                       Install a Logstash plugin
    remove                        Remove a Logstash plugin
    update                        Update a plugin
    pack                          Package currently installed plugins, Deprecated: Please use prepare-offline-pack instead
    unpack                        Unpack packaged plugins, Deprecated: Please use prepare-offline-pack instead
    generate                      Create the foundation for a new plugin
    uninstall                     Uninstall a plugin. Deprecated: Please use remove instead
    prepare-offline-pack          Create an archive of specified plugins to use for offline installation

Options:
    -h, --help                    print help

安装插件是  # logstash-plugin install logstash-filter-multiline

# logstash-plugin install logstash-filter-multiline
Validating logstash-filter-multiline
Installing logstash-filter-multiline
Installation successfu

二、multiline 使用方法

codec =>multiline {
     charset=>...          #可选                    字符编码
     max_bytes=>...        #可选     bytes类型       设置最大的字节数
     max_lines=>...        #可选     number类型      设置最大的行数,默认是500行
     multiline_tag...      #可选     string类型      设置一个事件标签,默认是multiline
     pattern=>...          #必选     string类型      设置匹配的正则表达式
     patterns_dir=>...     #可选     array类型       可以设置多个正则表达式
     negate=>...           #可选     boolean类型     默认false不显示，可设置ture
     what=>...             #必选                    向前previous ， 向后 next

 }

## negate  只支持布尔值，true 或者false，默认为false。

如果设置为true，表示与正则表达式（pattern）不匹配的内容都需要整合，
具体整合在前还是在后，看what参数。如果设置为false，即与pattern匹配的内容

## what  前一行 或者后一行，指出上面对应的规则与前一行内容收集为一行，还是与后一行整合在一起

简单来说：

negate默认是 false，不显示
与patten匹配的行
由what决定 向前或向后 匹配

negate 设置为true
则与patten 不匹配的行
由what决定 向前或向后 匹配