logstash 抓取IIS日志文件写入Elasticsearch

如果需要对IIS日志进行分析可以使用logstash从文件中抓取出来进行分析；

输入部分：

input {
    file {
        type => "iis_log_monitor"
        path => ["D:/k/iislog/monitor*/W3SVC4/*.log"]
        start_position => "beginning"
        sincedb_path => "../config-demo/log/iis_log_monitor.log"
        sincedb_write_interval => 5
        discover_interval => 2
    }
    file {
        type => "iis_log_weixin"
        path => ["D:/k/iislog/weixin*/W3SVC18/*.log"]
        start_position => "beginning"
        sincedb_path => "../config-demo/log/iis_log_weixin.log"
        sincedb_write_interval => 5
        discover_interval => 2
    }
    file {
        type => "iis_log_imagedas"
        path => ["D:/k/iislog/imagedas/*.log"]
        start_position => "beginning"
        sincedb_path => "../config-demo/log/iis_log_imagedas.log"
        sincedb_write_interval => 5
        discover_interval => 2
    }
}

input中可以支持多个数据源的。

筛选部分：

filter{if [message] =~ "^#" {
        drop {}
    }
     grok {
            match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} (%{IPORHOST:s-ip}|-) (%{WORD:cs-method}|-) %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} (%{NUMBER:s-port}|-) (%{WORD:cs-username}|-) (%{IPORHOST:c-ip}|-) %{NOTSPACE:cs-useragent} (%{NUMBER:sc-status}|-) (%{NUMBER:sc-substatus}|-) (%{NUMBER:sc-win32-status}|-) (%{NUMBER:time-taken}|-)"]
    }
    date {
        match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
        timezone => "Asia/Shanghai"
    }
    useragent {
        source=> "cs-useragent"
    }
}

筛选的流程是：

1. 删除以“#”开头的记录、
2. 使用grok格式化日志
3. 使用日志的时间作为logstash的@timestamp
4. 解析出用户的ua信息

输出到es：

output{
    # stdout{
    #     codec => rubydebug
    # }
    elasticsearch {
        hosts => ["xxx.xxx.xxx.xxx:9200"]
        index => "iislog"
        document_type => "iisloginfo"
        workers => 1
        template => "../config-demo/templates/iislog.json"
        template_name => "iislog"
        template_overwrite => true
    }
}