Logstash grok解析IIS 10.0 日志实例

Logstash解析IIS日志的核心在于logstash配置文件

IIS 日志位置和格式如下：

#Software: Microsoft Internet Information Services 10.0
#Version: 1.0
#Date: 2018-07-11 08:20:39
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2018-07-11 08:20:39 10.1.2.4 GET / - 80 - 106.75.92.187 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:52.0)+Gecko/20100101+Firefox/52.0 - 200 0 0 296
2018-07-11 08:24:14 10.1.2.4 GET / - 80 - 167.220.255.62 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/67.0.3396.99+Safari/537.36 - 200 0 0 962
2018-07-11 08:24:14 10.1.2.4 GET /favicon.ico - 80 - 167.220.255.62 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/67.0.3396.99+Safari/537.36 http://40.73.97.180/ 404 0 2 166
2018-07-11 08:25:29 10.1.2.4 GET / - 80 - 40.73.101.81 - - 200 0 0 15
2018-07-11 08:25:51 10.1.2.4 GET / - 80 - 40.73.101.81 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/67.0.3396.99+Safari/537.36 - 200 0 0 0
2018-07-11 08:26:29 10.1.2.4 GET / - 80 - 40.73.101.81 - - 200 0 0 15
2018-07-11 08:26:35 10.1.2.4 GET / - 80 - 40.73.101.81 PostmanRuntime/7.1.5 - 200 0 0 15

logstash配置文件如下

input {
  file {
    type => "iis-w3c"
    path => "C:/inetpub/logs/LogFiles/W3SVC*/*.log"
  }
}

filter {
    # ignore log comments
    if [message] =~ "^#" {
        drop {}
    }
     # check that fields match your IIS log settings
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:s-ip} %{WORD:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} %{NOTSPACE:UserAgent} %{NOTSPACE:Referer} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken:int}"]
    }

    # set the event timestamp from the log
    # https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html
    date {
        match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
        target => "log-timestamp"
    }

    # matches the big, long nasty useragent string to the actual browser name, version, etc
    # https://www.elastic.co/guide/en/logstash/current/plugins-filters-useragent.html
    useragent {
        source=> "UserAgent"
        prefix=> "browser_"
    }
    mutate {
        remove_field => [ "log_timestamp"]
    }
}

output {
    elasticsearch{
        hosts => ["10.1.1.4:9200"]
        index => "logstash-iisw3c-%{+YYYY.MM.dd}"
      }
    stdout {codec => rubydebug}
}

从Elasticsearch看解析后的IIS日志

IIS日志的各种信息，包括浏览器信息等，都被解析出来