logstahs 匹配isslog

2016-11-30 06:33:33 192.168.5.116 GET /Hotel/HotelDisplay/cncqcqb230 - 80 - 192.168.9.2 Mozilla/5.0+(Macintosh;+U;+Intel+Mac+OS+X+10.9;+en-US;+rv:1.9pre)+Gecko - 200 0 0 45

\s*(?<time>([0-9]{4}\-[0-9]{2}\-[0-9]{2}\s+[0-9]{2}:[0-9]{2}:[0-9]{2}))\s+%{IPORHOST:clientip}\s+%{WORD:verb}\s+%{URIPATHPARAM:request}\s+\-\s+(?<port>([0-9]{2}.*?))\s+\-\s+%{IPORHOST:sourceip}\s+(?<http_user_agent>(\S+\s+).*?).*

{
  "time": [
    [
      "2016-11-30 06:33:33"
    ]
  ],
  "clientip": [
    [
      "192.168.5.116"
    ]
  ],
  "verb": [
    [
      "GET"
    ]
  ],
  "request": [
    [
      "/Hotel/HotelDisplay/cncqcqb230"
    ]
  ],
  "port": [
    [
      "80"
    ]
  ],
  "sourceip": [
    [
      "192.168.9.2"
    ]
  ],
  "http_user_agent": [
    [
      "Mozilla/5.0+(Macintosh;+U;+Intel+Mac+OS+X+10.9;+en-US;+rv:1.9pre)+Gecko "
    ]
  ]
}

logstash 配置:
input {
    stdin {
    }
}
filter {
    grok {
        match => [
             "message" ,"\s*(?<time>([0-9]{4}\-[0-9]{2}\-[0-9]{2}\s+[0-9]{2}:[0-9]{2}:[0-9]{2}))\s+%{IPORHOST:clientip}\s+%{WORD:verb}\s+%{URIPATHPARAM:request}\s+\-\s+(?<port>([0-9]{2}.*?))\s+\-\s+%{IPORHOST:sourceip}\s+(?<http_user_agent>(\S+\s+).*?).*"
                ]
       }
   #      date {
   #     match => ["time", "HH:mm:ss"]
   # }
}
output {
 stdout {
                        codec => rubydebug
                } 

}

此时输出:

[elk@Vsftp gw]$ ../../bin/logstash -f gw.conf
Settings: Default pipeline workers: 4
Pipeline main started
2016-11-30 06:33:33 192.168.5.116 GET /Hotel/HotelDisplay/cncqcqb230 - 80 - 192.168.9.2 Mozilla/5.0+(Macintosh;+U;+Intel+Mac+OS+X+10.9;+en-US;+rv:1.9pre)+Gecko - 200 0 0 45
{
            "message" => "2016-11-30 06:33:33 192.168.5.116 GET /Hotel/HotelDisplay/cncqcqb230 - 80 - 192.168.9.2 Mozilla/5.0+(Macintosh;+U;+Intel+Mac+OS+X+10.9;+en-US;+rv:1.9pre)+Gecko - 200 0 0 45",
           "@version" => "1",
         "@timestamp" => "2016-11-30T07:15:13.887Z",
               "host" => "Vsftp",
               "time" => "2016-11-30 06:33:33",
           "clientip" => "192.168.5.116",
               "verb" => "GET",
            "request" => "/Hotel/HotelDisplay/cncqcqb230",
               "port" => "80",
           "sourceip" => "192.168.9.2",
    "http_user_agent" => "Mozilla/5.0+(Macintosh;+U;+Intel+Mac+OS+X+10.9;+en-US;+rv:1.9pre)+Gecko "
}

当前时间为 15:16

配置date插件:

[elk@Vsftp gw]$ cat gw.conf
input {
    stdin {
    }
}
filter {
    grok {
        match => [
             "message" ,"\s*(?<time>([0-9]{4}\-[0-9]{2}\-[0-9]{2}\s+[0-9]{2}:[0-9]{2}:[0-9]{2}))\s+%{IPORHOST:clientip}\s+%{WORD:verb}\s+%{URIPATHPARAM:request}\s+\-\s+(?<port>([0-9]{2}.*?))\s+\-\s+%{IPORHOST:sourceip}\s+(?<http_user_agent>(\S+\s+).*?).*"
                ]
       }
         date {
        match => ["time", "yyyy-MM-dd HH:mm:ss"]
    }
}
output {
 stdout {
                        codec => rubydebug
                } 

}
[elk@Vsftp gw]$ ../../bin/logstash -f gw.conf
Settings: Default pipeline workers: 4
Pipeline main started
2016-11-30 06:33:33 192.168.5.116 GET /Hotel/HotelDisplay/cncqcqb230 - 80 - 192.168.9.2 Mozilla/5.0+(Macintosh;+U;+Intel+Mac+OS+X+10.9;+en-US;+rv:1.9pre)+Gecko - 200 0 0 45
{
            "message" => "2016-11-30 06:33:33 192.168.5.116 GET /Hotel/HotelDisplay/cncqcqb230 - 80 - 192.168.9.2 Mozilla/5.0+(Macintosh;+U;+Intel+Mac+OS+X+10.9;+en-US;+rv:1.9pre)+Gecko - 200 0 0 45",
           "@version" => "1",
         "@timestamp" => "2016-11-29T22:33:33.000Z",
               "host" => "Vsftp",
               "time" => "2016-11-30 06:33:33",
           "clientip" => "192.168.5.116",
               "verb" => "GET",
            "request" => "/Hotel/HotelDisplay/cncqcqb230",
               "port" => "80",
           "sourceip" => "192.168.9.2",
    "http_user_agent" => "Mozilla/5.0+(Macintosh;+U;+Intel+Mac+OS+X+10.9;+en-US;+rv:1.9pre)+Gecko "
}

{
            "message" => "2016-11-30 06:33:33 192.168.5.116 GET /Hotel/HotelDisplay/cncqcqb230 - 80 - 192.168.9.2 Mozilla/5.0+(Macintosh;+U;+Intel+Mac+OS+X+10.9;+en-US;+rv:1.9pre)+Gecko - 200 0 0 45",
           "@version" => "1",
         "@timestamp" => "2016-11-30T07:15:13.887Z",
               "host" => "Vsftp",
               "time" => "2016-11-30 06:33:33",

{
            "message" => "2016-11-30 06:33:33 192.168.5.116 GET /Hotel/HotelDisplay/cncqcqb230 - 80 - 192.168.9.2 Mozilla/5.0+(Macintosh;+U;+Intel+Mac+OS+X+10.9;+en-US;+rv:1.9pre)+Gecko - 200 0 0 45",
           "@version" => "1",
         "@timestamp" => "2016-11-29T22:33:33.000Z",
               "host" => "Vsftp",
               "time" => "2016-11-30 06:33:33",

坑爹 nxlog 收到的日志里记录的时间本来就是 UTC时间,在转换一次 -8个小时

正常时间  06:33 表示 14:33  

这时候06:33 在减去8  22:33:33