nginx-access.log的logstash解析

1、nginx的access的格式：

log_format hehe '$http_host $server_addr $remote_addr [$time_local] "$request" $request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time $upstream_response_time $upstream_addr';

2、nginx日志

evo.xinpinget.com 10.1.217.143 6.52.65.86 [22/May/2018:15:10:46 +0800] "GET /api/merchant/listForSelection?filterByUserSelector=true&token=1cmqfl2RiO9BV9xIU8F3LG1I HTTP/1.1" - 200 268 "http://evo.xinpinget.com/order/newAfterSales" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36 QQBrowser/4.1.4132.400" 0.021 0.021 10.1.132.201:4010

3、grok

%{IPORHOST:http_host} %{IPORHOST:server_addr} %{IPORHOST:remote_addr} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATH:uri}\?%{NOTSPACE:params} HTTP/%{NUMBER:http_version}" %{NOTSPACE:request_body|-} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{NUMBER:request_time:float} %{NUMBER:upstream_response_time:float} %{IPORHOST:upstream_addr}

4、logstash

patterns：

# cat nginx
HOSTPORT1 (%{IPV4}:%{POSINT}[, ]{0,2})+
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:server_addr} %{IPORHOST:remote_addr} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATH:uri}\?%{NOTSPACE:params} HTTP/%{NUMBER:http_version}" %{NOTSPACE:request_body|-} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{NUMBER:request_time:float} %{NUMBER:upstream_response_time:float} %{IPORHOST:upstream_addr}

logstash.conf

filter{

if 'access' in [tags] {
grok {
patterns_dir => "/etc/logstash/patters.d"
match =>{ "message" => "%{NGINXACCESS}"
overwrite => [ "message"]
}
}
kv {
source => "params"
field_split => "&?"
value_split => "="
include_keys => [ "token", "user", "from", "channelId" ]
}
}

}