logstash 正则调试；

nginx 配置；

log_format  main  '$remote_addr [$time_local] "$request" ';

logstash:

 "message" =>"%{IPORHOST:clientip} \[%{HTTPDATE:time}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\""

 输出:

 {

        "message" => " 121.40.205.143 [29/Aug/2016:12:36:32 +0800] \"GET /favicon.ico HTTP/1.1\" - 404 2319 \"-\" \"Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN\" 0.000 -",

       "@version" => "1",

     "@timestamp" => "2016-08-29T04:39:16.608Z",

           "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",

           "host" => "0.0.0.0",

           "type" => "uat_nginx_access",

       "clientip" => "121.40.205.143",

           "time" => "29/Aug/2016:12:36:32 +0800",

           "verb" => "GET",

        "request" => "/favicon.ico",

    "httpversion" => "1.1"

}

此时grok 能正常匹配:

            "message" => "%{IPORHOST:clientip} \[%{HTTPDATE:time}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\"

			%{NUMBER:http_status_code} %{NUMBER:bytes} \"(?<http_referer>\S+)\" \"(?<http_user_agent>\S+)\" \"(?<http_x_forwarded_for>\S+)\""

			    log_format  main  '$http_host $server_addr $remote_addr [$time_local] "$request" '

                      '$request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" '

                      '$request_time $upstream_response_time';

继续加；

    log_format  main  '$remote_addr [$time_local] "$request"'

                       '$status $body_bytes_sent';

日志格式:

 121.40.205.143 [29/Aug/2016:12:51:18 +0800] "GET /resources/plugins/artDialog/ui-dialog.css HTTP/1.1"304 0

 121.40.205.143 [29/Aug/2016:12:51:18 +0800] "GET /wechat/images/account/icons.7a340e21.png HTTP/1.1"304 0

 121.40.205.143 [29/Aug/2016:12:51:18 +0800] "GET /wechat/images/nav-icon.44c2022c.png?v=1 HTTP/1.1"304 0

 121.40.205.143 [29/Aug/2016:12:51:19 +0800] "GET /favicon.ico HTTP/1.1"404 2319

 121.40.205.143 [29/Aug/2016:12:51:19 +0800] "GET /favicon.ico HTTP/1.1"404 2319

 121.40.205.143 [29/Aug/2016:12:52:25 +0800] "GET /favicon.ico HTTP/1.1"404 2319

 121.40.205.143 [29/Aug/2016:12:52:25 +0800] "GET /favicon.ico HTTP/1.1"404 2319

 121.40.205.143 [29/Aug/2016:12:53:28 +0800] "GET /favicon.ico HTTP/1.1"404 2319

filter {

    grok {

        match=> {

       "message" =>"%{IPORHOST:clientip} \[%{HTTPDATE:time}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\"%{NUMBER:http_status_code} %{NUMBER:bytes}"

        }

logstash 输出:

Pipeline main started

{

             "message" => " 121.40.205.143 [29/Aug/2016:12:56:10 +0800] \"GET /favicon.ico HTTP/1.1\"404 2319",

            "@version" => "1",

          "@timestamp" => "2016-08-29T04:58:54.908Z",

                "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",

                "host" => "0.0.0.0",

                "type" => "uat_nginx_access",

            "clientip" => "121.40.205.143",

                "time" => "29/Aug/2016:12:56:10 +0800",

                "verb" => "GET",

             "request" => "/favicon.ico",

         "httpversion" => "1.1",

    "http_status_code" => "404",

               "bytes" => "2319"

}					   

继续；

 121.40.205.143 [29/Aug/2016:13:00:16 +0800] "GET /favicon.ico HTTP/1.1"404 2319 "-"

 121.40.205.143 [29/Aug/2016:13:00:22 +0800] "GET /favicon.ico HTTP/1.1"404 2319 "-"

 121.40.205.143 [29/Aug/2016:13:00:30 +0800] "GET /favicon.ico HTTP/1.1"404 2319 "-"

 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/login.html HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/account.html"

 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/images/login/icon_01.6e839367.png HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/css/wechat.2a00a782.css"

 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/images/login/icon_02.5065faba.png HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/css/wechat.2a00a782.css"

 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /resources/plugins/jquery/jquery.md5.js?v=1 HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/login.html"

 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/js/libs/dialog-min.88247f5e.js?v=1 HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/login.html"

 121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/js/login.a87fbd64.js HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/login.html"

{

       "message" => " 121.40.205.143 [29/Aug/2016:13:05:24 +0800] \"GET /wechat/account_balance.html HTTP/1.1\"200 3059 \"https://uatest.winfae.com/wechat/account.html\"

	   \"Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN\"",

121.40.205.143 [29/Aug/2016:13:05:24 +0800] "GET /wechat/account_balance.html HTTP/1.1"200 3059 "https://uatest.winfae.com/wechat/account.html" "Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN"

121.40.205.143 [29/Aug/2016:13:05:45 +0800] "GET /wechat/home.html?useragent=android_h5_zjcap&apiver=2 HTTP/1.1"200 11601 "-" "okhttp/2.6.0"

{

             "message" => " 121.40.205.143 [29/Aug/2016:13:13:11 +0800] \"GET /wechat/js/regain.431efde9.js HTTP/1.1\"304 0 \"https://uatest.winfae.com/wechat/regain.html\" \"Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN\"",

            "@version" => "1",

          "@timestamp" => "2016-08-29T05:15:55.609Z",

                "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",

                "host" => "0.0.0.0",

                "type" => "uat_nginx_access",

            "clientip" => "121.40.205.143",

                "time" => "29/Aug/2016:13:13:11 +0800",

                "verb" => "GET",

             "request" => "/wechat/js/regain.431efde9.js",

         "httpversion" => "1.1",

    "http_status_code" => "304",

               "bytes" => "0",

        "http_referer" => "https://uatest.winfae.com/wechat/regain.html"

\S+ 和 [^\n\t\r\f]+ 语法一样   非空格

 my $str="  begin 123.456 end  ";

if ($str =~/(?<request_time>\d+\.\d+)/)

   {

    my ($request_time) = ($+{request_time});

   print $request_time."\n";};

zjtest7-frontend:/root/0825# perl a1.pl

123.456  

 "http_referer" => "https://uatest.winfae.com/wechat/regain.html"

  \"(?<http_referer>\S+)\"

 \"(?<http_user_agent>\S+)\"

 \"Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN\"

              "message" => " 121.40.205.143 [29/Aug/2016:13:54:08 +0800] \"GET /resources/plugins/artDialog/ui-dialog.css HTTP/1.1\"200 9985 \"https://uatest.winfae.com/wechat/home.html?useragent=ios_h5_zjcap&apiver=2&WKWebView=1\" \"ios_h5_zjcap\"",

            "@version" => "1",

          "@timestamp" => "2016-08-29T05:56:53.217Z",

                "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",

                "host" => "0.0.0.0",

                "type" => "uat_nginx_access",

            "clientip" => "121.40.205.143",

                "time" => "29/Aug/2016:13:54:08 +0800",

                "verb" => "GET",

             "request" => "/resources/plugins/artDialog/ui-dialog.css",

         "httpversion" => "1.1",

    "http_status_code" => "200",

               "bytes" => "9985",

        "http_referer" => "https://uatest.winfae.com/wechat/home.html?useragent=ios_h5_zjcap&apiver=2&WKWebView=1",

     "http_user_agent" => "ios_h5_zjcap"

}

{

       "message" => " 121.40.205.143 [29/Aug/2016:13:59:35 +0800] \"GET /resources/js/toolbar.49fc367e.js?_v=${last.updated}&_=1472450673142 HTTP/1.1\"200 1800 \"https://uatest.winfae.com/products/productList.html\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36\"",

      "@version" => "1",

    "@timestamp" => "2016-08-29T06:02:18.775Z",

          "path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",

          "host" => "0.0.0.0",

          "type" => "uat_nginx_access",

          "tags" => [

        [0] "_grokparsefailure"

    ]

}

logstash grok正则调试的更多相关文章

  1. ELK——为调试 Logstash Grok 表达式，安装 GrokDebuger 环境

      内容 安装 RVM 安装 Ruby 和 Gems 安装 Rails 安装 jls-grok Ruby grok 解析 调试 grok 注意:不要用 root 执行以下操作. 用 logstash ...

  2. grok 正则解析日志例子<1>

    <pre name="code" class="html">下面是日志的样子 55.3.244.1 GET /index.html 15824 0. ...

  3. logstash grok内置规则

    logstash grok 内置正则 https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns USERNAME [ ...

  4. [elk]logstash grok原理

    logstash语法 http://www.ttlsa.com/elk/elk-logstash-configuration-syntax/ https://www.elastic.co/guide/ ...

  5. logstash -grok插件语法介绍

      介绍 logstash拥有丰富的filter插件,它们扩展了进入过滤器的原始数据,进行复杂的逻辑处理,甚至可以无中生有的添加新的 logstash 事件到后续的流程中去!Grok 是 Logsta ...

  6. Grok 正则捕获

    Grok 正则捕获: \s+(?<request_time>\d+(?:\.\d+)?)\s+ 回顾下: (?:pattern) 匹 配 pattern 但不获取匹配结果,也就是说这是一个 ...

  7. grok 正则捕获(就是perl的正则捕获)

    2.3.2 grok 正则捕获: \s+ 和 [\n\t\r\f]+ 一样 1.命名分组格式为(?<grp name>),反向引用时用\k<grp name> 2.命名分组的匹 ...

  8. logstash 使用grok正则解析日志

    http://xiaorui.cc/2015/01/27/logstash%E4%BD%BF%E7%94%A8grok%E6%AD%A3%E5%88%99%E8%A7%A3%E6%9E%90%E6%9 ...

  9. logstash的grok正则匹配规则文件

    文件路径:logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.1.0/patterns/grok-patterns 在线调试g ...

随机推荐

  1. Qt 文件搜索（写入文本文件）

    代码无意间在网上找到,下载回来后改了几个格式,编译后经测试可以正常使用,这个个文件搜索的很好的例子,有两种搜索方式:一种是按文件名作为关键字进行搜索,一种是以文档中所包含的关键字进行搜索,贴两张图先: ...

  2. XmlDocument，XDocument相互转换

    XmlDocument,XDocument相互转换 using System; using System.Xml; using System.Xml.Linq; namespace MyTest { ...

  3. 2、elasticsearch 的安装和插件的安装

    1.安装Elasticsearch集群 1.下载elasticsearch-2.0.0.tar.gz,执行tar -zxvf elasticsearch-2.0.0.tar.gz解压 2.修改conf ...

  4. Mac 下纯lua（三）

    文件处理 直接使用io调用 io.close();文件流关闭 io.flush():如果文件流以bufferd缓存模式处理,输入不会立即存入文件,需要调用本函数 io.input(file):输入 i ...

  5. SQL Server2005使用CTE实现递归

    本文来自:http://www.cnblogs.com/wenjl520/archive/2010/01/18/1650393.html CTE递归原理: 递归CTE是由两个最小查询构建的.第一个是定 ...

  6. ChromiumFX中js调用C#方法

    server端代码: ChromiumWebBrowser wb; wb.AddGlobalJSFunction("CfxHelloWorld").Execute += CfxHe ...

  7. 玩转iOS开发 - JSON 和 Xml 数据解析

    前言 Json 和xml是网络开发中经常使用的数据格式,JSON轻量级.xml相对较复杂.所以如今用JSON的比例很大.基本上从server获取的返回数据都是JSON格式的,作为iOS开发人员,解析J ...

  8. 用数据说话,外贸B2C产品选择(上篇)-热门搜索法

    当选择了外贸这条路,那就是选择了跟外国人做生意.那面对全球这么大的市场究竟选什么样的产品才干脱颖而出?什么样的产品才是全球卖家喜欢的呢?什么样的产品才干让自己財源滚滚?我想这都是全部刚開始外贸创业的人 ...

  9. Android实战技巧之十九：android studio导出jar包（Module）并获得手机信息

    AS中并没有独立的Module 工程,可是能够在普通的Project中增加Module.所谓的Module就是我们通常所指的模块化的一个单元.并经常以jar包的形式存在.以下以一个获取手机信息的样例演 ...

  10. a:hover span 隐藏/显示 问题

    :hover是我们在CSS设计中最常运用的伪类之一,许多绚丽效果的实现离不开伪类:hover,比如我们常见的纯CSS菜单.相册效果等等. 或许用了这么久的伪类:hover,还有部分朋友还不完全了解ho ...