logstash grok正则调试
logstash 正则调试；
nginx 配置；
log_format main '$remote_addr [$time_local] "$request" '; logstash:
"message" =>"%{IPORHOST:clientip} \[%{HTTPDATE:time}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\"" 输出: {
"message" => " 121.40.205.143 [29/Aug/2016:12:36:32 +0800] \"GET /favicon.ico HTTP/1.1\" - 404 2319 \"-\" \"Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN\" 0.000 -",
"@version" => "1",
"@timestamp" => "2016-08-29T04:39:16.608Z",
"path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
"host" => "0.0.0.0",
"type" => "uat_nginx_access",
"clientip" => "121.40.205.143",
"time" => "29/Aug/2016:12:36:32 +0800",
"verb" => "GET",
"request" => "/favicon.ico",
"httpversion" => "1.1"
} 此时grok 能正常匹配: "message" => "%{IPORHOST:clientip} \[%{HTTPDATE:time}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\"
%{NUMBER:http_status_code} %{NUMBER:bytes} \"(?<http_referer>\S+)\" \"(?<http_user_agent>\S+)\" \"(?<http_x_forwarded_for>\S+)\"" log_format main '$http_host $server_addr $remote_addr [$time_local] "$request" '
'$request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time'; 继续加；
log_format main '$remote_addr [$time_local] "$request"'
'$status $body_bytes_sent'; 日志格式:
121.40.205.143 [29/Aug/2016:12:51:18 +0800] "GET /resources/plugins/artDialog/ui-dialog.css HTTP/1.1"304 0
121.40.205.143 [29/Aug/2016:12:51:18 +0800] "GET /wechat/images/account/icons.7a340e21.png HTTP/1.1"304 0
121.40.205.143 [29/Aug/2016:12:51:18 +0800] "GET /wechat/images/nav-icon.44c2022c.png?v=1 HTTP/1.1"304 0
121.40.205.143 [29/Aug/2016:12:51:19 +0800] "GET /favicon.ico HTTP/1.1"404 2319
121.40.205.143 [29/Aug/2016:12:51:19 +0800] "GET /favicon.ico HTTP/1.1"404 2319
121.40.205.143 [29/Aug/2016:12:52:25 +0800] "GET /favicon.ico HTTP/1.1"404 2319
121.40.205.143 [29/Aug/2016:12:52:25 +0800] "GET /favicon.ico HTTP/1.1"404 2319
121.40.205.143 [29/Aug/2016:12:53:28 +0800] "GET /favicon.ico HTTP/1.1"404 2319 filter {
grok {
match=> {
"message" =>"%{IPORHOST:clientip} \[%{HTTPDATE:time}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\"%{NUMBER:http_status_code} %{NUMBER:bytes}" } logstash 输出:
Pipeline main started
{
"message" => " 121.40.205.143 [29/Aug/2016:12:56:10 +0800] \"GET /favicon.ico HTTP/1.1\"404 2319",
"@version" => "1",
"@timestamp" => "2016-08-29T04:58:54.908Z",
"path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
"host" => "0.0.0.0",
"type" => "uat_nginx_access",
"clientip" => "121.40.205.143",
"time" => "29/Aug/2016:12:56:10 +0800",
"verb" => "GET",
"request" => "/favicon.ico",
"httpversion" => "1.1",
"http_status_code" => "404",
"bytes" => "2319"
} 继续； 121.40.205.143 [29/Aug/2016:13:00:16 +0800] "GET /favicon.ico HTTP/1.1"404 2319 "-"
121.40.205.143 [29/Aug/2016:13:00:22 +0800] "GET /favicon.ico HTTP/1.1"404 2319 "-"
121.40.205.143 [29/Aug/2016:13:00:30 +0800] "GET /favicon.ico HTTP/1.1"404 2319 "-"
121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/login.html HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/account.html"
121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/images/login/icon_01.6e839367.png HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/css/wechat.2a00a782.css"
121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/images/login/icon_02.5065faba.png HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/css/wechat.2a00a782.css"
121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /resources/plugins/jquery/jquery.md5.js?v=1 HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/login.html"
121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/js/libs/dialog-min.88247f5e.js?v=1 HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/login.html"
121.40.205.143 [29/Aug/2016:13:00:32 +0800] "GET /wechat/js/login.a87fbd64.js HTTP/1.1"304 0 "https://uatest.winfae.com/wechat/login.html" {
"message" => " 121.40.205.143 [29/Aug/2016:13:05:24 +0800] \"GET /wechat/account_balance.html HTTP/1.1\"200 3059 \"https://uatest.winfae.com/wechat/account.html\"
\"Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN\"", 121.40.205.143 [29/Aug/2016:13:05:24 +0800] "GET /wechat/account_balance.html HTTP/1.1"200 3059 "https://uatest.winfae.com/wechat/account.html" "Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN"
121.40.205.143 [29/Aug/2016:13:05:45 +0800] "GET /wechat/home.html?useragent=android_h5_zjcap&apiver=2 HTTP/1.1"200 11601 "-" "okhttp/2.6.0" {
"message" => " 121.40.205.143 [29/Aug/2016:13:13:11 +0800] \"GET /wechat/js/regain.431efde9.js HTTP/1.1\"304 0 \"https://uatest.winfae.com/wechat/regain.html\" \"Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN\"",
"@version" => "1",
"@timestamp" => "2016-08-29T05:15:55.609Z",
"path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
"host" => "0.0.0.0",
"type" => "uat_nginx_access",
"clientip" => "121.40.205.143",
"time" => "29/Aug/2016:13:13:11 +0800",
"verb" => "GET",
"request" => "/wechat/js/regain.431efde9.js",
"httpversion" => "1.1",
"http_status_code" => "304",
"bytes" => "0",
"http_referer" => "https://uatest.winfae.com/wechat/regain.html" \S+ 和 [^\n\t\r\f]+ 语法一样 非空格 my $str=" begin 123.456 end ";
if ($str =~/(?<request_time>\d+\.\d+)/)
{
my ($request_time) = ($+{request_time});
print $request_time."\n";};
zjtest7-frontend:/root/0825# perl a1.pl
123.456 "http_referer" => "https://uatest.winfae.com/wechat/regain.html" \"(?<http_referer>\S+)\" \"(?<http_user_agent>\S+)\"
\"Mozilla/5.0 (Linux; Android 5.1.1; vivo X6S A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile MQQBrowser/6.2 TBS/036558 Safari/537.36 MicroMessenger/6.3.25.861 NetType/WIFI Language/zh_CN\" "message" => " 121.40.205.143 [29/Aug/2016:13:54:08 +0800] \"GET /resources/plugins/artDialog/ui-dialog.css HTTP/1.1\"200 9985 \"https://uatest.winfae.com/wechat/home.html?useragent=ios_h5_zjcap&apiver=2&WKWebView=1\" \"ios_h5_zjcap\"",
"@version" => "1",
"@timestamp" => "2016-08-29T05:56:53.217Z",
"path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
"host" => "0.0.0.0",
"type" => "uat_nginx_access",
"clientip" => "121.40.205.143",
"time" => "29/Aug/2016:13:54:08 +0800",
"verb" => "GET",
"request" => "/resources/plugins/artDialog/ui-dialog.css",
"httpversion" => "1.1",
"http_status_code" => "200",
"bytes" => "9985",
"http_referer" => "https://uatest.winfae.com/wechat/home.html?useragent=ios_h5_zjcap&apiver=2&WKWebView=1",
"http_user_agent" => "ios_h5_zjcap"
} {
"message" => " 121.40.205.143 [29/Aug/2016:13:59:35 +0800] \"GET /resources/js/toolbar.49fc367e.js?_v=${last.updated}&_=1472450673142 HTTP/1.1\"200 1800 \"https://uatest.winfae.com/products/productList.html\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36\"",
"@version" => "1",
"@timestamp" => "2016-08-29T06:02:18.775Z",
"path" => "/rsyslog/data/nginx/uat/nginx_access01_log.2016-08-29",
"host" => "0.0.0.0",
"type" => "uat_nginx_access",
"tags" => [
[0] "_grokparsefailure"
]
}
logstash grok正则调试的更多相关文章
- ELK——为调试 Logstash Grok 表达式，安装 GrokDebuger 环境
内容 安装 RVM 安装 Ruby 和 Gems 安装 Rails 安装 jls-grok Ruby grok 解析 调试 grok 注意:不要用 root 执行以下操作. 用 logstash ...
- grok 正则解析日志例子<1>
<pre name="code" class="html">下面是日志的样子 55.3.244.1 GET /index.html 15824 0. ...
- logstash grok内置规则
logstash grok 内置正则 https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns USERNAME [ ...
- [elk]logstash grok原理
logstash语法 http://www.ttlsa.com/elk/elk-logstash-configuration-syntax/ https://www.elastic.co/guide/ ...
- logstash -grok插件语法介绍
介绍 logstash拥有丰富的filter插件,它们扩展了进入过滤器的原始数据,进行复杂的逻辑处理,甚至可以无中生有的添加新的 logstash 事件到后续的流程中去!Grok 是 Logsta ...
- Grok 正则捕获
Grok 正则捕获: \s+(?<request_time>\d+(?:\.\d+)?)\s+ 回顾下: (?:pattern) 匹 配 pattern 但不获取匹配结果,也就是说这是一个 ...
- grok 正则捕获(就是perl的正则捕获)
2.3.2 grok 正则捕获: \s+ 和 [\n\t\r\f]+ 一样 1.命名分组格式为(?<grp name>),反向引用时用\k<grp name> 2.命名分组的匹 ...
- logstash 使用grok正则解析日志
http://xiaorui.cc/2015/01/27/logstash%E4%BD%BF%E7%94%A8grok%E6%AD%A3%E5%88%99%E8%A7%A3%E6%9E%90%E6%9 ...
- logstash的grok正则匹配规则文件
文件路径:logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.1.0/patterns/grok-patterns 在线调试g ...
随机推荐
- C语言超级经典400道题目
C语言超级经典400道题目 1.C语言程序的基本单位是____ A) 程序行 B) 语句 C) 函数 D) 字符.C.1 2.C语言程序的三种基本结构是____构A.顺序结构,选择结构,循环结 B.递 ...
- QTableView中嵌入复选框CheckBox 的四种方法总结
搜索了一下,QTableView中嵌入复选框CheckBox方法有四种: 第一种不能之前显示,必须双击/选中后才能显示,不适用. 第二种比较简单,通常用这种方法. 第三种只适合静态显示静态数据用 第四 ...
- Linux下，连接器ld链接顺序的总结
原来ld对于链接一系列的库的顺序是很敏感的,不然会报undefined referenced 的函数符号错误,意思就是未找到函数定义.实际上库是能正确打开的.如果库libA.a依赖于库libB.a,那 ...
- 动态创建和移除HTML标签
1.w3school定义 添加新内容的四个 jQuery 方法: append() - 在被选元素的结尾插入内容 prepend() - 在被选元素的开头插入内容 after() - 在被选元素之后插 ...
- N种方法妙讲LIS算法
LIS算法经典汇总 假设存在一个序列d[1..9] = 2 1 5 3 6 4 8 9 7,可以看出来它的LIS长度为5.下面一步一步试着找出它.我们定义一个序列B,然后令 i = 1 to 9 逐个 ...
- Unity 梯子生成算法
Unity之生成梯子算法的实现. 1.通过预制物体动态生成角度可设置的梯子形状. 1.1 主要涉及到的数学知识点,角度与弧度的转化. 弧度=角度乘以π后再除以180 角度=弧度除以π再乘以180 1. ...
- NET基础课--NET中程序集0-1
程序集 1.表现形式:.dll 和. exe . 2.程序集组成:PE头,CLR头,清单,元数据,CIL代码,资源文件.实际上这些内容包含在一个叫做Module的逻辑结构中. 单模块程序集:程序集就 ...
- asp.net 后台对话框,确认跳转
Response.Write("<script>alert('不合法!'); window.location.href='" + ResolveClientUrl(&q ...
- UVA 12125 March of the Penguins
题意: 给定一些冰块,每个冰块上有一些企鹅,每个冰块有一个可以跳出的次数限制,每个冰块位于一个坐标,现在每个企鹅跳跃力为d,问所有企鹅能否跳到一点上,如果可以输出所有落脚冰块,如果没有方案就打印-1 ...
- Oracle运维服务的四根救命稻草
企业信息化系统建设按生命周期可分为IT规划阶段.IT建设阶段和IT运维阶段,其中,IT运维阶段的时间最长,IT运维管理关乎着IT运维的质量.成本和速度,更关乎着IT系统的安全.连续和可用.大数据云计算 ...