logstash filter grok 用法

在elk+filebeat都安装好，且明白了基本流程后，主要的就是写logstash的filter了，以此来解析特定格式的日志

logstash的filter是用插件实现的，grok是其中一个，用来解析自定义结构的日志，就是说可以完全自定义的去解析

grok的语法是%{pattern:field}

pattern是指模板（或者说是类型），logstash内置了很多，也可以自定义（用正则）

field是指解析后的键值中的键，当然值就是从日志解析出来的内容

（无论用什么pattern解析到elasticsearch默认都是用string存储，可以通过%{pattern:field:int}将之转换为number类型就可以进行加减计算了，只支持int和float）

比如，一个日志文件中是序号+ip，这样

1 127.0.0.1
2 127.0.0.2
3 127.0.0.3

filter就可以这样写（input省略了，肯定是从filebeat来的，output中指定index，方便测试）

filter {
    grok {
        match => { "message" => "%{NUMBER:myid} %{IP:myip}"}
    }
}
output {
    elasticsearch {
        hosts => "localhost"
        index => "mytest"
    }
}

那么日志到 elasticsearch中，就是这样：

{
  ...
  "hits" : {
    ...
    "hits" : [ {
      ...
      "_source" : {
        "message" : "1 127.0.0.1",
        ...
        "myid" : "",
        "myip" : "127.0.0.1"
      }
    }, {
      ...
      "_source" : {
        "message" : "2 127.0.0.2",
        ...
        "myid" : "",
        "myip" : "127.0.0.2"
      }
    }, {
      ...
      "_source" : {
        ...
        "myid" : "",
        "myip" : "127.0.0.3"
      }
    } ]
  }
}

kibana中的效果

aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyQAAACPCAIAAACEW4jjAAAQeklEQVR4nO3cv67jxmIH4H04PsA+Arvt/AR8gDXcuFTrZksXRJDCuICBxAGSAE50m7TGBjEQIIvAhovrJlAKkdL8pShRs+dI+30gsDoSdzgajmZ+GvKcNwcAAJp589IVAAB4ZsIWAEBDwhYAQEPCFgBAQ8IWAEBDwhYAQEPCFgBAQ8IWAEBDwhYAQEPCFgBAQ3cMWx8/vOtS7z58/Ol913Xvf7rfceCxlD4CPhbAa2ecup8GK1tOD1ziUwLw5Wgftk4//vS+67p379+/mxe9fvpwevhx2vm8OnZ+Dl6RC93444d3p8770/vjw+ATMXfw9++FLeBz2jJ2xf/X9HyDzx22grM67XTePTjXvvfzSl3qxh8/vEufil979+HjXIoeDnwuW8au7P8avK712Ve2zrk5/K7/7sPHKFdLW7xSl7rxacTKR670m6MODnw2G8au6P9GUzVrvbawld5ePxXyx8d//ad//ss/Jtu//8f/3r/6sOTigDWPVD+9n18rha3zcAbwGWwYu+Jp3VfFW7y2sFWLy3/7+Nd/+fu//MPfBdsP//aff/zf/asPSy4PWNPjd+/S3axsAS9mw9hlZWu7VxS2soCdnM0ob0lavIwVA9Z0f0N+Tdw9W8BL2TJ2pfdsyVpXe01h63A607WJaMpbkhYvZk03jnbKd+v8NiLwuW0Zu/w24mYP9xfk//bf//U/khavm4uEwCOqjF1xBuMGDxe24FU7rV2JWsADWRq7hK3NhC0AgIaELQCAhoQtAICGhC0AgIaELQCAhoQtAICGhC0AgIaELQCAhoQtAICGhC0AgIbefPrtd5vNZrPZbDZbo03YstlsNpvNZmu4uYwIANCQsAUA0JCwBQDQkLAFANCQsAUA0JCwBQDQkLAFANCQsAUA0JCwBQDQkLAFANCQsAUA0NDjhq1x6Prd/qVrkXut9QIAXsRdw9Z+1wdBYxy6Ybxn8bFWoWa/67tZUP/z0+FRC5W4pl6VYx3GofTs8emsTYNCbmvvcrHTCyuLjE99+L7SJqvXPtnt3AjL7Xner9AIcb0Od2itQ/20Abxy+XhfHNAWx9VMeX5cqMCa0fPJBto3h8Ph8Mv333z7w6/bCzu2zdzYDxq2oiPM76D8cHPYGsewZ0+lBvng9PDYssNu1ydtut/1m1NDodi5SivLjk98VsxSGftdX3p5f67Svli7WmGFZBU8FZycu/SfK+IowEuqjffFeSj+f5e/8Jbmx0od0untsqcYaN8cDtM5eDuMW/PWftcPw6lVzrNsEFCDZk6WjeYfs6WHcTh2j64bxqm3jIdpshznnYNzlq9dHIufn187xQaVSgNWN4xZ8A/27Xf7+dW1HeTc76IemHTHLHfcpw8W4szxqUJQGofSElS/G8ufx7yEpIBK2Are2cVPelDnaMdCvYL2vMd3gStyIMBrsDBslfLP5QG4ND8uHLs2vd1U4wdyDlvd9rx1nDjnxp5ns3AJYQyjWNjmhRA2Px6H4+Q8/TvvHE7a8y7lAqIFjnG4eHbn5jid3fwa2cLySLAgtD65V78ZxD+lfe4Uia5KkZmsKwdtfDFsTfsUP5GlJJWHrco68bXhODmztXpdG4OLsi4C8BDq0aUUlC7PmPX58dIRLi4WPNVAG4Wtruvefv3j7YVNc+vU2tNsF+eRaMEiy13pvT5zyJpDWzh7RuXOxRYLuDEXn6/QXRu2wmy5Iios9b/LYWv1+m1dvs54LvJCiadPYiFs1RatqnUIDzY13dIFynJFFusVBvg7fIK3XcQF+Owq82FxAlmxONEybAVlPv5AG4ett1999/Oftxd2ml33u77r+34xbM2nPF2Lys7TyrB1ilXFBZabztTpENVl0u1hK9vl2suIeSNcLS42uz5ab7ss28afuatqk73vlevS8y55CM3rlSyn3uemLb97CjyO0nxYHshW3mvR/DLicwy0QdjamLQO0VJGsHRQvow4/dQPQ7ryULjJpxK2ogWxc8wrXpDasrKVXubLV9Ti6q4OW8WKJXcQ5pcNK0fLU+3a7lltn3X3bM1l5DdMVW6YLxWQnLgk0UaPSzcVVM9vVK/ww32PsPUcX7iAL0jptpHSMFZ+OrqcMlmaH+uLCVH5pWLjYz78QDuHre1J65BcN0rWsLKVj9MLhVuzV15GnO6bT4rNCrgubAULO+VSs24a731F2EoWYJI+Gj6VL9VEXyQKFVsVthaKnQu5MWzV33lSQKX60XmIry5mh188vUm9ql3xGpUuAvB6Vcb7yjy0EMHqd9gW/oZR6VpVbYLNM9wTDbTHv7P1x6dPm5MWr8w9ft0OANjqcf+CPIue4Ro3ADwDYQsAoCFhCwCgIWELAKAhYQsAoCFhCwCgIWELAKAhYQsAoCFhCwCgIWELAKAhYQsAoCFhCwCgoTeffvvdZrPZbDabzdZos7IFANCQsAUA0JCwBQDQkLAFANCQsAUA0JCwBQDQkLAFANCQsAUA0JCwBQDQkLAFANCQsAUA0JCwBQDQ0OOGrXHo+t3+pWuRe631AgBexF3D1n7XB0FjHLphvGfxsVahZr/ru1lQ//PT4VELlbimXpVjHcah9Ozx6axNg0Jua+9ysdMLK4uMT334vtImq9c+2e3cCMvted6v0AhxvQ53aK2gGKEaeDD5eF+chxbH1Ux5flyowMoh+JkG2jeHw+Hwy/fffPvDr9sLOzb43DIPGraiI8zvoPxwc9gax7BnT6UG+eD08Niyw27XJ2263/UbUkO12LlKK8uOT3xWzFIZ+11fenl/rtK+WLtaYYVkFTwVnJyb+884dF3X73ZWMIEHUhvvi/NQ/P8uf+EtzY+VOqTTW73Qpxpo3xwO0zl4O4xb89Z+1w/Dqa3Ps2wQnINmTpaN5h+zpYdxOHaPrhvGqbeMh2myHOedg9ORr10ci5+fX3vmgkqlAasbxiz4B/v2u/386tocdO53UQ9MumOWO65Ye7pw8KSU41OFoDR9ApIlqH43lj+PeQlJAZWwFbyzi5/0oM7RjoV6Be259buAy8XAA1r4+lrKP5cH4NL8uHDs2vRWLfxJBtpz2Oq2563jxDk39jybhY01hlEsbPNCCJsfj8Nxcp7+nXcOJ+15l3IB0QLHOFw8c3NznHpMfo1sYXkkWBBa1ZeSdkn6avxT+ik5RaKrUmQm+/AFbXwxbE37FD+RpSSVh63KkvK14Tg5s7V6XRuD64d7ljEA+ILUw1YpKF2eMevz46UjrFkseJ6BNgpbXde9/frH2wub5taptafZLm6raMEiy13pvT5zyJpDWzh7RuXOxRYLuOZCVPxuiqHpctgKs+WKfrLU/y6HrdXrt3X5OuO5yAslnj6JhbBVW7Sq1iE82NR0SxcoyxVZrFcY4DfmrecZA4AvSGU+LE4gKxYnhK2V4rD19qvvfv7z9sJOs+t+13d93y+GrfmUp2tRWcOuDFunWFVcYLlpYj0dorpMuj1sZbtcexkxb4SrxcVm10frbZdl2/gzd1Vtsve9cl163iUPoXm9kuXUTR/i5xkDgC9IaT4sD2cr77VwGXGVIGxtTFqHaCkjWDooX0acfuqHIV15KNzkUwlb0YLYOeYVL0htWdlKL/PlK2pxdVeHrWLFkjsI88uGlaPlqXZtJ622z7p7tuYy8humKjfMlwpITlySaKPHpZsKquc3qlf44Ra2gC9Q6baR0gBafjq6nDJZmh/riwlR+aVi8wIf2hy2tietQ3LdKFnDylY+Ti8Ubs1eeRlxum8+KTYr4LqwFSzslEvNumm89xVhK1mASfpo+FS+VBN9kShUbFXYWih2LuTGsFV/50kBlepH5yG+upgdfvH0JvWqdsX1spW/JxkKgKdWGe8r89BCBKvfYVv4G0ala1X5yJkX+3QD7fHvbP3x6dPmpMUrs/XX7QCAe3jcvyDPoudZfAWAxyZsAQA0JGwBADQkbAEANCRsAQA0JGwBADQkbAEANCRsAQA0JGwBADQkbAEANCRsAQA0JGwBADT05tNvv9tsNpvNZrPZGm1WtgAAGhK2AAAaErYAABoStgAAGhK2AAAaErYAABoStgAAGhK2AAAaErYAABoStgAAGhK2AAAaetywNQ5dv9u/dC1yr7VeAMCLuGvY2u/6IGiMQzeM9yw+1irU7Hd9Nwvqf346PGqhEtfUq3KswziUnj0+nbVpUMht7V0udnphZZHxqQ/fV9pk9donu50bYbk9z/sVGiGu1+EOrRUfUa4GHkk+3hfnocVxNVOeHxcqsGYIfrKB9s3hcDj88v033/7w6/bCjg0+N8uDhq3oCPM7KD/cHLbGMezZU6lBPjg9PLbssNv1SZvud/2G1FAtdq7SyrLjE58Vs1TGfteXXt6fq7Qv1q5WWCFZBU8FJ+f2/rMfx/25/GcYBYDnVxvvi/NQ/P8uf+EtzY+VOqTTW23XJxto3xwO0zl4O4xb89Z+1w/Dqa3Ps2wQnINmTpaN5h+zpYdxOHaPrhvGqbeMh2myHOedgzORr10ci5+fX3vSgkqlAasbxiz4B/v2u/386tocdO5NUb9KOlmWO65Ye7pw8KSU41OFoDQOpSWofjeWP495CUkBlbAVvLOLn/SgztGOhXoF7XmX7wL3aX+Az2Xh62sp1VwegEvz48Kxa9Pb2iM8qnPY6rbnrePEObfLPJuFSwhjGMXCNi+EsPnxOBwn5+nfeedw0p53KRcQLXCMw8WzOzfH6eTm18gWlkeCBaGb+lLcreKf0k/JKRJtW2nNPnxBG18MW9M+xU9kKUnlYauypHxtOE7ObK1e18bgBc/xfQv4ktTDVinTXJ4x6/PjpSOszFBPMtBGYavrurdf/3h7YdPcOjXNNNvFeSRasMhyV3qvzxyy5tAWzp5RuXOxxQKuuRAVv5tiaLoctsJsuaKPLPW/y2Fr9fptXb7OeC7y4lX186krfCNaX5/0eujUdEsXKMsVWaxXGOA35q0nGQCAL0tlPixOICsWJxqHrecZaOOw9far737+8/bCTrPrftd3fd8vhq35lKdrUVm7rgxbp1hVXGC5aWI9HaK6TLo9bGW7XHsZMW+Eq8XFZtdH622XZdv4M3dVbbL3vXJdet4lD6F5vZLl1DuuBQI8hNLoVZ48Vt5r0e4y4lMNtEHY2pi0DtFSRrB0UL6MOP3UD0O68lC4yacStqIFsXPMK16Q2rKylV7my1fU4uquDlvFiiV3EOaXDStHy1Pt2jBRbZ9192zNZeQ3TFVumC8VkJy4JNFGj0s3FVTPb1Sv8MO9IWy1/90MgDZKt42UBtDy09HllMnS/FhfTIjKrxT7TAPtHLa2J61Dct0oWcPKVj5OLxRuzV55GXG6bz4pNivgurAVLOyUS826abz3FWErWYBJ+mj4VL5UE32RKFRsVdhaKHYu5MawVX/nSQGV6kfnIb66mB1+8fQm9ap2xSskS39P88ULeGaV8b4yDy1EsPodtoW/YVS6VlWbYPM57IkG2uPf2frj06fNSYtX5i6/bgcAbPS4f0GeRU+2AgsAD0vYAgBoSNgCAGhI2AIAaEjYAgBoSNgCAGhI2AIAaEjYAgBoSNgCAGhI2AIAaEjYAgBoSNgCAGhI2AIAaEjYAgBoSNgCAGhI2AIAaEjYAgBoSNgCAGhI2AIAaOj/AXUoX+BGX032AAAAAElFTkSuQmCC" alt="" />

grok的基础用法就是这样了，一般来说内置的pattern就够用了

更多pattern可以看https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns

像IP、NUMBER这样的基本pattern位于grok-patterns中，常用的有：

INT	int类型
NUMBER	数字
DATA	数据，可以对应字符串
GREEDYDATA	数据，可以对应字符串，贪婪匹配
QUOTEDSTRING	带引号的字符串，可以简写为QS
WORD	一个词
IP	ip地址，v4 或 v6
DATE	日期
TIME	时间
DATESTAMP	日期+时间
PATH	系统路径
HOSTNAME	计算机名
MAC	mac地址
UUID	uuid
LOGLEVEL	日志等级
EMAILADDRESS	email地址

另外，还有可以直接用的模板集合，比如官方文档中的例子：

filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}

这个COMBINEDAPACHELOG就是一个内置的集合，这种集合还有很多，也在上面那个链接中，COMBINEDAPACHELOG就位于httpd，它的定义是：

COMBINEDAPACHELOG %{HTTPD_COMBINEDLOG}
HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} %{QS:referrer} %{QS:agent}
HTTPD_COMMONLOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} ...

（HTTPD_COMMONLOG的定义有点长，就省略了~~）

当日志的结构比较多，就需要自定义为模板集合，方便管理（自定义模板是一样的方式，感觉没必要，用内置的组合成自己的集合就行了）

mkdir -p /etc/logstash/patterns

vi /etc/logstash/patterns/test.conf

MYPATTERN %{NUMBER:myid} %{IP:myip}

filter改为

filter {
    grok {
        patterns_dir => ["/etc/logstash/patterns"]
        match => { "message" => "%{MYPATTERN}"}
    }
}

测试下，效果当然跟上面是一样的。

还有一点，pattern本质都是正则，%{}之间的空格不是结构要求，而是因为日志就是空格分隔的，如果日志都用空格和-来分隔，那就是：

MYPATTERN %{TIME:mytime} - %{DATA:mythread} - %{LOGLEVEL:mylevel} - %{DATA:myclass} - %{GREEDYDATA:myinfo}

特别说明一个情况，java中log4j的日志会出现日志级别ERROR和DEBUG后面一个空格，而其他INFO那些后面两个空格

原因应该是ERROR和DEBUG比INFO那些多一个字母，为了对齐，这种情况难在不知道以致浪费时间到处查解析正确的原因，知道了当然就简单了

%{LOGLEVEL:mylevel}  %{DATA:myclass}
改为
%{LOGLEVEL:mylevel}\s{1,2}%{DATA:myclass}

over