07_Flume_regex interceptor实践

实践一：regex filter interceptor

1、目标场景

regex filter interceptor的作用：

1）将event body的内容和配置中指定的正则表达式进行匹配
2）如果内容匹配，则将该event丢弃
3）如果内容不匹配，则将该event放行

2、Flume Agent配置文件

#  define agent name, source/sink/channel
a1.sources = r1
a1.sinks = k1
a1.channels = c1

#  source,http,jsonhandler
a1.sources.r1.type = http
a1.sources.r1.bind = master
a1.sources.r1.port =
a1.sources.r1.handler = org.apache.flume.source.http.JSONHandler

# 03 regex filter interceptor, match event body for filter
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = regex_filter
a1.sources.r1.interceptors.i1.regex = ^[0-9]*$
# filter matched event
a1.sources.r1.interceptors.i1.excludeEvents = true  

#  logger sink
a1.sinks.k1.type = logger

#  channel,memory
a1.channels.c1.type = memory
a1.channels.c1.capacity =
a1.channels.c1.transactionCapacity = 

#  bind source,sink to channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

3、验证regex filter interceptor

1) 通过curl -X POST -d 'json数据' 发送带有不同body的HTTP请求，其中有1个满足regex

2）观察终端打印出的event，body为1234的event被过滤， 并没有出现

4、regex filter interceptor的官方文档

实践二：regex extractor interceptor

1、目标场景

regex extractor interceptor的作用：
1）将event body的内容和配置中指定的正则表达式进行匹配
2）如果内容匹配，将配合配置文件中给定的key, 组成key:value添加到event的header中
3）event body中的内容不会变化

2、Flume Agent的配置文件

#  define agent name, source/sink/channel
a1.sources = r1
a1.sinks = k1
a1.channels = c1

#  source,http,jsonhandler
a1.sources.r1.type = http
a1.sources.r1.bind = master
a1.sources.r1.port =
a1.sources.r1.handler = org.apache.flume.source.http.JSONHandler

# 03 regex extractor interceptor,match event body to extract character and digital
a1.sources.r1.interceptors = i1  
a1.sources.r1.interceptors.i1.type = regex_extractor
a1.sources.r1.interceptors.i1.regex = (^[a-zA-Z]*)\\s([0-9]*$)  # regex匹配并进行分组，匹配结果将有两个部分, 注意\s空白字符要进行转义
# specify key for 2 matched part
a1.sources.r1.interceptors.i1.serializers = s1 s2
# key name
a1.sources.r1.interceptors.i1.serializers.s1.name = word
a1.sources.r1.interceptors.i1.serializers.s2.name = digital 

#  logger sink
a1.sinks.k1.type = logger

#  channel,memory
a1.channels.c1.type = memory
a1.channels.c1.capacity =
a1.channels.c1.transactionCapacity = 

#  bind source,sink to channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

3、验证regex extractor interceptor

1) 通过curl -X POST -d 'json数据'的方式发送HTTP请求，body中的内容为"shayzhang 1234", 其中shayzhang，1234将被正则表达式匹配

2) 观察logger打印到终端的event，header中将增加两部分 word:shayzhang, digital:1234