Processor 介绍

1.概述

Sink Group允许用户将多个Sink组合成一个实体。

Flume Sink Processor 可以通过切换组内Sink用来实现负载均衡的效果，或在一个Sink故障时切换到另一个Sink。

 

sinks – 用空格分隔的Sink集合

processor.type default 类型名称，必须是 default、failover 或 load_balance

 

2.Default Sink Processor

Default Sink Processor 只接受一个 Sink。

不要求用户为单一Sink创建processor

 

3.Failover Sink Processor

Failover Sink Processor 维护一个sink们的优先表。确保只要一个是可用的就事件就可以被处理。

失败处理原理是，为失效的sink指定一个冷却时间，在冷却时间到达后再重新使用。

sink们可以被配置一个优先级，数字越大优先级越高。

如果sink发送事件失败，则下一个最高优先级的sink将会尝试接着发送事件。

如果没有指定优先级，则优先级顺序取决于sink们的配置顺序，先配置的默认优先级高于后配置的。

在配置的过程中，设置一个group processor ，并且为每个sink都指定一个优先级。

优先级必须是唯一的。

另外可以设置maxpenalty属性指定限定失败时间。

 

sinks – Space-separated list of sinks that are participating in the group

processor.type default The component type name, needs to be failover

processor.priority.<sinkName> – Priority value. <sinkName> must be one of the sink instances associated with the current sink group A higher priority value Sink gets activated earlier. A larger absolute value indicates higher priority

processor.maxpenalty 30000 The maximum backoff period for the failed Sink (in millis)

 

Example for agent named a1:

------

a1.sinkgroups = g1

a1.sinkgroups.g1.sinks = k1 k2

a1.sinkgroups.g1.processor.type = failover

a1.sinkgroups.g1.processor.priority.k1 = 5

a1.sinkgroups.g1.processor.priority.k2 = 10

a1.sinkgroups.g1.processor.maxpenalty = 10000

------

 

4.Load balancing Sink Processor

Load balancing Sink processor 提供了在多个sink之间实现负载均衡的能力。

它维护了一个活动sink的索引列表。

它支持轮询 或 随机方式的负载均衡，默认值是轮询方式，可以通过配置指定。

也可以通过实现AbstractSinkSelector接口实现自定义的选择机制。

 

!processor.sinks – Space-separated list of sinks that are participating in the group

!processor.type default The component type name, needs to be load_balance

processor.backoff false Should failed sinks be backed off exponentially.

processor.selector round_robin Selection mechanism. Must be either round_robin, random or FQCN of custom class that inherits from AbstractSinkSelector

processor.selector.maxTimeOut 30000 Used by backoff selectors to limit exponential backoff (in milliseconds)

 

------

a1.sinkgroups = g1

a1.sinkgroups.g1.sinks = k1 k2

a1.sinkgroups.g1.processor.type = load_balance

a1.sinkgroups.g1.processor.backoff = true

a1.sinkgroups.g1.processor.selector = random