Flume学习——Flume的架构

Flume有三个组件：Source、Channel 和 Sink。在源码中对应同名的三个接口。

When a Flume source receives an event, it stores it into one or more channels. The channel is a passive store that keeps the event until it’s consumed by a Flume sink.

public interface Source extends LifecycleAware, NamedComponent {

  /**
   * Specifies which channel processor will handle this source's events.
   *
   * @param channelProcessor
   */
  public void setChannelProcessor(ChannelProcessor channelProcessor);

  /**
   * Returns the channel processor that will handle this source's events.
   */
  public ChannelProcessor getChannelProcessor();

}

Source并没有与定义与Event有关的接口，它的接口只是对ChannelProcessor的get和set方法。Source通过获取对应的ChannelProcessor来完成消息的投递。

public interface Sink extends LifecycleAware, NamedComponent {
  public void setChannel(Channel channel);
  public Channel getChannel();
  public Status process() throws EventDeliveryException;
  public static enum Status {
    READY, BACKOFF
  }
}

Sink也有与Channel有关的set和get方法，不过它是对应于Channel，而是ChannelProcessor。ChannelProcessor是位于Source和Channel之间的一个消息分发器一样的角色。因此，一个Source的消息可以通过ChannelProcessor发给多个Channel(简单的分发到所有，或者有选择的发送给特定channel)，但是Sink直接对应于Channel，所以每个Sink只从唯一一个Channel中获得消息。

ChannelProcessor的API为：

void	close()
void	configure(Context context)           The Context of the associated Source is passed.
ChannelSelector	getSelector()
void	initialize()
void	processEvent(Event event)           Attempts to put the given event into each configured channel.
void	processEventBatch(List<Event> events)           Attempts to put the given events into each configured channel.

通过ChannelProcessor， Flume可以实现下面的消息流

public interface Channel extends LifecycleAware, NamedComponent {
  public void put(Event event) throws ChannelException;
  public Event take() throws ChannelException;
  public Transaction getTransaction();
}

A channel connects a Source to a Sink. The source acts as producer while the sink acts as a consumer of events. The channel itself is the buffer between the two.

A channel exposes a Transaction interface that can be used by its clients to ensure atomic put and take semantics. This is necessary to guarantee single hop reliability between agents. For instance, a source will successfully produce an event if and only if that event can be committed to the source's associated channel. Similarly, a sink will consume an event if and only if its respective endpoint can accept the event. The extent of transaction support varies for different channel implementations ranging from strong to best-effort semantics.

Channels are associated with unique names that can be used for separating configuration and working namespaces.

Channel连接起了Source和Sink。Source相当于消息的生产者，Sink相当于消息的消费者。Channel相当于二者之间的缓冲层。

更重要的是，Channel提供了Transaction的机制，来确保了消息的可靠传递。

Flume uses a transactional approach to guarantee the reliable delivery of the events. The sources and sinks encapsulate in a transaction the storage/retrieval, respectively, of the events placed in or provided by a transaction provided by the channel. This ensures that the set of events are reliably passed from point to point in the flow. In the case of a multi-hop flow, the sink from the previous hop and the source from the next hop both have their transactions running to ensure that the data is safely stored in the channel of the next hop.

Flume使用了事务来保证消息的可靠传递。Source和Sink对于消息的存储和获取都被包装在由Channel提供的事务中。一个消息只有被成功存入下一个agent的Channel（或者至最终的存储位置），才会被从当前的channel中删除。这就使用在消息流中，消息可以可靠地从一点传送到另一点。在一个包括agent间传递的消息流中，前一个agent的sink和下一个agent的source都有各自的事务，来保证消息被安全存入下一个agent的channel.