【原创】大数据基础之Flume(2)Sink代码解析
flume sink核心类结构
1 核心接口Sink
org.apache.flume.Sink
/**
* <p>Requests the sink to attempt to consume data from attached channel</p>
* <p><strong>Note</strong>: This method should be consuming from the channel
* within the bounds of a Transaction. On successful delivery, the transaction
* should be committed, and on failure it should be rolled back.
* @return READY if 1 or more Events were successfully delivered, BACKOFF if
* no data could be retrieved from the channel feeding this sink
* @throws EventDeliveryException In case of any kind of failure to
* deliver data to the next hop destination.
*/
public Status process() throws EventDeliveryException; public static enum Status {
READY, BACKOFF
}
process为核心接口,返回值为状态,只有两个:ready和backoff,调用方会根据返回值做相应处理,后边会看到;
这个接口也是扩展flume sink需要实现的接口,比如KuduSink;
2 Sink封装
org.apache.flume.SinkProcessor
/**
* <p>
* Interface for a device that allows abstraction of the behavior of multiple
* sinks, always assigned to a SinkRunner
* </p>
* <p>
* A sink processors {@link SinkProcessor#process()} method will only be
* accessed by a single runner thread. However configuration methods
* such as {@link Configurable#configure} may be concurrently accessed.
*
* @see org.apache.flume.Sink
* @see org.apache.flume.SinkRunner
* @see org.apache.flume.sink.SinkGroup
*/
public interface SinkProcessor extends LifecycleAware, Configurable {
/**
* <p>Handle a request to poll the owned sinks.</p>
*
* <p>The processor is expected to call {@linkplain Sink#process()} on
* whatever sink(s) appropriate, handling failures as appropriate and
* throwing {@link EventDeliveryException} when there is a failure to
* deliver any events according to the delivery policy defined by the
* sink processor implementation. See specific implementations of this
* interface for delivery behavior and policies.</p>
*
* @return Returns {@code READY} if events were successfully consumed,
* or {@code BACKOFF} if no events were available in the channel to consume.
* @throws EventDeliveryException if the behavior guaranteed by the processor
* couldn't be carried out.
*/
Status process() throws EventDeliveryException;
这个类负责封装单个sink或者sink group的处理,常用的子类有:
1)单个sink
org.apache.flume.sink.DefaultSinkProcessor
@Override
public Status process() throws EventDeliveryException {
return sink.process();
}
DefaultSinkProcessor的process会直接调用内部sink的process;
2)sink group
org.apache.flume.sink.LoadBalancingSinkProcessor
org.apache.flume.sink.FailoverSinkProcessor.FailedSink
3 sink的调用方为SinkRunner
org.apache.flume.SinkRunner
/**
* <p>
* A driver for {@linkplain Sink sinks} that polls them, attempting to
* {@linkplain Sink#process() process} events if any are available in the
* {@link Channel}.
* </p>
*
* <p>
* Note that, unlike {@linkplain Source sources}, all sinks are polled.
* </p>
*
* @see org.apache.flume.Sink
* @see org.apache.flume.SourceRunner
*/
public class SinkRunner implements LifecycleAware {
...
private static final long backoffSleepIncrement = 1000;
private static final long maxBackoffSleep = 5000; org.apache.flume.SinkRunner.PollingRunner public static class PollingRunner implements Runnable { private SinkProcessor policy;
private AtomicBoolean shouldStop;
private CounterGroup counterGroup; @Override
public void run() {
logger.debug("Polling sink runner starting"); while (!shouldStop.get()) {
try {
if (policy.process().equals(Sink.Status.BACKOFF)) {
counterGroup.incrementAndGet("runner.backoffs"); Thread.sleep(Math.min(
counterGroup.incrementAndGet("runner.backoffs.consecutive")
* backoffSleepIncrement, maxBackoffSleep));
} else {
counterGroup.set("runner.backoffs.consecutive", 0L);
}
} catch (InterruptedException e) {
logger.debug("Interrupted while processing an event. Exiting.");
counterGroup.incrementAndGet("runner.interruptions");
} catch (Exception e) {
logger.error("Unable to deliver event. Exception follows.", e);
if (e instanceof EventDeliveryException) {
counterGroup.incrementAndGet("runner.deliveryErrors");
} else {
counterGroup.incrementAndGet("runner.errors");
}
try {
Thread.sleep(maxBackoffSleep);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
}
logger.debug("Polling runner exiting. Metrics:{}", counterGroup);
} }
无论process返回backoff或者抛exception,都会sleep一段时间,所以flume的sink一旦遇到大量异常数据或者自定义sink返回backoff,都会非常慢;
【原创】大数据基础之Flume(2)Sink代码解析的更多相关文章
- 【原创】大数据基础之Flume(2)kudu sink
kudu中的flume sink代码路径: https://github.com/apache/kudu/tree/master/java/kudu-flume-sink kudu-flume-sin ...
- 【原创】大数据基础之Flume(2)应用之kafka-kudu
应用一:kafka数据同步到kudu 1 准备kafka topic # bin/kafka-topics.sh --zookeeper $zk:2181/kafka -create --topic ...
- 【原创】大数据基础之Zookeeper(2)源代码解析
核心枚举 public enum ServerState { LOOKING, FOLLOWING, LEADING, OBSERVING; } zookeeper服务器状态:刚启动LOOKING,f ...
- 大数据系列之Flume+kafka 整合
相关文章: 大数据系列之Kafka安装 大数据系列之Flume--几种不同的Sources 大数据系列之Flume+HDFS 关于Flume 的 一些核心概念: 组件名称 功能介绍 Agent ...
- 【原创】大数据基础之词频统计Word Count
对文件进行词频统计,是一个大数据领域的hello word级别的应用,来看下实现有多简单: 1 Linux单机处理 egrep -o "\b[[:alpha:]]+\b" test ...
- 【原创】大数据基础之Impala(1)简介、安装、使用
impala2.12 官方:http://impala.apache.org/ 一 简介 Apache Impala is the open source, native analytic datab ...
- 【原创】大数据基础之Benchmark(2)TPC-DS
tpc 官方:http://www.tpc.org/ 一 简介 The TPC is a non-profit corporation founded to define transaction pr ...
- 【原创】大数据基础之Spark(5)Shuffle实现原理及代码解析
一 简介 Shuffle,简而言之,就是对数据进行重新分区,其中会涉及大量的网络io和磁盘io,为什么需要shuffle,以词频统计reduceByKey过程为例, serverA:partition ...
- 【原创】大数据基础之Spark(4)RDD原理及代码解析
一 简介 spark核心是RDD,官方文档地址:https://spark.apache.org/docs/latest/rdd-programming-guide.html#resilient-di ...
随机推荐
- 使用wget命令下载JDK失败(文件特别小)
问题RT: 我们在网页上下载的时候要点一下 “Accept License Agreement ” ,使用wget下载的时候也需要提交这个 accept,方法如下: wget --no-check-c ...
- MySQL的随笔
数据库引擎 MySQL5.0支持的存储引擎包括MyISAM,InnoDB.MEMORY.MERGE.BDB等等,其中InnoDB和BDB提供事务安全表,其他存储引擎都是非事务安全表. MyISAM M ...
- setDefaultKeyMode设置Activity的五种按键模式
setDefaultKeyMode (int mode) 用来设置一个Activity的默认的按键模式, mode一共有五种 DEFAULT_KEYS_DISABLE DEFAULT_KEYS_DIA ...
- luogu 2480 古代猪文 数论合集(CRT+Lucas+qpow+逆元)
一句话题意:G 的 sigma d|n C(n d) 次幂 mod 999911659 (我好辣鸡呀还是不会mathjax) 分析: 1.利用欧拉定理简化模运算 ,将上方幂设为x,则x=原式mod ...
- ads出现村田电容电感无法仿真的问题解决(`BJT1' is an instance of an undefined model `BJTM1')
需要的控件是 murata include,该控件是跟随村田库一起倒入ADS中的
- Docker(二)搭建和使用Docker
摘自 https://mp.weixin.qq.com/s/E9tqhe00EjfV8y1pqWkZfw 一.Docker的架构 Docker使用C/S结构,即客户端/服务器体系结构.Docker客户 ...
- c# 读取excels
DataTable ExcelTable; DataSet ds = new DataSet(); //Excel 文件一般都保存为统一的xls的连接 其 ...
- Nginx 关闭日志生成文件
nginx 关闭日志:其实一种方法就是写入/dev/null 文件 或者设置关闭: nginx 日志有两个类型 access.log http 记录访问日志. error.log server ...
- Python 获取文件中最长行的长度和最长行
1, 使用文件 #vim /etc/motd "1 hello world" 2 ...... yes 3 no you are a shadiao 4 hahh maye you ...
- Service Mesh
概念 A service mesh is a dedicated infrastructure layer for handling service-to-service communication. ...