flume+kafka
这里演示在单机fulume环境下,kafka作为source ,chanel , sink时三种情况
下面的测试都是基于下面的基本的配置文件进行修改的
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# For each one of the sources, the type is defined
#agent.sources.seqGenSrc.type = seq
a1.sources.r1.type = netcat
a1.sources.r1.bind=mini1
a1.sources.r1.port= # The channel can be defined as follows.
#agent.sources.seqGenSrc.channels = memoryChannel
a1.channels.c1.type=memory
a1.channels.c1.capacity=
a1.channels.c1.transactionCapacity = # Each sink's type must be defined
#agent.sinks.loggerSink.type = logger
a1.sinks.k1.type = logger
#Specify the channel the sink should use
#agent.sinks.loggerSink.channel = memoryChannel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
# Each channel's type is defined.
#agent.channels.memoryChannel.type = memory # In this case, it specifies the capacity of the memory channel
#agent.channels.memoryChannel.capacity =
kafka作为source时的配置和produce程序
a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.channels = c1
a1.sources.r1.batchSize =
a1.sources.r1.batchDurationMillis =
a1.sources.r1.kafka.bootstrap.servers = mini1:
a1.sources.r1.kafka.topics = Operator
a1.sources.r1.kafka.consumer.group.id = custom.g.id
public static void main(String[] args) throws IOException {
Properties props = new Properties();
props.load(TestConsumer.class.getClass().getResourceAsStream("/kafkaProduce.properties"));
Producer<Integer, String> producer = new KafkaProducer<>(props);
for (int i = ; i <; i++)
producer.send(new ProducerRecord<Integer, String>("Operator", i, getRandomPhoneNum()));
producer.close();
// System.out.println(getRandomPhoneNum());
} public static String getRandomPhoneNum(){
String[] basePrefix=new String[]{"","","","","",""};
return basePrefix[new Random().nextInt(basePrefix.length)]+ RandomUtils.nextInt(,);
}
kafka作为channel时 ,topic必须是一个新的topic如果topic中存在数据那么在启动时会报错
a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers = mini1:,mini2:,mini3:
a1.channels.c1.kafka.topic = flumedat
a1.channels.c1.kafka.consumer.group.id = flume-consumer #修改source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /home/hadoop/flume/test/logs/flume.dat
a1.sources.r1.channels = c1
按照官网的说明,当kafka作为channel时可以不需要sink或者source
The Kafka channel can be used for multiple scenarios:
- With Flume source and sink - it provides a reliable and highly available channel for events
- With Flume source and interceptor but no sink - it allows writing Flume events into a Kafka topic, for use by other apps
- With Flume sink, but no source - it is a low-latency, fault tolerant way to send events from Kafka to Flume sinks such as HDFS, HBase or Solr
kafka作为sink时
a1.sources.r1.type = spooldir
a1.sources.r1.channels = c1
a1.sources.r1.spoolDir = /home/hadoop/flume/test/logs/kfksink
a1.sources.r1.deletePolicy = immediate
a1.sources.r1.fileHeader = true a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = flumesink
a1.sinks.k1.kafka.bootstrap.servers = mini1:
a1.sinks.k1.kafka.flumeBatchSize =
a1.sinks.k1.kafka.producer.acks =
a1.sinks.k1.kafka.producer.linger.ms =
#压缩
a1.sinks.ki.kafka.producer.compression.type = snappy
此时打开kafka消费程序
Properties props = new Properties();
props.load(TestConsumer.class.getClass().getResourceAsStream("/kfkConsumer.properties"));
KafkaConsumer<Integer, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("flumesink"));
while (true) {
ConsumerRecords<Integer, String> records = consumer.poll();
for (ConsumerRecord<Integer, String> record : records) {
System.out.print("Thread : " + Thread.currentThread().getName());
System.out.printf(" offset = %d, key = %s, value = %s, partition = %d %n", record.offset(), record.key(), record.value(), record.partition());
}
consumer.commitSync();
} }
配置文件来源于http://flume.apache.org/FlumeUserGuide.html
flume+kafka的更多相关文章
- 简单测试flume+kafka+storm的集成
集成 Flume/kafka/storm 是为了收集日志文件而引入的方法,最终将日志转到storm中进行分析.storm的分析方法见后面文章,这里只讨论集成方法. 以下为具体步骤及测试方法: 1.分别 ...
- 【转】flume+kafka+zookeeper 日志收集平台的搭建
from:https://my.oschina.net/jastme/blog/600573 flume+kafka+zookeeper 日志收集平台的搭建 收藏 jastme 发表于 10个月前 阅 ...
- hadoop 之 kafka 安装与 flume -> kafka 整合
62-kafka 安装 : flume 整合 kafka 一.kafka 安装 1.下载 http://kafka.apache.org/downloads.html 2. 解压 tar -zxvf ...
- Flume+Kafka+Strom基于伪分布式环境的结合使用
目录: 一.Flume.Kafka.Storm是什么,如何安装? 二.Flume.Kafka.Storm如何结合使用? 1) 原理是什么? 2) Flume和Kafka的整合 3) Kafka和St ...
- flume+kafka (分区实现 默认单分区)
这篇文章主要是log4j+flume+kafka的内容 首先从从下面的地址下载flume+kafka的插件包 https://github.com/beyondj2ee/flumeng-kafka-p ...
- Flume - Kafka日志平台整合
1. Flume介绍 Flume是Cloudera提供的一个高可用的,高可靠的,分布式的海量日志采集.聚合和传输的系统,Flume支持在日志系统中定制各类数据发送方,用于收集数据:同时,Flume提供 ...
- Flume+Kafka+Storm+Hbase+HDSF+Poi整合
Flume+Kafka+Storm+Hbase+HDSF+Poi整合 需求: 针对一个网站,我们需要根据用户的行为记录日志信息,分析对我们有用的数据. 举例:这个网站www.hongten.com(当 ...
- Flume+Kafka+Storm整合
Flume+Kafka+Storm整合 1. 需求: 有一个客户端Client可以产生日志信息,我们需要通过Flume获取日志信息,再把该日志信息放入到Kafka的一个Topic:flume-to-k ...
- 大数据处理框架之Strom:Flume+Kafka+Storm整合
环境 虚拟机:VMware 10 Linux版本:CentOS-6.5-x86_64 客户端:Xshell4 FTP:Xftp4 jdk1.8 storm-0.9 apache-flume-1.6.0 ...
- Flume+Kafka整合
脚本生产数据---->flume采集数据----->kafka消费数据------->storm集群处理数据 日志文件使用log4j生成,滚动生成! 当前正在写入的文件在满足一定的数 ...
随机推荐
- Java里面类型转换总结
1.String 转 int int i = Integer.valueOf(my_str).intValue(); int i = Integer.parseInt(str); 2.String 转 ...
- Java中的初始化顺序(静态成员、静态初始化块,普通成员、普通初始化块、构造函数)
本文链接 http://blog.csdn.net/xiaodongrush/article/details/29170099 參考文章 http://my.oschina.net/le ...
- 使用js的indexOf,lastIndexOf,slice三函数轻易得到url的服务器,路径和页名
js的indexOf,lastIndexOf,slice能帮我们在js字符串处理时少走一些弯路. 程序如下: var url="http://www.cnblogs.com/xiandeda ...
- HTTP浏览器缓存机制
来自:http://kb.cnblogs.com/page/165307/ 浏览器缓存机制 浏览器缓存机制,其实主要就是HTTP协议定义的缓存机制(如: Expires: Cache-control等 ...
- Android高效异步图片加载框架
概述 Android高效异步图片加载框架:一个高效的异步加载显示的图片加载框架,同时具备图片压缩,缓存机制等特性. 详细 代码下载:http://www.demodashi.com/demo/1214 ...
- 为什么我没有拔出钥匙 ——开锁引发的程序bug解决方案的思考
http://blog.csdn.net/wojiushiwo987/article/details/8851204为什么我没有拔出钥匙 ——开 ...
- Android 5.0最应该实现的8个期望
毫无疑问,Android 5 将是令人兴奋的操作系统,因为 Android4.0 至 4.4 版本之间并没有显著的差异,显然谷歌会在 5.0 版本中进行一些较大幅度的革新.那么,代号为“柠檬芝士蛋糕” ...
- linux配置java环境变量(详细) [转]
一. 解压安装jdk 在shell终端下进入jdk-6u14-linux-i586.bin文件所在目录, 执行命令 ./jdk-6u14-linux-i586.bin 这时会出现一段协议,连继敲回车, ...
- 如何将.sof转换成.jic
因为不同版本的QUARTUS II可能界面稍有差异,因此就不做截图演示了,只说操作步骤: 1.通过综合生成包含FPGA配置数据的.sof文件 2.选择转换编程文件,菜单File->convert ...
- unity, stateMachine, change state name
如图,假设wingsLayer下有个state,叫“New State“,现在想给它改名,则一定要保证当前wingsLayer为选中状态,然后才能在Inspector中可以为其改名.否则若当前选中的不 ...