Using KafkaBolt to write to a kafka topic
https://community.hortonworks.com/questions/27187/using-kafkabolt-to-write-to-a-kafka-topic.html
--------------------------------------------------------------------------------------------------------------
I'm building a kafka and storm based streaming application based on the usecase:
1. Application produces JSON message to a kafka topic
2. Kafka storm spout ingests the message and does processing
3. Produce output to another kafka topic using a storm bolt
Here are components and versions I'm using:
Storm: 0.10.0
Kafka_2.10: 0.9.0.1
HDP: 2.3.4
storm-kafka: 0.10.0
I accomplished steps 1 and 2 using "storm.kafka.KafkaSpout" that comes with storm-kafka
I'm trying to use "storm.kafka.KafkaBolt" to write processed data to a topic and I couldn't figure out exactly how to do it. Here is the code snippet:
Properties props = new Properties();
props.put("metadata.broker.list", "192.168.56.102:9092");
props.put("request.required.acks", "1");
props.put("serializer.class", "kafka.serializer.StringEncoder"); Config conf = new Config();
conf.put(KafkaBolt.KAFKA_BROKER_PROPERTIES, props); KafkaBolt kafkaBolt = new KafkaBolt().withTopicSelector(new DefaultTopicSelector("OUTBOUND_TOPIC"))
.withTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper()); builder.setBolt("kafka_outbound_bolt", kafkaBolt, 3).shuffleGrouping("process_bolt");
"process_bolt" sends out a tuple that is a serializable domain model object (SubModel.java). I would like to send SubModel.toString() data to the outbound topic. I'm running storm in local cluster mode connected to kafka in a local VM.
- Am I doing anything wrong ?
- How do I use "FieldNameBasedTupleToKafkaMapper"?
Using KafkaBolt to write to a kafka topic的更多相关文章
- Kafka Topic Partition Replica Assignment实现原理及资源隔离方案
本文共分为三个部分: Kafka Topic创建方式 Kafka Topic Partitions Assignment实现原理 Kafka资源隔离方案 1. Kafka Topic创建方式 ...
- Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure is thrown while creating kafka topic
Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentica ...
- 用canal同步binlog到kafka,spark streaming消费kafka topic乱码问题
canal 1.1.1版本之后, 默认支持将canal server接收到的binlog数据直接投递到MQ, 目前默认支持的MQ系统有kafka和RocketMQ. 在投递的时候我们使用的是非压平的消 ...
- Kafka Topic Api
Pom: <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10 ...
- [Flume][Kafka]Flume 与 Kakfa结合例子(Kakfa 作为flume 的sink 输出到 Kafka topic)
Flume 与 Kakfa结合例子(Kakfa 作为flume 的sink 输出到 Kafka topic) 进行准备工作: $sudo mkdir -p /flume/web_spooldir$su ...
- 手动删除Kafka Topic
一.删除Kafka topic 运行./bin/kafka-topics --delete --zookeeper [zookeeper server] --topic [topic name]: ...
- kafka topic制定规则
kafka topic的制定,我们要考虑的问题有很多,比如生产环境中用几备份.partition数目多少合适.用几台机器支撑数据量,这些方面如何去考量?笔者根据实际的维护经验,写一些思考,希望大家指正 ...
- kafka topic 完全删除
kafka topic 完全删除 1.自动删除脚本(得配置server.properties 中 delete.topic.enable=true) ./kafka-topics.sh --zoo ...
- kafka topic查看删除
1,查看kafka topic列表,使用--list参数 >bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --list __consumer_of ...
随机推荐
- 第2章 安装Nodejs 2-4 Linux下安装Nodejs
linux下编译安装Nodejs GCC和G++分别是GNU的C和C++编译器.它们在执行编译工作的时候把源代码通过预处理转化成汇编语言生成.i后缀的文件,再由汇编变成目标机器代码,最后连接目标代码 ...
- 线性预测与Levinson-Durbin算法实现
在学习信号处理的时候,线性预测是一个比较难理解的知识点,为了加快很多朋友的理解,这里给出Levinson-Durbin算法的线性预测实现和一个测试Demo,Demo中很明确的把输入信号.预测信号.预测 ...
- BZOJ 4525 二分
思路: 满足二分性质... 二分一下 就完了 //By SiriusRen #include <cstdio> #include <algorithm> using ...
- BZOJ 4488/4052 gcd
思路: 一开始 我是想 对于固定的左端点 从左到右 最多有 log种取值 且单调递减 那不妨倍增预处理+二分GCD在哪变了.. 复杂度O(nlog^2n) gcd最多log种取值.. 好了我们可以 ...
- doctype声明 过渡transitional 严格strict 框架frameset
DOCTYPE是document type(文档类型)的简写,用来说明你用的XHTML或者HTML是什么版本. 其中的DTD(例如上例中的xhtml1-transitional.dtd)叫文档类型定义 ...
- HBase编程 API入门系列之delete(客户端而言)(3)
心得,写在前面的话,也许,中间会要多次执行,连接超时,多试试就好了. 前面的基础,如下 HBase编程 API入门系列之put(客户端而言)(1) HBase编程 API入门系列之get(客户端而言) ...
- Spring Boot (3) 热部署devtools
热部署:当发现程序修改时自动启动应用程序. spring boot为开发者提供了一个名为spring-boot-devtools的模块来使sring boot应用支持热部署,提高开发者的开发效率,无需 ...
- java线程中断
public void Thread.interrupt() // 无返回值 public boolean Thread.isInterrupted() // 有返回值 public static b ...
- 搭建Hadoop所遇过的坑
问题1: 报错信息如下: Container exited with a non-zero exit code 143 Killed by external signal 解决方案: 分配的资源不够, ...
- cannot find Toolkit in /usr/local/cuda-8.0
对于新版本ubuntukylin17.04安装CUDA时出现 cannot find Toolkit in /usr/local/cuda-8.0 问题 尝试了各种方法,均不妥当...... 参考:h ...