Using KafkaBolt to write to a kafka topic
https://community.hortonworks.com/questions/27187/using-kafkabolt-to-write-to-a-kafka-topic.html
--------------------------------------------------------------------------------------------------------------
I'm building a kafka and storm based streaming application based on the usecase:
1. Application produces JSON message to a kafka topic
2. Kafka storm spout ingests the message and does processing
3. Produce output to another kafka topic using a storm bolt
Here are components and versions I'm using:
Storm: 0.10.0
Kafka_2.10: 0.9.0.1
HDP: 2.3.4
storm-kafka: 0.10.0
I accomplished steps 1 and 2 using "storm.kafka.KafkaSpout" that comes with storm-kafka
I'm trying to use "storm.kafka.KafkaBolt" to write processed data to a topic and I couldn't figure out exactly how to do it. Here is the code snippet:
Properties props = new Properties();
props.put("metadata.broker.list", "192.168.56.102:9092");
props.put("request.required.acks", "1");
props.put("serializer.class", "kafka.serializer.StringEncoder"); Config conf = new Config();
conf.put(KafkaBolt.KAFKA_BROKER_PROPERTIES, props); KafkaBolt kafkaBolt = new KafkaBolt().withTopicSelector(new DefaultTopicSelector("OUTBOUND_TOPIC"))
.withTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper()); builder.setBolt("kafka_outbound_bolt", kafkaBolt, 3).shuffleGrouping("process_bolt");
"process_bolt" sends out a tuple that is a serializable domain model object (SubModel.java). I would like to send SubModel.toString() data to the outbound topic. I'm running storm in local cluster mode connected to kafka in a local VM.
- Am I doing anything wrong ?
- How do I use "FieldNameBasedTupleToKafkaMapper"?
Using KafkaBolt to write to a kafka topic的更多相关文章
- Kafka Topic Partition Replica Assignment实现原理及资源隔离方案
本文共分为三个部分: Kafka Topic创建方式 Kafka Topic Partitions Assignment实现原理 Kafka资源隔离方案 1. Kafka Topic创建方式 ...
- Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure is thrown while creating kafka topic
Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentica ...
- 用canal同步binlog到kafka,spark streaming消费kafka topic乱码问题
canal 1.1.1版本之后, 默认支持将canal server接收到的binlog数据直接投递到MQ, 目前默认支持的MQ系统有kafka和RocketMQ. 在投递的时候我们使用的是非压平的消 ...
- Kafka Topic Api
Pom: <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10 ...
- [Flume][Kafka]Flume 与 Kakfa结合例子(Kakfa 作为flume 的sink 输出到 Kafka topic)
Flume 与 Kakfa结合例子(Kakfa 作为flume 的sink 输出到 Kafka topic) 进行准备工作: $sudo mkdir -p /flume/web_spooldir$su ...
- 手动删除Kafka Topic
一.删除Kafka topic 运行./bin/kafka-topics --delete --zookeeper [zookeeper server] --topic [topic name]: ...
- kafka topic制定规则
kafka topic的制定,我们要考虑的问题有很多,比如生产环境中用几备份.partition数目多少合适.用几台机器支撑数据量,这些方面如何去考量?笔者根据实际的维护经验,写一些思考,希望大家指正 ...
- kafka topic 完全删除
kafka topic 完全删除 1.自动删除脚本(得配置server.properties 中 delete.topic.enable=true) ./kafka-topics.sh --zoo ...
- kafka topic查看删除
1,查看kafka topic列表,使用--list参数 >bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --list __consumer_of ...
随机推荐
- 线性回归(最小二乘法、批量梯度下降法、随机梯度下降法、局部加权线性回归) C++
We turn next to the task of finding a weight vector w which minimizes the chosen function E(w). Beca ...
- 123D
后缀数组+单调栈 看了好长时间,最后看了张神的程序才搞懂 意思就是求所有子串*n*(n+1)/2 n是子串出现次数 事实上,lcp可以看成宽度为1,高度为lcp值的长方形,所有lcp放在一起就是一堆长 ...
- Node.js+express 4.x 入门笔记
一.新建node项目并实现访问 二.在express4.x下,让ejs模板文件,使用扩展名为html的文件 三.实现路由功能 四.session使用 五.页面访问控制及提示 六.代码下载地址 一.新建 ...
- Oracle Instant Client 安装配置
一.下载 下载地址:http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html 这是Ora ...
- sublime如何汉化
1.将sublime安装文件夹里面的defavlut.sublime-package这个文件zip解压. 2.然后查找到sublime-menu文件. 3.打开文件将json里面的caption里面的 ...
- oj测试点相关 (整理摘编)
Accepted 通过!(AC) Wrong Answer 答案错.(WA) Runtime Error ...
- BZOJ 4563 错排+高精度
思路: 把障碍移到对角线 就发现 这是个错位排列问题 用错排公式即可解 s[i]=(s[i-1]+s[i-2])*i //By SiriusRen #include <cstdio> #i ...
- 在linux系统中,使用tomcat的shutdown.sh脚本停止应用,但是进程还在的解决办法
基本原理为启动tomcat时记录启动tomcat的进程id(pid),关闭时强制杀死该进程 第一步 :vi 修改tomcat下bin/catalina.sh文件,增加几行脚本,主要是记录tomcat的 ...
- 通过委托事件实现winform窗体之间的互相刷新
新建winform窗体Form1和Form2; 接下来要通过点击Form2的按钮,实现Form1界面的同步刷新. 先展示一下最终效果: 1.Form1界面如下: 2.点击按钮弹出Form2,界面如下: ...
- Spring Boot (2) Restful风格接口
Rest接口 动态页面jsp早已过时,现在流行的是vuejs.angularjs.react等前端框架 调用 rest接口(json格式),如果是单台服务器,用动态还是静态页面可能没什么大区别,如果服 ...