https://community.hortonworks.com/questions/27187/using-kafkabolt-to-write-to-a-kafka-topic.html

--------------------------------------------------------------------------------------------------------------

I'm building a kafka and storm based streaming application based on the usecase:

1. Application produces JSON message to a kafka topic

2. Kafka storm spout ingests the message and does processing

3. Produce output to another kafka topic using a storm bolt

Here are components and versions I'm using:

Storm: 0.10.0

Kafka_2.10: 0.9.0.1

HDP: 2.3.4

storm-kafka: 0.10.0

I accomplished steps 1 and 2 using "storm.kafka.KafkaSpout" that comes with storm-kafka

I'm trying to use "storm.kafka.KafkaBolt" to write processed data to a topic and I couldn't figure out exactly how to do it. Here is the code snippet:

  

Properties props = new Properties();
props.put("metadata.broker.list", "192.168.56.102:9092");
props.put("request.required.acks", "1");
props.put("serializer.class", "kafka.serializer.StringEncoder"); Config conf = new Config();
conf.put(KafkaBolt.KAFKA_BROKER_PROPERTIES, props); KafkaBolt kafkaBolt = new KafkaBolt().withTopicSelector(new DefaultTopicSelector("OUTBOUND_TOPIC"))
                      .withTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper()); builder.setBolt("kafka_outbound_bolt", kafkaBolt, 3).shuffleGrouping("process_bolt");

  

"process_bolt" sends out a tuple that is a serializable domain model object (SubModel.java). I would like to send SubModel.toString() data to the outbound topic. I'm running storm in local cluster mode connected to kafka in a local VM.

  1. Am I doing anything wrong ?
  2. How do I use "FieldNameBasedTupleToKafkaMapper"?

Using KafkaBolt to write to a kafka topic的更多相关文章

  1. Kafka Topic Partition Replica Assignment实现原理及资源隔离方案

    本文共分为三个部分:   Kafka Topic创建方式 Kafka Topic Partitions Assignment实现原理 Kafka资源隔离方案   1. Kafka Topic创建方式 ...

  2. Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure is thrown while creating kafka topic

    Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentica ...

  3. 用canal同步binlog到kafka,spark streaming消费kafka topic乱码问题

    canal 1.1.1版本之后, 默认支持将canal server接收到的binlog数据直接投递到MQ, 目前默认支持的MQ系统有kafka和RocketMQ. 在投递的时候我们使用的是非压平的消 ...

  4. Kafka Topic Api

    Pom: <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10 ...

  5. [Flume][Kafka]Flume 与 Kakfa结合例子(Kakfa 作为flume 的sink 输出到 Kafka topic)

    Flume 与 Kakfa结合例子(Kakfa 作为flume 的sink 输出到 Kafka topic) 进行准备工作: $sudo mkdir -p /flume/web_spooldir$su ...

  6. 手动删除Kafka Topic

    一.删除Kafka topic 运行./bin/kafka-topics  --delete --zookeeper [zookeeper server]  --topic [topic name]: ...

  7. kafka topic制定规则

    kafka topic的制定,我们要考虑的问题有很多,比如生产环境中用几备份.partition数目多少合适.用几台机器支撑数据量,这些方面如何去考量?笔者根据实际的维护经验,写一些思考,希望大家指正 ...

  8. kafka topic 完全删除

    kafka topic 完全删除   1.自动删除脚本(得配置server.properties 中 delete.topic.enable=true) ./kafka-topics.sh --zoo ...

  9. kafka topic查看删除

    1,查看kafka topic列表,使用--list参数 >bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --list __consumer_of ...

随机推荐

  1. Help Me with the Game(模拟)

    http://poj.org/problem?id=2996 #include<stdio.h> #include<string.h> ][]; void find1(char ...

  2. [Apple开发者帐户帮助]六、配置应用服务(2)创建DeviceCheck私钥

    要验证与DeviceCheck服务的通信,您将使用启用了DeviceCheck的私钥. 首先创建并下载启用了DeviceCheck 的私钥.然后获取密钥标识符(kid)以创建JSON Web令牌(JW ...

  3. 【BZOJ4590】自动刷题机

    [思路分析] 比赛的时候想到了用二分+贪心,二分的部分与贪心的部分也写对了,但是由于数据范围未看没有开long long,且二分左端点赋值过小导致WA掉 正解:二分+贪心 二分代码的长度,贪心判断能否 ...

  4. B - Is your horseshoe on the other hoof?

    Problem description Valera the Horse is going to the party with friends. He has been following the f ...

  5. SQL Server中char与varchar数据类型区别

    在SQL Server中char类型的长度是不可变的,而varchar的长度是可变的 . 存入数据时: 如果数据类型为char时,当定义一个字段固定长度时,如果存进去数据长度小于char的长度,那么存 ...

  6. Flex使用总结

    最近做的项目因为对浏览器的兼容要求是IE10以上,所以大胆的使用了Flex布局,这里总结一些使用心得仅供参考. 一,Flex简单介绍 Flex是Flexible Box的缩写,意为”弹性布局”.任何一 ...

  7. Project Management -- How to use GitHub with Git

    如何通过Git使用GitHub ——PM曾子轩 从未使用过Git 一.从官网下载Git 此部分略 二.用Git连接上GitHub(为保持完整性,此部分引用博客:https://www.cnblogs. ...

  8. 【Oracle】进入sqlplus 删除键backspace时出现^H

    当oracle进入sqlplus后,输入命令时候出现错误,我们按平时的习惯使用backspace键删除错误信息,此时会出现^H 解决办法:进入sqlplus之前,使用stty erase '^H'命令 ...

  9. java编程题(一)

    [程序1]    题目:古典问题:有一对兔子,从出生后第3个月起每个月都生一对兔子,小兔子长到第三个月后每个月又生一对兔子,假如兔子都不死,问每个月的兔子总数为多少? //这是一个菲波拉契数列问题 p ...

  10. Visual Studio UI Automation 学习(三)

    昨天了解到UI Automation是微软的.Net Framework框架里的4个DLL文件,可以在Visual studio里写代码时引入引用和引用命名空间.然后去写自动化代码. 今天本来是跟着一 ...