https://community.hortonworks.com/questions/27187/using-kafkabolt-to-write-to-a-kafka-topic.html

--------------------------------------------------------------------------------------------------------------

I'm building a kafka and storm based streaming application based on the usecase:

1. Application produces JSON message to a kafka topic

2. Kafka storm spout ingests the message and does processing

3. Produce output to another kafka topic using a storm bolt

Here are components and versions I'm using:

Storm: 0.10.0

Kafka_2.10: 0.9.0.1

HDP: 2.3.4

storm-kafka: 0.10.0

I accomplished steps 1 and 2 using "storm.kafka.KafkaSpout" that comes with storm-kafka

I'm trying to use "storm.kafka.KafkaBolt" to write processed data to a topic and I couldn't figure out exactly how to do it. Here is the code snippet:

  

Properties props = new Properties();
props.put("metadata.broker.list", "192.168.56.102:9092");
props.put("request.required.acks", "1");
props.put("serializer.class", "kafka.serializer.StringEncoder"); Config conf = new Config();
conf.put(KafkaBolt.KAFKA_BROKER_PROPERTIES, props); KafkaBolt kafkaBolt = new KafkaBolt().withTopicSelector(new DefaultTopicSelector("OUTBOUND_TOPIC"))
                      .withTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper()); builder.setBolt("kafka_outbound_bolt", kafkaBolt, 3).shuffleGrouping("process_bolt");

  

"process_bolt" sends out a tuple that is a serializable domain model object (SubModel.java). I would like to send SubModel.toString() data to the outbound topic. I'm running storm in local cluster mode connected to kafka in a local VM.

  1. Am I doing anything wrong ?
  2. How do I use "FieldNameBasedTupleToKafkaMapper"?

Using KafkaBolt to write to a kafka topic的更多相关文章

  1. Kafka Topic Partition Replica Assignment实现原理及资源隔离方案

    本文共分为三个部分:   Kafka Topic创建方式 Kafka Topic Partitions Assignment实现原理 Kafka资源隔离方案   1. Kafka Topic创建方式 ...

  2. Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure is thrown while creating kafka topic

    Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentica ...

  3. 用canal同步binlog到kafka,spark streaming消费kafka topic乱码问题

    canal 1.1.1版本之后, 默认支持将canal server接收到的binlog数据直接投递到MQ, 目前默认支持的MQ系统有kafka和RocketMQ. 在投递的时候我们使用的是非压平的消 ...

  4. Kafka Topic Api

    Pom: <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10 ...

  5. [Flume][Kafka]Flume 与 Kakfa结合例子(Kakfa 作为flume 的sink 输出到 Kafka topic)

    Flume 与 Kakfa结合例子(Kakfa 作为flume 的sink 输出到 Kafka topic) 进行准备工作: $sudo mkdir -p /flume/web_spooldir$su ...

  6. 手动删除Kafka Topic

    一.删除Kafka topic 运行./bin/kafka-topics  --delete --zookeeper [zookeeper server]  --topic [topic name]: ...

  7. kafka topic制定规则

    kafka topic的制定,我们要考虑的问题有很多,比如生产环境中用几备份.partition数目多少合适.用几台机器支撑数据量,这些方面如何去考量?笔者根据实际的维护经验,写一些思考,希望大家指正 ...

  8. kafka topic 完全删除

    kafka topic 完全删除   1.自动删除脚本(得配置server.properties 中 delete.topic.enable=true) ./kafka-topics.sh --zoo ...

  9. kafka topic查看删除

    1,查看kafka topic列表,使用--list参数 >bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --list __consumer_of ...

随机推荐

  1. warning: here-document at line 7 delimited by end-of-file (wanted `rui')

  2. Django day07 (二)单表操作

    单表操作 -mysql数据库:settings里配置: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME ...

  3. SnackDown Online Pre-elimination round A

    1. 应该n是偶数,就行吧.应该判断1个人,只能出现一次吧. #include<bits/stdc++.h> #define pb push_back typedef long long ...

  4. 总结Linq或者lamdba的写法

    var head = new OmsEcorderHead { PkEcorderHead = OrderHeadId, AppId = appid, Integral = Convert.ToDec ...

  5. iconfont在ios(safari)中的坑

    最近公司决定将项目图标整体迁移到iconfont,按网上常规方法,在安卓.pc端都没问题,唯独在ios的safari浏览器及微信内置浏览器中,iconfont始终在正常位置向下偏移,导致图标错乱. 网 ...

  6. Visual Studio UI Automation 学习(一)

    这几天需要研究自动化测试工具,因为团队开发使用visual studio,所以需要研究一下Visual studio自带的框架. 刚开始安装的时候,没有选自定义安装,所以安装完成后没有找到UI Aut ...

  7. Window8.1下安装Matplotlib库

    有两种方法: 直接选用一些预打包库软件,如WinPython, Python(x,y), Enthought Canopy, or Continuum Anaconda.这些软件中已包含有Matplo ...

  8. iOS-如何返回某个字符串的拼音助记码

    我也是看了网上的一个示例代码后,在它的基础上进行的修改.因为项目上会用到,我相信很多人的项目上也会用到.所以实现后,也赶紧分享出来,希望后来人不需要花费时间了. 提示:这里用到了正则表达式,使用了一个 ...

  9. logging模块-logging.basicConfig、logger.setLevel、handler.setLevel优先级

    logging.basicConfig < handler.setLevel < logger.setLevel 1.脚本中没有配置logger.setLevel会使用handler.se ...

  10. Git更新代码

    此次更新,创建新的文件与文件夹,并且删除了原有文件 通过“git status” 命令查看当前变更.通过变更信息可以看出,删除了test_case.py文件.这个删除只是在项目目录下进行删除,Git对 ...