using kafkacat reset kafka offset
1. install kafkacat
Ubuntu
apt-get install kafkacat
CentOS
install deepenency
yum install librdkafka-devel
download source from github
build source on centos
./configure <usual-configure-options>
make
sudo make install
2. watch the target topic data
lenmom@M1701:~/workspace/software/confluent-community-5.1.-2.11$ bin/kafka-console-consumer --bootstrap-server localhost: --from-beginning --topic connect-offsets --property print.key=true
["jdbc_source_inventory_customers",{"query":"query"}] {"incrementing":}
there is only one record in kafka topic connect-offsets.
3. dump the record from topic
lenmom@M1701:~/workspace/software/confluent-community-5.1.-2.11$ kafkacat -b localhost: -t connect-offsets -C -K# -o-
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
["jdbc_source_inventory_customers",{"query":"query"}]#{"incrementing":}
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
% Reached end of topic connect-offsets [] at offset
the value:
["jdbc_source_inventory_customers",{"query":"query"}]#{"incrementing":1005}
is what we want!
4. use the value get in step 3 as template to and send to the topic again
lenmom@M1701:~/workspace/software/confluent-community-5.1.-2.11$ echo '["jdbc_source_inventory_customers",{"query":"query"}]#{"incrementing":1}' | \
> kafkacat -b localhost: -t connect-offsets -P -Z -K#
here, we modify the incrementing value from 1005 to 1.
for timestamp+increment
echo '["jdbc_source_inventory_orders",{"query":"query"}]#{"timestamp_nanos":0,"incrementing":0,"timestamp":0}' | \
kafkacat -b localhost: -t connect-offsets -P -Z -K#
5. watch the topic again
lenmom@M1701:~/workspace/software/confluent-community-5.1.-2.11$ bin/kafka-console-consumer --bootstrap-server localhost: --from-beginning --topic connect-offsets --property print.key=true
["jdbc_source_inventory_customers",{"query":"query"}] {"incrementing":}
["jdbc_source_inventory_customers",{"query":"query"}] {"incrementing":}
we can see, there are two values with the same key in the topic now.
refernce
https://docs.confluent.io/current/app-development/kafkacat-usage.html
using kafkacat reset kafka offset的更多相关文章
- Kafka Offset相关命令总结
Kafka Offset相关命令总结 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.查询topic的offset的范围 1>.查询某个topic的offset的最小值 [ ...
- kafka集群监控工具之三--kafka Offset Monitor
1.介绍 一般情况下,功能简单的kafka项目 使用运维命令+kafka Offset Monitor 就足够用了. 2.使用2.1 部署 github下载jar包 KafkaOffsetMonit ...
- Kafka Offset 1
Kafka Offset Storage 1.概述 目前,Kafka 官网最新版[0.10.1.1],已默认将消费的 offset 迁入到了 Kafka 一个名为 __consumer_offse ...
- Spark createDirectStream 维护 Kafka offset(Scala)
createDirectStream方式需要自己维护offset,使程序可以实现中断后从中断处继续消费数据. KafkaManager.scala import kafka.common.TopicA ...
- Kafka Offset Storage
1.概述 目前,Kafka 官网最新版[0.10.1.1],已默认将消费的 offset 迁入到了 Kafka 一个名为 __consumer_offsets 的Topic中.其实,早在 0.8.2. ...
- kafka offset 设置
from kafka import KafkaConsumer from kafka import TopicPartition from kafka.structs import OffsetAnd ...
- 关于 Kafka offset
查询topic的offset的范围 用下面命令可以查询到topic:Mytopic broker:SparkMaster:9092的offset的最小值: bin/kafka-run-class.sh ...
- kafka offset的存储问题
注意:从kafka-0.9版本及以后,kafka的消费者组和offset信息就不存zookeeper了,而是存到broker服务器上,所以,如果你为某个消费者指定了一个消费者组名称(group.id) ...
- kafka offset存储
存储方式 方式 方式来源 存储位置 自动提交 kafka kafka 异步提交 kafka kafka checkpoint spark streaming hdfs hbase存储 程序开发 hba ...
随机推荐
- 用BCB 画 Code128 B模式条码
//--------------------------------------------------------------------------- #include <vcl.h> ...
- vue路由分区结合require.context使用
1.先说路由分区 在router文件夹下新建你要分区的模块例如 登录 订单模块 新建文件 logn.router.js order.router.js 代码如下: export default { ...
- java中的finally用法总结
不管 try 语句块正常结束还是异常结束,finally 语句块是保证要执行的.如果 try 语句块正常结束,那么在 try 语句块中的语句都执行完之后,再执行 finally 语句块.如果 try ...
- Codeforces Round #533 (Div. 2) E. Helping Hiasat(最大独立集)
题目链接:https://codeforces.com/contest/1105/problem/E 题意:有 n 个事件,op = 1 表示我可以修改昵称,op = 2 表示一个名为 s_i 的朋友 ...
- jquery仿排列顺序,变换颜色更换class
ps:箭头是字体图标,需要的可以去阿里字体下载,也可自己替换图片 点击之后的 代码部分 <ul class="Ep_sxxz"> <span class=&quo ...
- BZOJ3678 wangxz与OJ (平衡树 无旋treap)
题面 维护一个序列,支持以下操作: 1.在某个位置插入一段值连续的数. 2.删除在当前序列位置连续的一段数. 3.查询某个位置的数是多少. 题解 显然平衡树,一个点维护一段值连续的数,如果插入或者删除 ...
- php面向对象之数据隐藏
什么是数据隐藏? 看到这个有的人会觉得挺不理解的.在前面的文章中,介绍类的时候,我们说定义变量用的关键词是public,但是不止这一个,还有public.private.protected.stati ...
- TensorFlow(十):卷积神经网络实现手写数字识别以及可视化
上代码: import tensorflow as tf from tensorflow.examples.tutorials.mnist import input_data mnist = inpu ...
- P5346 【XR-1】柯南家族(后缀数组+主席树)
题目 P5346 [XR-1]柯南家族 做法 聪明性是具有传递性的,且排列是固定的 那么先预处理出每个点的名次,用主席树维护\(k\)大值 一眼平衡树,遍历的同时插入\(O(log^2n)\),总时间 ...
- TCP/IP 这猝不及防的爱情
前言 前几天看了老刘的一篇文章,TCP/IP 大明邮差.正好最近也在读<计算机自顶向下>一书 心血来潮,想写一个女版的TCP/IP 正文 一天,我正在百花会上赏花,赏着赏着,就出现了一个令 ...