版本 0.9.2

创建topic

bin/kafka-topics.sh --create --topic topic_name --partition 6 --replication-factor 1 -zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181

开启console consumer查看消息

bin/kafka-console-consumer.sh --topic rt_live_pcweb -zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --from-begin

改变topic的partition数, kafka topic的partition数只能增加,不能减少。

[root@ip-10-27-100-100 kafka]# bin/kafka-topics.sh --zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --alter --topic rt_live_pcweb --partitions 8

WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected

Error while executing topic command The number of partitions for a topic can only be increased

kafka.admin.AdminOperationException: The number of partitions for a topic can only be increased

at kafka.admin.AdminUtils$.addPartitions(AdminUtils.scala:98)

at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:109)

at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:93)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:93)

at kafka.admin.TopicCommand$.main(TopicCommand.scala:52)

at kafka.admin.TopicCommand.main(TopicCommand.scala)

迁移topic的partition到指定的broker上

1. 将需要迁移的topic按如下格式写成json文件

[root@ip-10-27-111-107 kafka]# cat topics-to-move.json

{"topics":

[{"topic": "rt_live_mobile_new"}],

"version":1

}

2. 执行如下命令,其中 0,1,2,3 为要迁移到的broker id

[root@ip-10-27-111-107 kafka]# bin/kafka-reassign-partitions.sh --zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate

Current partition replica assignment

{"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[6]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[7]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]}]}

Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[3]}]}

3. 将上面命令的输出(红色部门)存入json文件reassignment.json中

4. 执行如下命令,即将partition都迁移至broker 0,1,2,3 服务器上。

[root@ip-10-27-111-107 kafka]# bin/kafka-reassign-partitions.sh --zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --reassignment-json-file reassignment.json --execute

Current partition replica assignment

{"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[6]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[7]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]}]}

Save this to use as the --reassignment-json-file option during rollback

Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]}]}

5.隔一会之后(partition迁移需要一段时间),执行describe查看,partition已经迁移成功

[root@ip-10-27-111-107 kafka]# bin/kafka-topics.sh --topic rt_live_mobile_new --describe -zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181

Topic:rt_live_mobile_new PartitionCount:12 ReplicationFactor:1 Configs:

Topic: rt_live_mobile_new Partition: 0 Leader: 2 Replicas: 2 Isr: 2

Topic: rt_live_mobile_new Partition: 1 Leader: 3 Replicas: 3 Isr: 3

Topic: rt_live_mobile_new Partition: 2 Leader: 0 Replicas: 0 Isr: 0

Topic: rt_live_mobile_new Partition: 3 Leader: 1 Replicas: 1 Isr: 1

Topic: rt_live_mobile_new Partition: 4 Leader: 2 Replicas: 2 Isr: 2

Topic: rt_live_mobile_new Partition: 5 Leader: 3 Replicas: 3 Isr: 3

Topic: rt_live_mobile_new Partition: 6 Leader: 0 Replicas: 0 Isr: 0

Topic: rt_live_mobile_new Partition: 7 Leader: 1 Replicas: 1 Isr: 1

Topic: rt_live_mobile_new Partition: 8 Leader: 2 Replicas: 2 Isr: 2

Topic: rt_live_mobile_new Partition: 9 Leader: 3 Replicas: 3 Isr: 3

Topic: rt_live_mobile_new Partition: 10 Leader: 0 Replicas: 0 Isr: 0

Topic: rt_live_mobile_new Partition: 11 Leader: 1 Replicas: 1 Isr: 1

[bigdata] kafka基本命令 -- 迁移topic partition到指定的broker的更多相关文章

  1. kafka学习(四)-Topic & Partition

    topic中partition存储分布 Topic在逻辑上可以被认为是一个queue.每条消费都必须指定它的topic,可以简单理解为必须指明把这条消息放进哪个queue里.为了使得 Kafka的吞吐 ...

  2. Kafka Topic Partition Replica Assignment实现原理及资源隔离方案

    本文共分为三个部分:   Kafka Topic创建方式 Kafka Topic Partitions Assignment实现原理 Kafka资源隔离方案   1. Kafka Topic创建方式 ...

  3. Kafka Topic Partition Offset 这一长串都是啥?

    摘要:Offset 偏移量,是针对于单个partition存在的概念. 本文分享自华为云社区<Kafka Topic Partition Offset 这一长串都是啥?>,作者: gent ...

  4. 删除kafka的topic及kafka基本命令

    kafka的topic默认是不允许被删除的,删除后在topic后会出现”marked for deletion”字样,实际并未删除,现在创建同样的topic会提示topic已经存在. 解决办法: se ...

  5. kafka数据迁移实践

    欢迎大家前往云+社区,获取更多腾讯海量技术实践干货哦~ 作者:mikealzhou 本文重点介绍kafka的两类常见数据迁移方式:1.broker内部不同数据盘之间的分区数据迁移:2.不同broker ...

  6. Kafka数据迁移

    1.概述 Kafka的使用场景非常广泛,一些实时流数据业务场景,均依赖Kafka来做数据分流.而在分布式应用场景中,数据迁移是一个比较常见的问题.关于Kafka集群数据如何迁移,今天笔者将为大家详细介 ...

  7. kafka的迁移干货

    随着业务的发展, 服务器所在网段/机群不允许kafka继续保留在那, 需要移动到先机器上. 哎呀上面是废话,总的说就是: 2台老kafka不要了,数据要迁移到新的2台kafka上面.要求数据不丢失 通 ...

  8. Kafka动态增加Topic的副本

    一.kafka的副本机制 由于Producer和Consumer都只会与Leader角色的分区副本相连,所以kafka需要以集群的组织形式提供主题下的消息高可用.kafka支持主备复制,所以消息具备高 ...

  9. Kafka数据迁移MaxCompute最佳实践

    摘要: 本文向您详细介绍如何使用DataWorks数据同步功能,将Kafka集群上的数据迁移到阿里云MaxCompute大数据计算服务. 前提条件 搭建Kafka集群 进行数据迁移前,您需要保证自己的 ...

随机推荐

  1. jQuery中的$.extend方法来扩展JSON对象及合并,方便调用对象方法

    $.extend方法可以扩展JSON对象,用一个或多个其他对象来扩展一个对象,返回被扩展的对象. 例一 合并 settings 和 options,修改并返回 settings var setting ...

  2. [FromBody]与[FromUrl]

    我们都知道,前台请求后台控制的方法有get方法和post方法两种, get:只支持ulr传数据,不管你是手动把参数拼接在Url里面还是写在data里面,只要是用get方法,都会自动绑定到url里面的形 ...

  3. Mysql(windows)安装

    h3 { color: rgb(255, 255, 255); background-color: rgb(30,144,255); padding: 3px; margin: 10px 0px } ...

  4. jquery toggle方法使用出错?请看这里-遁地龙卷风

    这个函数在1.9之前和1.9之后的用法大不相同 1 1.9之间,用于点击元素时函数的轮流执行 toggle(function() { alert(1);//1,3,5,7... },function( ...

  5. (十一)Maven远程仓库的各种配置

    1.远程仓库的配置 在平时的开发中,我们往往不会使用默认的中央仓库,默认的中央仓库访问的速度比较慢,访问的人或许很多,有时候也无法满足我们项目的需求,可能项目需要的某些构件中央仓库中是没有的,而在其他 ...

  6. tomcat 应用部署方式(转)

    tomcat部署web应用的三种方式 1.直接放到Webapps目录下    Tomcat的Webapps目录是Tomcat默认的应用目录,当服务器启动时,会加载所有这个目录下的应用.也可以将JSP程 ...

  7. lnmp 预设iptables设置

    「LNMP」iptables初始配置   首先使用命令iptables -P INPUT ACCEPT允许所有连接,否则容易把自己关在外边.然后使用iptables -F;iptables -X;ip ...

  8. 6-3 bash脚本编程之五 字符串测试及for循环

    1. 字符测试 ==:等号两边要有空格,否则会被认为是赋值. !=:  测试是否相等,记住如果不等为真,等为假. -n string: 测试指定字符串是否为空,空位真,不空为假. -s string: ...

  9. 微软本月将推Win10 望打破Win8所带来差评影响

    系统妈 据香港“文汇报”7月15日报道,美国电脑巨擘微软13日宣布,新视窗系统10(Windows 10)将于7月29日推出.微软高层迈赫迪在网志表示,届时全球将有13个城市举办特别活动,表扬参与Wi ...

  10. 原: 安装VMtools过程流水帐

    以下基于 vm12.0.0 1. 一定要打开虚拟机的 cd设置 2. 然后 cd   '/medal/VMware tools '   (注意一定要加 '', 因为VMware tools 有空格) ...