参考官网site:

http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool

说明:

当我们对kafka集群扩容时,需要满足2点要求:

  1. 将指定topic迁移到集群内新增的node上。
  2. 将topic的指定partition迁移到新增的node上。

1. 迁移topic到新增的node上

假如现在一个kafka集群运行三个broker,broker.id依次为101,102,103,后来由于业务数据突然暴增,需要新增三个broker,broker.id依次为104,105,106.目的是要把push-token-topic迁移到新增node上。

1、脚本migration-push-token-topic.json文件内容如下:

  1. {
  2. "topics":
  3. [
  4. {
  5. "topic": "push-token-topic"
  6. }
  7. ],
  8. "version":1
  9. }

2、执行脚本如下所示:

  1. root@localhost:$  ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183 --topics-to-move-json-file  migration-push-token-topic.json  --broker-list  "104,105,106"  --generate

生成分配partitions的json脚本 备份恢复使用:

Current partition replica assignment

{"version":1,"partitions":[{"topic":"cluster-switch-topic","partition":10,"replicas":[8]},{"topic":"cluster-switch-topic","partition":5,"replicas":[4]},{"topic":"cluster-switch-topic","partition":3,"replicas":[5]},{"topic":"cluster-switch-topic","partition":4,"replicas":[5]},{"topic":"cluster-switch-topic","partition":9,"replicas":[5]},{"topic":"cluster-switch-topic","partition":1,"replicas":[5]},{"topic":"cluster-switch-topic","partition":11,"replicas":[4]},{"topic":"cluster-switch-topic","partition":7,"replicas":[5]},{"topic":"cluster-switch-topic","partition":2,"replicas":[4]},{"topic":"cluster-switch-topic","partition":0,"replicas":[4]},{"topic":"cluster-switch-topic","partition":6,"replicas":[4]},{"topic":"cluster-switch-topic","partition":8,"replicas":[4]}]}

重新分配parttions的json脚本如下:

migration-topic-cluster-switch-topic.json

{"version":1,"partitions":[{"topic":"cluster-switch-topic","partition":10,"replicas":[5]},{"topic":"cluster-switch-topic","partition":5,"replicas":[4]},{"topic":"cluster-switch-topic","partition":4,"replicas":[5]},{"topic":"cluster-switch-topic","partition":3,"replicas":[4]},{"topic":"cluster-switch-topic","partition":9,"replicas":[4]},{"topic":"cluster-switch-topic","partition":1,"replicas":[4]},{"topic":"cluster-switch-topic","partition":11,"replicas":[4]},{"topic":"cluster-switch-topic","partition":7,"replicas":[4]},{"topic":"cluster-switch-topic","partition":2,"replicas":[5]},{"topic":"cluster-switch-topic","partition":0,"replicas":[5]},{"topic":"cluster-switch-topic","partition":6,"replicas":[5]},{"topic":"cluster-switch-topic","partition":8,"replicas":[5]}]}

3、执行:

  1. root@localhost:$   bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183 --reassignment-json-file migration-topic-cluster-switch-topic.json --execute

执行后会生成一个json格式文件expand-cluster-reassignment.json

4、查询执行状态:
  1. bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183 --reassignment-json-file expand-cluster-reassignment.json --verify
正常执行后会返回当前数据迁移的不用partion的,信息状态类似下面

  1. Reassignment of partition [push-token-topic,0] completed successfully   //移动成功
  2. Reassignment of partition [push-token-topic,1] is in progress          //这行代表数据在移动中
  3. Reassignment of partition [push-token-topic,2] is in progress
  4. Reassignment of partition [push-token-topic,1] completed successfully
  5. Reassignment of partition [push-token-topic,2] completed successfully

这样做不会影响原来集群上的topic业务

2.topic修改(replicats-factor)副本个数

假如初始时push-token-topic为一个副本,为了提高可用性,需要改为2副本模式。

脚本replicas-update-push-token-topic.json文件内容如下:

{

"partitions":

[

{

"topic": "log.mobile_nginx",

"partition": 0,

"replicas": [101,102,104]

},

{

"topic": "log.mobile_nginx",

"partition": 1,

"replicas": [102,103,106]

}

],

"version":1

}

2、执行:

  1. root@localhost:$ ./bin/kafka-reassign-partitions.sh --zookeeper   192.168.2.225:2183 --reassignment-json-file  replicas-update-push-token-topic.json  --execute

执行后会列出当前的partition和修改后的patition

3、verify
  1. bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2181 --reassignment-json-file replicas-update-push-token-topic.json --verify

如下:

Status of partition reassignment:
Reassignment of partition [log.mobile_nginx,0] completed successfully
Reassignment of partition [log.mobile_nginx,1] completed successfully

3.自定义分区和迁移

1、The first step is to hand craft the custom reassignment plan in a json file-

> cat custom-reassignment.json
{"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":1,"replicas":[2,3]}]}
2、Then, use the json file with the --execute option to start the reassignment process-

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --execute

Current partition replica assignment

{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":[1,2]},
{"topic":"foo2","partition":1,"replicas":[3,4]}]
} Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},
{"topic":"foo2","partition":1,"replicas":[2,3]}]
}

3、The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same expand-cluster-reassignment.json (used with the --execute option) should be used with the --verify option

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --verify

Status of partition reassignment:
Reassignment of partition [foo1,0] completed successfully
Reassignment of partition [foo2,1] completed successfully

4.topic的分区扩容用法

a.先扩容分区数量,脚本如下:

例如:push-token-topic初始分区数量为12,目前到增加到15个

root@localhost:$ ./bin/kafka-topics.sh --zookeeper 192.168.2.225:2183 --alter --partitions 15 --topic   push-token-topic

b.设置topic分区副本

root@localhost:$ ./bin/kafka-reassign-partitions.sh --zookeeper  192.168.2.225:2183

--reassignment-json-file partitions-extension-push-token-topic.json  --execute

脚本partitions-extension-push-token-topic.json文件内容如下:

{

"partitions":

[

{

"topic": "push-token-topic",

"partition": 12,

"replicas": [101,102]

},

{

"topic": "push-token-topic",

"partition": 13,

"replicas": [103,104]

},

{

"topic": "push-token-topic",

"partition": 14,

"replicas": [105,106]

}

],

"version":1

}

kafka迁移与扩容的更多相关文章

  1. Kafka迁移与扩容工具用法

    1.迁移topic到新增的node上 假如现在一个kafka集群运行三个broker,broker.id依次为101,102,103,后来由于业务数据突然暴增,需要新增三个broker,broker. ...

  2. (三)kafka集群扩容后的topic分区迁移

    kafka集群扩容后的topic分区迁移 kafka集群扩容后,新的broker上面不会数据进入这些节点,也就是说,这些节点是空闲的:它只有在创建新的topic时才会参与工作.除非将已有的partit ...

  3. kafka集群扩容以及数据迁移

    一 kafka集群扩容比较简单,机器配置一样的前提下只需要把配置文件里的brokerid改一个新的启动起来就可以.比较需要注意的是如果公司内网dns更改的不是很及时的话,需要给原有的旧机器加上新服务器 ...

  4. kafka集群扩容后的topic分区迁移

    https://www.cnblogs.com/honeybee/p/5691921.html kafka集群扩容后,新的broker上面不会数据进入这些节点,也就是说,这些节点是空闲的:它只有在创建 ...

  5. Mysql系列九:使用zookeeper管理远程Mycat配置文件、Mycat监控、Mycat数据迁移(扩容)

    一.使用zookeeper管理远程Mycat配置文件 环境准备: 虚拟机192.168.152.130: zookeeper,具体参考前面文章 搭建dubbo+zookeeper+dubboadmin ...

  6. kafka迁移数据目录

    问题 先前存储kafka日志的磁盘空间太小,zabbix警报不断,于是加了磁盘,将日志存到新磁盘上. 解决方案 依次在每台机器上操作,保证有机器能响应producer和consumer的操作. 加磁盘 ...

  7. redis cluster异地数据迁移,扩容,缩容

    由于项目的服务器分布在重庆,上海,台北,休斯顿,所以需要做异地容灾需求.当前的mysql,redis cluster,elastic search都在重庆的如果重庆停电了,整个应用都不能用了. 现在考 ...

  8. kafka迁移topic

    1. 准备移动 这里假设要移动的topic名称为:topicA.topicB vim topics-to-move.json {"topics": [{"topic&qu ...

  9. 【转】apache kafka技术分享系列(目录索引)

    转自:  http://blog.csdn.net/lizhitao/article/details/39499283   估计大神会不定期更新,所以还是访问这个链接看最新的目录list比较好 apa ...

随机推荐

  1. Chapter 3

    1.序列类型可以使用成员操作符in,大小计算函数(len()),分片([]),都可以迭代.Python内置的序列类型:str,list,tuple,bytearray,bytes.标准库中的序列类型: ...

  2. LCT模板

    之前一直用的LCT模板,因为其实个人对LCT和Splay不是很熟,所以用起来总觉得略略的坑爹,过了一段时间就忘了,但事实上很多裸的LCT要改的东西是不多的,所以今天写了些注释,以后可能套起模板来会得心 ...

  3. POJ2402/UVA 12050 Palindrome Numbers 数学思维

    A palindrome is a word, number, or phrase that reads the same forwards as backwards. For example,the ...

  4. C#DataGrdviewl加入checkBox全选删除

    #region 加入checkBox /// <summary> /// 加入checkBox /// </summary> /// <param name=" ...

  5. lintcode:等价二叉树

    等价二叉树 检查两棵二叉树是否等价.等价的意思是说,首先两棵二叉树必须拥有相同的结构,并且每个对应位置上的节点上的数都相等. 样例 1 1 / \ / \ 2 2 and 2 2 / / 4 4 就是 ...

  6. python_pycharm介绍1

    1. 常用设置 修改编程风格 File-Setting中,Editor下Colors&Fonts修改即可调整风格. 修改字体大小 pycharm默认字体太小,需调整些,Settings--&g ...

  7. JAVA数据库连接池实现(转)

    连接池的管理用了了享元模式,这里对连接池进行简单设计. 一.设计思路 1.连接池配置属性DBbean:里面存放可以配置的一些属性 2.连接池接口IConnectionPool:里面定义一些基本的获取连 ...

  8. Linux资源监控命令/工具(调试)

    1.直接将指令丢到背景中执行:& [root@linux ~]# tar -zpcvf /tmp/etc.tar.gz /etc > /tmp/log.txt 2>&1 & ...

  9. POJ1088滑雪

    Description Michael喜欢滑雪百这并不奇怪, 因为滑雪的确很刺激.可是为了获得速度,滑的区域必须向下倾斜,而且当你滑到坡底,你不得不再次走上坡或者等待升降机来载你.Michael想知道 ...

  10. 在CentOS 7上给一个网卡分配多个IP地址

    有时你也许想要给一个网卡多个地址.你该怎么做呢?另外买一个网卡来分配地址?在小型网络中其实不用这么做.我们现在可以在CentOS/RHEL 7中给一个网卡分配多个ip地址.想知道怎么做么?好的,跟随我 ...