kafka关于修改副本数和分区的数的案例实战（也可用作leader节点均衡案例）

　　　　　　　　　　　　　　kafka关于修改副本数和分区的数的案例实战（也可用作leader节点均衡案例）

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

一.关于topic分区数的修改

1>.创建1分区1个的topic，名称为yinzhengjie-channel

[root@node101 ~]# kafka-topics.sh --zookeeper node102.yinzhengjie.org.cn: --create --replication-factor  -partitions  --topic yinzhengjie-channel
Created topic "yinzhengjie-channel".
[root@node101 ~]# 

2>.查看topic的信息

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel
Topic:yinzhengjie-channel    PartitionCount:    ReplicationFactor:    Configs:
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr: 103                              #可以很明显的看出kafka 的分区数和副本数都是1
[root@node101 ~]# 

3>.将之前创建的topic修改为3个分区

[root@node101 ~]# kafka-topics.sh --alter --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel --partitions
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[root@node101 ~]# 

4>.再次查看topic的分区数

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel
Topic:yinzhengjie-channel    PartitionCount:    ReplicationFactor:    Configs:
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:                                     #这是第一个分区，它的副本数依然是1一个，当前0号分区的副本数存放在103这个节点上。说明你的数据修改成功啦！
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:
[root@node101 ~]# 

二.关于topic副本数的修改

1>.编写分配脚本

[root@node101 ~]# cat addReplicas.json
{"topics":
    [{"topic":"yinzhengjie-channel"}],
    "version":
}
[root@node101 ~]# 

2>.执行分配计划，用于生成json格式的文件

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn: --topics-to-move-json-file addReplicas.json --broker-list "101,102,103" --generate
Current partition replica assignment            　　　　　　#这是当前的分区情况，你可以结合--describe参数查看当前的分区情况
{"version":,"partitions":[{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]}]}

Proposed partition reassignment configuration     　　　　#这是推荐分区计划
{"version":,"partitions":[{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]}]}
[root@node101 ~]#
[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel　　　　　　#查看当前分区的情况
Topic:yinzhengjie-channel    PartitionCount:    ReplicationFactor:    Configs:
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:
[root@node101 ~]# 

3>.Proposed partition reassignment configuration 后是根据命令行的指定的brokerlist生成的分区分配计划json格式。将 Proposed partition reassignment configuration的配置copy保存到一个文件中 topic-reassignment.json并对它进行相应的修改

[root@node101 ~]# cat topic-reassignment.json　　　　#注意，我在复制下来之后，对副本数进行了修改，由之前的1个副本升级为2个副本。
{"version":,"partitions":[{"topic":"yinzhengjie-channel","partition":,"replicas":[,]},{"topic":"yinzhengjie-channel","partition":,"replicas":[,]},{"topic":"yinzhengjie-channel","partition":,"replicas":[,]}]}
[root@node101 ~]# 

4>.根据上一步生成的分配计划配置json文件topic-reassignment.json，进行topic的重新分配。

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn: --reassignment-json-file topic-reassignment.json --execute
Current partition replica assignment

{"version":,"partitions":[{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
[root@node101 ~]# 

5>.查看分配的进度

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn: --reassignment-json-file topic-reassignment.json --verify
Status of partition reassignment:
Reassignment of partition [yinzhengjie-channel,] completed successfully　　　　　　#如果这里的参数是：is still in progress，说明正在进行分配，如果看到当前的提示说明分配完成。
Reassignment of partition [yinzhengjie-channel,] completed successfully
Reassignment of partition [yinzhengjie-channel,] completed successfully
[root@node101 ~]# 

　　　　　温馨提示，上述的方法不仅仅可以用来修改副本数，还可以用来修改你的leader节点，下图我就是我在生产环境中用来均衡leader节点的实操截图：（是不是上面我提到的2种状态都有呢？）

6>.如果分配完成，我们再次查看

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel　　　　　　#查看当前分区的情况，这是还没有重新分配的时候
Topic:yinzhengjie-channel    PartitionCount:    ReplicationFactor:    Configs:
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr: 103　　　　　　　　#这里的副本数只有一个！
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:
[root@node101 ~]#
root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel
Topic:yinzhengjie-channel    PartitionCount:    ReplicationFactor:    Configs:
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas: ,    Isr: ,　　　　　　#副本数编程了2个！
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas: ,    Isr: ,
    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas: ,    Isr: ,
[root@node101 ~]#