Kafka迁移与扩容工具用法

１．迁移topic到新增的node上

假如现在一个kafka集群运行三个broker，broker.id依次为101,102,103,后来由于业务数据突然暴增，需要新增三个broker,broker.id依次为104,105,106.目的是要把chatmessage迁移到新增node上。脚本(json格式)如下所示：

kafka-reassign-partitions.sh --zookeeper 192.168.1.10: --topics-to-move-json-file  migration-chatmessage-topic.json  --broker-list  "104,105,106"  --generate

migration-chatmessage-topic.json文件内容如下：

{
    "topics":
        [
            {
                "topic": "chatmessage"
            }
        ],
    "version":
}

生成分配partitions的json脚本：

{
    "version":,
    "partitions":
        [
            {
                "topic":"chatmessage",
                "partition":,"replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            },
            {
                "topic":"chatmessage",
                "partition":,
                "replicas":[]
            }
        ]
    }
}

重新分配parttions的json脚本如下：migration-topic-chatmessage-topic.json

{"version":,
    "partitions":
        [
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]},
            {"topic":"cluster-switch-topic","partition":,"replicas":[]}
        ]
}

执行一下脚本

./kafka-reassign-partitions.sh --zookeeper 192.168.1.10:  --reassignment-json-file migration-topic-chatmessage-topic.json --execute

topic修改(replicats-factor)副本个数

　　假如初始时chatmessage为一个副本，为了提高可用性，需要改为2副本模式。脚本replicas-chatmessage-topic.json文件内容如下:

{
        "partitions":
                [
                {
                        "topic": "chatmessage",
                        "partition": ,
                        "replicas": [,,]
                },
                {
                        "topic": "chatmessage",
                        "partition": ,
                        "replicas": [,,]
                },
              .......
        "version":
}

编辑好json文件后执行一下命令：

./kafka-reassign-partitions.sh --zookeeper  192.168.1.10: --reassignment-json-file  replicas-chatmessage-topic.json  --execute

3.topic的分区扩容用法

　　先扩容分区数量，脚本如下:

./kafka-topics.sh --zookeeper 192.168.1.10:  --alter   --partitions    --topic   chatmessage

　　把topic为chatmessage的分区数量扩展到１５个。

　　设置topic分区副本：

{
        "partitions":
                [
                {
                        "topic": "chatmessage",
                        "partition": ,
                        "replicas": [,]
                },
                {
                        "topic": "chatmessage",
                        "partition": ,
                        "replicas": [,]
                },
                {
                        "topic": "chatmessage",
                        "partition": ,
                        "replicas": [,]
                }
                ],
        "version":
}

　　编辑json文件，执行一下脚本：

./bin/kafka-reassign-partitions.sh --zookeeper  192.168.１.10: --reassignment-json-file partitions-extension-chatmessage-topic.json  --execute