Kafka 1.0.0集群增加节点

原有环境

主机名	IP 地址	安装路径	系统
sht-sgmhadoopdn-01	172.16.101.58	/opt/kafka_2.12-1.0.0 /opt/kafka（软连接）	CentOS Linux release 7.3.1611 (Core)
sht-sgmhadoopdn-02	172.16.101.59
sht-sgmhadoopdn-03	172.16.101.60

向集群增加节点

sht-sgmhadoopdn-04（172.16.101.66）

过程

一. 新节点配置和集群节点环境一致

二. zookeeper配置

1. 集群各节点增加新节点的zookeeper配置

tickTime=
initLimit=
syncLimit=
dataDir=/opt/kafka/data
clientPort=
server.=sht-sgmhadoopdn-::
server.=sht-sgmhadoopdn-::
server.=sht-sgmhadoopdn-::
server.=sht-sgmhadoopdn-::

2. 新节点创建server-id

# echo  > /opt/kafka/data/myid

3. 启动zookeeper

# /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties

4. 查看新节点zookeeper状态

# echo stat | nc sht-sgmhadoopdn-  | grep Mode
Mode: follower

三.  kafka配置

1.新节点配置文件server.properties

broker.id=
listeners=PLAINTEXT://172.16.101.66:9092
advertised.listeners=PLAINTEXT://172.16.101.66:9092
log.dirs=/opt/kafka/data
zookeeper.connect=sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-:

2. 向集群中所有节点kafka配置文件增加对新zookeeper节点的支持

zookeeper.connect=sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-:

3. 启动kafka

/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties

4. 查看集群

# echo dump | nc sht-sgmhadoopdn-  | grep broker
    /brokers/ids/
    /brokers/ids/
    /brokers/ids/
    /brokers/ids/

四. 分区重分配

1. 查看现有集群的topic以及分区方案

# kafka-topics.sh --zookeeper 172.16.101.58:,172.16.101.59:,172.16.101.60:,172.16.101.66: --list
__consumer_offsets
test-topic

# kafka-topics.sh --zookeeper 172.16.101.58:,172.16.101.59:,172.16.101.60:,172.16.101.66: --describe --topic test-topic
Topic:test-topic    PartitionCount:    ReplicationFactor:    Configs:
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,

可以看到test-topic的6个分区均集中在在老集群中，新添加的节点并未参与分区方案。

现在将执行分区重分配，将数据均匀分散在左右节点上

2. 创建json文件

# cat topics-to-move.json
{"topics":[{"topic":"test-topic"}],"version":}

3. 产生分区分配方案

[root@sht-sgmhadoopdn-01 kafka]# kafka-reassign-partitions.sh --zookeeper 172.16.101.58:2182,172.16.101.59:2182,172.16.101.60:2182,172.16.101.66:2182 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test-topic","partition":0,"replicas":[1,2,0],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":5,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":3,"replicas":[1,2,0],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":2,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":4,"replicas":[2,0,1],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":1,"replicas":[2,0,1],"log_dirs":["any","any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test-topic","partition":0,"replicas":[3,0,1],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":5,"replicas":[0,2,3],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":3,"replicas":[2,3,0],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":2,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":4,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]}]}

注意“Proposed partition reassignment configuration”为kafka提供的分区方案，实际上并没有真正执行，我们将该分区方案保存为另外一个文件expand_cluster_reassignment.json，然后再真正执行这个分区方案。

4. 执行分区重分配

# kafka-reassign-partitions.sh --zookeeper 172.16.101.58:,172.16.101.59:,172.16.101.60:,172.16.101.66: --reassignment-json-file expand_cluster_reassignment.json --execute
Current partition replica assignment

{"version":,"partitions":[{"topic":"test-topic","partition":,"replicas":[,,],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":,"replicas":[,,],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":,"replicas":[,,],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":,"replicas":[,,],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":,"replicas":[,,],"log_dirs":["any","any","any"]},{"topic":"test-topic","partition":,"replicas":[,,],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

通过--verify查看分区进程

# kafka-reassign-partitions.sh --zookeeper 172.16.101.58:,172.16.101.59:,172.16.101.60:,172.16.101.66: --reassignment-json-file expand_cluster_reassignment.json --verify
Status of partition reassignment:
Reassignment of partition test-topic- is still in progress
Reassignment of partition test-topic- completed successfully
Reassignment of partition test-topic- is still in progress
Reassignment of partition test-topic- is still in progress
Reassignment of partition test-topic- is still in progress
Reassignment of partition test-topic- is still in progress

5. 等到上述分区过程结束后，再次查看topic分区情况

# kafka-topics.sh --zookeeper 172.16.101.58: --describe --topic test-topic
Topic:test-topic    PartitionCount:    ReplicationFactor:    Configs:
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,