kafka 常用命令

Kafka

解压，进入kafka目录下的bin目录

启动zk

nohup ./zookeeper-server-start.sh ../config/zookeeper.properties &

nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties &

启动kafka

nohup ./kafka-server-start.sh ../config/server.properties &

nohup ./bin/kafka-server-start.sh ./config/server.properties &

创建topic

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

或

./kafka-topics.sh --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test2 --create

列出topic

./kafka-topics.sh --list --zookeeper localhost:2181

或

./kafka-topic --zookeeper localhost:2181 --list

启动生产者并发送消息

./kafka-console-producer.sh --broker-list localhost:9092 --topic zyl

输入消息

nihao

china

另外开个终端，启动消费者接受消息

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic zyl --from-beginning

可以接受到

nihao

china

另外开个终端，可以看到有4个进程（分别为zk、kafka、控制台消费者、控制台生产者）

[root@aa bin]# jps | grep -v Jps

9802 QuorumPeerMain

10392 Kafka

10956 ConsoleConsumer

10887 ConsoleProducer

使用集群3个节点（多个节点也同理,加上之前的，我一共弄了4个节点）

拷贝并修改配置文件

cp server.properties server1.properties

cp server.properties server2.properties

cp server.properties server3.properties

nohup ./kafka-server-start.sh ../config/server1.properties &

nohup ./kafka-server-start.sh ../config/server2.properties &

nohup ./kafka-server-start.sh ../config/server3.properties &

[root@aa bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic（注意，指定的是3）

Created topic "my-replicated-topic".

[root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:

Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,3,0 Isr: 2,3,0

[root@aa bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 4 --partitions 1 --topic my-replicated-topic

Error while executing topic command Topic "my-replicated-topic" already exists.

kafka.common.TopicExistsException: Topic "my-replicated-topic" already exists.

at kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:171)

at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:156)

at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:86)

at kafka.admin.TopicCommand$.main(TopicCommand.scala:50)

at kafka.admin.TopicCommand.main(TopicCommand.scala)

[root@aa bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 4 --partitions 1 --topic my-replicated-topic2

Created topic "my-replicated-topic2".

[root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic2

Topic:my-replicated-topic2 PartitionCount:1 ReplicationFactor:4 Configs:

Topic: my-replicated-topic2 Partition: 0 Leader: 1 Replicas: 1,2,3,0 Isr: 1,2,3,0

[root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic zyl

Topic:zyl PartitionCount:1 ReplicationFactor:1 Configs:

Topic: zyl Partition: 0 Leader: 0 Replicas: 0 Isr: 0

启动生产者并发送消息

./kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic

另外开个终端，启动消费者接受消息

./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic

另外开个终端，可以看到有9个进程（分别为1个zk、4个kafka、2个控制台消费者、2个控制台生产者）

[root@aa ~]# jps | grep -v Jps

9802 QuorumPeerMain

10392 Kafka

11547 Kafka

12007 ConsoleProducer

10956 ConsoleConsumer

10887 ConsoleProducer

11469 Kafka

12054 ConsoleConsumer

11710 Kafka

查看单独的进程：

ps -ef | grep server.properties | grep -v grep

ps -ef | grep "server1.properties" | grep -v grep

测试leader

杀掉进程，然后--describe查看

[root@aaa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:

Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,3,0 Isr: 2,3,0

可以看到leader是2.

[root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic2

Topic:my-replicated-topic2 PartitionCount:1 ReplicationFactor:4 Configs:

Topic: my-replicated-topic2 Partition: 0 Leader: 1 Replicas: 1,2,3,0 Isr: 1,2,3,0

可以看到leader是2.

[root@aa bin]# ps -ef | grep server1.properties | grep -v grep

杀掉某个kafka进程

kill -9 11469

./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic2

Topic:my-replicated-topic2 PartitionCount:1 ReplicationFactor:4 Configs:

Topic: my-replicated-topic2 Partition: 0 Leader: 2 Replicas: 1,2,3,0 Isr: 2,3,0

[root@aa bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:

Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,3,0 Isr: 2,3,0

可以看到my-replicated-topic2的leader变成了2。

问题：

1.zookeeper-shell.sh如何使用

2.能不能通过kafka-server-stop.sh停止某个kafka呢

Command must include exactly one action: --list, --describe, --create or --alter

Option                                  Description

------                                  -----------

--alter                                 Alter the configuration for the topic.

--config <name=value>                   A topic configuration override for the

topic being created or altered.

--create                                Create a new topic.

--deleteConfig <name>                   A topic configuration override to be

removed for an existing topic

--describe                              List details for the given topics.

--help                                  Print usage information.

--list                                  List all available topics.

--partitions <Integer: # of partitions> The number of partitions for the topic

being created or altered (WARNING:

If partitions are increased for a

topic that has a key, the partition

logic or ordering of the messages

will be affected

--replica-assignment                    A list of manual partition-to-broker

<broker_id_for_part1_replica1 :         assignments for the topic being

broker_id_for_part1_replica2 ,          created or altered.

broker_id_for_part2_replica1 :

broker_id_for_part2_replica2 , ...>

--replication-factor <Integer:          The replication factor for each

replication factor>                     partition in the topic being created.

--topic <topic>                         The topic to be create, alter or

describe. Can also accept a regular

expression except for --create option

--topics-with-overrides                 if set when describing topics, only

show topics that have overridden

configs

--unavailable-partitions                if set when describing topics, only

show partitions whose leader is not

available

--under-replicated-partitions           if set when describing topics, only

show under replicated partitions

--zookeeper <urls>                      REQUIRED: The connection string for

the zookeeper connection in the form

host:port. Multiple URLS can be

given to allow fail-over.