解决kafka集群由于默认的__consumer_offsets这个topic的默认的副本数为1而存在的单点故障问题
抛出问题：
__consumer_offsets这个topic是由kafka自动创建的，默认50个，但是都存在一台kafka服务器上，这是不是就存在很明显的单点故障？
经测试，如果将存储consumer_offsets的这台机器kill掉，所有的消费者都停止消费了。请问这个问题是怎么解决的呢？
原因分析：
由于__consumer_offsets这个用于存储offset的分区是由kafka服务器默认自动创建的，那么它在创建该分区的时候，分区数和副本数的依据是什么？
分区数是固定的50，这个没什么可怀疑的，副本数呢？应该是一个默认值1，依据是，如果我们没有在server.properties文件中指定topic分区的副本数的话，它的默认值就是1。
__consumer_offsets是一个非常重要的topic，我们怎么能允许它只有一个副本呢？这样就存在单点故障，也就是如果该分区所在的集群宕机了的话，
我们的消费者就无法正常消费数据了。
我在笔记本上搭建了kafka集群，共3个Broker，来解决这个问题。下面是一些记录。
说明：如果你的__consumer_offsets这个topic已经被创建了，而且只存在一台broker上，如果你直接使用命令删除这个topic是会报错了，提示这是kafka内置的topic，禁止删除。可以在zookeeper中删除（我是使用ZooInspector这个客户端连上zookeeper，删除和__consumer_offsets这个topic有关的目录或节点）。
然后就是修改kafka的核心配置文件server.properties，下面是第一台Broker的配置文件：
[root@hadoop01 kafka-logs]# cat /opt/kafka/config/server.properties
broker.id=0
listeners=PLAINTEXT://:9092
port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/logs/kafka-logs
num.partitions=3
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=536870912
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.71.11:2181,192.168.71.12:2181,192.168.71.13:2181
izookeeper.connection.timeout.ms=6000
delete.topic.enable=true
host.name=192.168.71.11
advertised.host.name=192.168.71.11
auto.create.topics.enable=true
default.replication.factor=3
在上面的配置中，我开启了自动创建topic，并指定了topic的分区数为3，副本数为3(因为我有3台Broker，我们希望每台机器上都有一个副本，从而保证分区的高可用)。
下面是第二台Broker的server.properties文件的配置：
1 [root@hadoop02 kafka-logs]# cat /opt/kafka/config/server.properties
2 broker.id=1
3 listeners=PLAINTEXT://:9092
4 port=9092
5 num.network.threads=3
6 num.io.threads=8
7 socket.send.buffer.bytes=102400
8 socket.receive.buffer.bytes=102400
9 socket.request.max.bytes=104857600
10 log.dirs=/opt/logs/kafka-logs
11 num.partitions=3
12 num.recovery.threads.per.data.dir=1
13 log.retention.hours=168
14 log.segment.bytes=536870912
15 log.retention.check.interval.ms=300000
16 zookeeper.connect=192.168.71.11:2181,192.168.71.12:2181,192.168.71.13:2181
17 izookeeper.connection.timeout.ms=6000
18 delete.topic.enable=true
19 host.name=192.168.71.12
20 advertised.host.name=192.168.71.12
21 auto.create.topics.enable=true
22 default.replication.factor=3
23 [root@hadoop02 kafka-logs]#
第3台Broker的server.properties文件的配置：
1 [root@hadoop03 kafka-logs]# cat /opt/kafka/config/server.properties
2 broker.id=2
3 listeners=PLAINTEXT://:9092
4 port=9092
5 num.network.threads=3
6 num.io.threads=8
7 socket.send.buffer.bytes=102400
8 socket.receive.buffer.bytes=102400
9 socket.request.max.bytes=104857600
10 log.dirs=/opt/logs/kafka-logs
11 num.partitions=3
12 num.recovery.threads.per.data.dir=1
13 log.retention.hours=168
14 log.segment.bytes=536870912
15 log.retention.check.interval.ms=300000
16 zookeeper.connect=192.168.71.11:2181,192.168.71.12:2181,192.168.71.13:2181
17 izookeeper.connection.timeout.ms=6000
18 delete.topic.enable=true
19 host.name=192.168.71.13
20 advertised.host.name=192.168.71.13
21 auto.create.topics.enable=true
22 default.replication.factor=3
23 [root@hadoop03 kafka-logs]#
配置好之后，启动3个Broker。此时__consumer_offsets分区不会被创建，它会在消费者开始消费数据的时候被创建。
首先启动一个消费者客户端来消费数据。然后启动一个生产者客户端来发送数据到kafka集群。
下面查看第一个Broker的topic，发现有50个__consumer_offsets分区：
[root@hadoop01 kafka-logs]# ll
total 224
-rw-r--r--. 1 root root 0 Mar 24 13:19 cleaner-offset-checkpoint
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-0
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-1
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-10
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-11
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-12
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-13
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-14
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-15
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-16
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-17
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-18
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-19
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-2
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-20
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-21
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-22
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-23
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-24
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-25
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-26
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-27
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-28
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-29
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-3
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-30
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-31
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-32
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-33
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-34
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-35
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-36
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-37
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-38
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-39
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-4
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-40
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-41
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-42
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-43
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-44
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-45
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-46
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-47
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-48
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-49
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-5
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-6
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-7
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-8
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-9
drwxr-xr-x. 2 root root 4096 Mar 24 13:54 friend-0
drwxr-xr-x. 2 root root 4096 Mar 24 13:54 friend-1
drwxr-xr-x. 2 root root 4096 Mar 24 13:54 friend-2
-rw-r--r--. 1 root root 54 Mar 24 13:19 meta.properties
-rw-r--r--. 1 root root 1228 Mar 24 13:54 recovery-point-offset-checkpoint
-rw-r--r--. 1 root root 1228 Mar 24 13:54 replication-offset-checkpoint
[root@hadoop01 kafka-logs]#
启动第二台Broker服务器，它也有50个"__consumer_offsets"分区：
[root@hadoop02 kafka-logs]# ll
total 224
-rw-r--r--. 1 root root 0 Mar 24 13:19 cleaner-offset-checkpoint
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-0
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-1
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-10
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-11
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-12
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-13
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-14
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-15
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-16
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-17
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-18
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-19
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-2
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-20
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-21
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-22
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-23
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-24
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-25
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-26
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-27
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-28
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-29
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-3
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-30
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-31
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-32
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-33
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-34
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-35
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-36
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-37
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-38
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-39
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-4
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-40
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-41
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-42
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-43
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-44
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-45
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-46
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-47
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-48
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-49
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-5
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-6
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-7
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-8
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-9
drwxr-xr-x. 2 root root 4096 Mar 24 13:54 friend-0
drwxr-xr-x. 2 root root 4096 Mar 24 13:54 friend-1
drwxr-xr-x. 2 root root 4096 Mar 24 13:54 friend-2
-rw-r--r--. 1 root root 54 Mar 24 13:19 meta.properties
-rw-r--r--. 1 root root 1228 Mar 24 13:54 recovery-point-offset-checkpoint
-rw-r--r--. 1 root root 1228 Mar 24 13:54 replication-offset-checkpoint
[root@hadoop02 kafka-logs]#
第3台Broker上的"__consumer_offsets"分区情况：
[root@hadoop03 kafka-logs]# ll
total 224
-rw-r--r--. 1 root root 0 Mar 24 13:19 cleaner-offset-checkpoint
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-0
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-1
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-10
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-11
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-12
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-13
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-14
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-15
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-16
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-17
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-18
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-19
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-2
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-20
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-21
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-22
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-23
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-24
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-25
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-26
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-27
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-28
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-29
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-3
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-30
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-31
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-32
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-33
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-34
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-35
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-36
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-37
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-38
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-39
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-4
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-40
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-41
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-42
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-43
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-44
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-45
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-46
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-47
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-48
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-49
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-5
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-6
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-7
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-8
drwxr-xr-x. 2 root root 4096 Mar 24 13:46 __consumer_offsets-9
drwxr-xr-x. 2 root root 4096 Mar 24 13:54 friend-0
drwxr-xr-x. 2 root root 4096 Mar 24 13:54 friend-1
drwxr-xr-x. 2 root root 4096 Mar 24 13:54 friend-2
-rw-r--r--. 1 root root 54 Mar 24 13:19 meta.properties
-rw-r--r--. 1 root root 1228 Mar 24 13:54 recovery-point-offset-checkpoint
-rw-r--r--. 1 root root 1228 Mar 24 13:54 replication-offset-checkpoint
[root@hadoop03 kafka-logs]# /opt/kafka/bin/kafka-topics.sh --describe --topic __consumer_offsets --zookeeper localhost:2181
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: __consumer_offsets Partition: 1 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: __consumer_offsets Partition: 2 Leader: 0 Replicas: 0,1,2 Isr: 1,2,0
Topic: __consumer_offsets Partition: 3 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: __consumer_offsets Partition: 4 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: __consumer_offsets Partition: 5 Leader: 0 Replicas: 0,2,1 Isr: 2,1,0
Topic: __consumer_offsets Partition: 6 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: __consumer_offsets Partition: 7 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: __consumer_offsets Partition: 8 Leader: 0 Replicas: 0,1,2 Isr: 1,2,0
Topic: __consumer_offsets Partition: 9 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: __consumer_offsets Partition: 10 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: __consumer_offsets Partition: 11 Leader: 0 Replicas: 0,2,1 Isr: 2,1,0
Topic: __consumer_offsets Partition: 12 Leader: 1 Replicas: 1,2,0 Isr: 1,0,2
Topic: __consumer_offsets Partition: 13 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: __consumer_offsets Partition: 14 Leader: 0 Replicas: 0,1,2 Isr: 1,2,0
Topic: __consumer_offsets Partition: 15 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: __consumer_offsets Partition: 16 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: __consumer_offsets Partition: 17 Leader: 0 Replicas: 0,2,1 Isr: 2,1,0
Topic: __consumer_offsets Partition: 18 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: __consumer_offsets Partition: 19 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: __consumer_offsets Partition: 20 Leader: 0 Replicas: 0,1,2 Isr: 1,2,0
Topic: __consumer_offsets Partition: 21 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: __consumer_offsets Partition: 22 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: __consumer_offsets Partition: 23 Leader: 0 Replicas: 0,2,1 Isr: 2,1,0
Topic: __consumer_offsets Partition: 24 Leader: 1 Replicas: 1,2,0 Isr: 1,0,2
Topic: __consumer_offsets Partition: 25 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: __consumer_offsets Partition: 26 Leader: 0 Replicas: 0,1,2 Isr: 1,2,0
Topic: __consumer_offsets Partition: 27 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: __consumer_offsets Partition: 28 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: __consumer_offsets Partition: 29 Leader: 0 Replicas: 0,2,1 Isr: 2,1,0
Topic: __consumer_offsets Partition: 30 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: __consumer_offsets Partition: 31 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: __consumer_offsets Partition: 32 Leader: 0 Replicas: 0,1,2 Isr: 1,2,0
Topic: __consumer_offsets Partition: 33 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: __consumer_offsets Partition: 34 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: __consumer_offsets Partition: 35 Leader: 0 Replicas: 0,2,1 Isr: 2,1,0
Topic: __consumer_offsets Partition: 36 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: __consumer_offsets Partition: 37 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: __consumer_offsets Partition: 38 Leader: 0 Replicas: 0,1,2 Isr: 1,2,0
Topic: __consumer_offsets Partition: 39 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: __consumer_offsets Partition: 40 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: __consumer_offsets Partition: 41 Leader: 0 Replicas: 0,2,1 Isr: 2,1,0
Topic: __consumer_offsets Partition: 42 Leader: 1 Replicas: 1,2,0 Isr: 1,0,2
Topic: __consumer_offsets Partition: 43 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: __consumer_offsets Partition: 44 Leader: 0 Replicas: 0,1,2 Isr: 1,2,0
Topic: __consumer_offsets Partition: 45 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: __consumer_offsets Partition: 46 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: __consumer_offsets Partition: 47 Leader: 0 Replicas: 0,2,1 Isr: 2,1,0
Topic: __consumer_offsets Partition: 48 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: __consumer_offsets Partition: 49 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
[root@hadoop03 kafka-logs]#
[root@hadoop03 kafka-logs]# /opt/kafka/bin/kafka-topics.sh --describe --topic friend --zookeeper localhost:2181
Topic:friend PartitionCount:3 ReplicationFactor:3 Configs:
Topic: friend Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: friend Partition: 1 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: friend Partition: 2 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
[root@hadoop03 kafka-logs]#
下面是启动消费者消费数据时的日志：
======启动一个消费者后的日志情况======================================================================
D:\java\jdk1.8.0_121\bin\java -Didea.launcher.port=7532 -Didea.launcher.bin.path=D:\java\IDEA-14.1.4\bin -Dfile.encoding=UTF-8 -classpath D:\java\jdk1.8.0_121\jre\lib\charsets.jar;D:\java\jdk1.8.0_121\jre\lib\deploy.jar;D:\java\jdk1.8.0_121\jre\lib\javaws.jar;D:\java\jdk1.8.0_121\jre\lib\jce.jar;D:\java\jdk1.8.0_121\jre\lib\jfr.jar;D:\java\jdk1.8.0_121\jre\lib\jfxswt.jar;D:\java\jdk1.8.0_121\jre\lib\jsse.jar;D:\java\jdk1.8.0_121\jre\lib\management-agent.jar;D:\java\jdk1.8.0_121\jre\lib\plugin.jar;D:\java\jdk1.8.0_121\jre\lib\resources.jar;D:\java\jdk1.8.0_121\jre\lib\rt.jar;D:\java\jdk1.8.0_121\jre\lib\ext\access-bridge-64.jar;D:\java\jdk1.8.0_121\jre\lib\ext\cldrdata.jar;D:\java\jdk1.8.0_121\jre\lib\ext\dnsns.jar;D:\java\jdk1.8.0_121\jre\lib\ext\jaccess.jar;D:\java\jdk1.8.0_121\jre\lib\ext\jfxrt.jar;D:\java\jdk1.8.0_121\jre\lib\ext\localedata.jar;D:\java\jdk1.8.0_121\jre\lib\ext\nashorn.jar;D:\java\jdk1.8.0_121\jre\lib\ext\sunec.jar;D:\java\jdk1.8.0_121\jre\lib\ext\sunjce_provider.jar;D:\java\jdk1.8.0_121\jre\lib\ext\sunmscapi.jar;D:\java\jdk1.8.0_121\jre\lib\ext\sunpkcs11.jar;D:\java\jdk1.8.0_121\jre\lib\ext\zipfs.jar;D:\zp\git\zp-kafka\zp-consumer-friend\build\classes\main;D:\zp\git\zp-kafka\zp-consumer-friend\build\resources\main;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.postgresql\postgresql\9.4-1201-jdbc41\870b0e689b514304461a9c1aba11920dc5de4321\postgresql-9.4-1201-jdbc41.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.5\c73038a3688525aad5cf33409df483178290cd64\joda-time-2.5.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-lang3\3.3.2\90a3822c38ec8c996e84c16a3477ef632cbc87a3\commons-lang3-3.3.2.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-io\1.3.2\b6dde38349ba9bb5e6ea6320531eae969985dae5\commons-io-1.3.2.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\commons-net\commons-net\3.3\cd0d5510908225f76c5fe5a3f1df4fa44866f81e\commons-net-3.3.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-databind\2.4.2\8e31266a272ad25ac4c089734d93e8d811652c1f\jackson-databind-2.4.2.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.3.5\9783d89b8eea20a517a4afc5f979bd2882b54c44\httpclient-4.3.5.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.mybatis\mybatis\3.2.8\7b6bf82cea13570b5290d6ed841283a1fcce170\mybatis-3.2.8.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.mybatis\mybatis-spring\1.2.2\1e40a7f5373e4242075a1d386817e7dd49b1697d\mybatis-spring-1.2.2.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.github.miemiedev\mybatis-paginator\1.2.15\d5d9891d2d89b13b0856a00f04ff60dd1f95ffdb\mybatis-paginator-1.2.15.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.github.pagehelper\pagehelper\3.4.2\173072cf0dab08b102d7420932f678d0c60138b7\pagehelper-3.4.2.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\mysql\mysql-connector-java\5.1.32\d28c9a6cf0810fd0e2180e44029c10a54ca26de8\mysql-connector-java-5.1.32.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.alibaba\druid\1.0.9\f91e47c9018578e5ca4d2e808cc3351505ae3ebb\druid-1.0.9.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\jstl\jstl\1.2\74aca283cd4f4b4f3e425f5820cda58f44409547\jstl-1.2.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\commons-fileupload\commons-fileupload\1.3.1\c621b54583719ac0310404463d6d99db27e1052c\commons-fileupload-1.3.1.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\redis.clients\jedis\2.7.2\f2f47f1025ea5090263820e8598e56eb47f5c88a\jedis-2.7.2.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.apache.solr\solr-solrj\4.10.3\6e5ee2f18a6615d5419b61e60e3cae274ac66085\solr-solrj-4.10.3.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.alibaba\fastjson\1.2.4\fbcf8415e32859b473b336e5ac6422ee69b1185b\fastjson-1.2.4.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework.integration\spring-integration-kafka\1.3.0.RELEASE\6d46351ea70084d51cd43f62cd80bb20af1c9d96\spring-integration-kafka-1.3.0.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework.kafka\spring-kafka\1.1.3.RELEASE\a45832c6a155383b2a54659926c1dc325052d883\spring-kafka-1.1.3.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.apache.kafka\kafka_2.10\0.10.0.0\37899467b805929a2ae898d312cd789dbd902c3d\kafka_2.10-0.10.0.0.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-annotations\2.4.0\d6a66c7a5f01cf500377bd669507a08cfeba882a\jackson-annotations-2.4.0.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.4.2\ceb72830d95c512b4b300a38f29febc85bdf6e4b\jackson-core-2.4.2.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.3.2\31fbbff1ddbf98f3aa7377c94d33b0447c646b6e\httpcore-4.3.2.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.6\b7f0fc8f61ecadeb3695f0b9464755eee44374d4\commons-codec-1.6.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-pool2\2.3\62a559a025fd890c30364296ece14643ba9c8c5b\commons-pool2-2.3.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpmime\4.3.1\f7899276dddd01d8a42ecfe27e7031fcf9824422\httpmime-4.3.1.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.apache.zookeeper\zookeeper\3.4.6\1b2502e29da1ebaade2357cd1de35a855fa3755\zookeeper-3.4.6.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.codehaus.woodstox\wstx-asl\3.2.7\252c7faae9ce98cb9c9d29f02db88f7373e7f407\wstx-asl-3.2.7.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.noggit\noggit\0.5\8e6e65624d2e09a30190c6434abe23b7d4e5413c\noggit-0.5.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework.integration\spring-integration-core\4.1.6.RELEASE\7b95eb7f4c08070c345a723720fd5dbccf71b140\spring-integration-core-4.1.6.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.yammer.metrics\metrics-annotation\2.2.0\62962b54c490a95c0bb255fa93b0ddd6cc36dd4b\metrics-annotation-2.2.0.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.yammer.metrics\metrics-core\2.2.0\f82c035cfa786d3cbec362c38c22a5f5b1bc8724\metrics-core-2.2.0.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\io.projectreactor\reactor-core\2.0.6.RELEASE\a36e58c9d2d0ac1f47e17e8db60a7ebb9d09789c\reactor-core-2.0.6.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-messaging\4.3.5.RELEASE\480f1116f2060107493b91e72e21359b02aca776\spring-messaging-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework.retry\spring-retry\1.1.3.RELEASE\f9517754a9990194ed0daecb5653e48564d557ee\spring-retry-1.1.3.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.projectreactor\reactor-core\1.1.4.RELEASE\da621f1aef5f8cd5c22ae78afec53f1e8659caed\reactor-core-1.1.4.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.reactivestreams\reactive-streams\1.0.0\14b8c877d98005ba3941c9257cfe09f6ed0e0d74\reactive-streams-1.0.0.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.lmax\disruptor\3.2.1\db375f499e32c3f06549e9addf8b1647123d6426\disruptor-3.2.1.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\io.gatling\jsr166e\1.0\d1bf191a18dfe6e3157a4fbf6b527390d906ace6\jsr166e-1.0.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\commons-io\commons-io\2.3\cd8d6ffc833cc63c30d712a180f4663d8f55799b\commons-io-2.3.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.101tec\zkclient\0.8\c0f700a4a3b386279d7d8dd164edecbe836cbfdb\zkclient-0.8.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.scala-lang\scala-library\2.10.6\421989aa8f95a05a4f894630aad96b8c7b828732\scala-library-2.10.6.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\4.9\ee9e9eaa0a35360dcfeac129ff4923215fd65904\jopt-simple-4.9.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\jline\jline\0.9.94\99a18e9a44834afdebc467294e1138364c207402\jline-0.9.94.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\io.netty\netty\3.7.0.Final\7a8c35599c68c0bf383df74469aa3e03d9aca87\netty-3.7.0.Final.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\junit\junit\3.8.1\99129f16442844f6a4a11ae22fbbee40b14d774f\junit-3.8.1.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.apache.kafka\kafka-clients\0.10.1.1\52f03b809c26f9676ddfcf130f13c80dfc929b98\kafka-clients-0.10.1.1.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\net.jpountz.lz4\lz4\1.3.0\c708bb2590c0652a642236ef45d9f99ff842a2ce\lz4-1.3.0.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.xerial.snappy\snappy-java\1.1.2.6\48d92871ca286a47f230feb375f0bbffa83b85f6\snappy-java-1.1.2.6.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.2\4bfc12adfe4842bf07b657f0369c4cb522955686\commons-logging-1.2.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.goldmansachs\gs-collections\5.1.0\7114c5349c816ea645b1ea3ffcc21fa073cbabc\gs-collections-5.1.0.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.slf4j\slf4j-log4j12\1.7.21\7238b064d1aba20da2ac03217d700d91e02460fa\slf4j-log4j12-1.7.21.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.slf4j\slf4j-api\1.7.21\139535a69a4239db087de9bab0bee568bf8e0b70\slf4j-api-1.7.21.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\log4j\log4j\1.2.17\5af35056b4d257e4b64b9e8069c0746e8b08629f\log4j-1.2.17.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-beans\4.3.5.RELEASE\e12bbc3277da28e2e2608a187f83091dc6c300bf\spring-beans-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-core\4.3.5.RELEASE\80299e3f80e8c6d5c076db2ba6adf44a4b52f578\spring-core-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-context\4.3.5.RELEASE\ca3391c0e17d0138335ba51b51371661d20d56a8\spring-context-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\com.goldmansachs\gs-collections-api\5.1.0\ea605cdf64cab5fc7b48c99f061d4c8db05b6ff1\gs-collections-api-5.1.0.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-aop\4.3.5.RELEASE\4f113218af716bd8d174c411f19b26418b5a70f6\spring-aop-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-expression\4.3.5.RELEASE\3689dc6c5b942ecde4122eac889ed87977d6f287\spring-expression-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-webmvc\4.3.5.RELEASE\c624659217edab07d8279fb0f90462136f089220\spring-webmvc-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-jdbc\4.3.5.RELEASE\7c09e38b6f6e0b178973dea06bb8fdc6d19aa596\spring-jdbc-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-aspects\4.3.5.RELEASE\8c00c15865a1da0a1ba143a50622a8436a56e097\spring-aspects-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-web\4.3.5.RELEASE\6641daccf2fddafc8358144f3a4f999130fdf144\spring-web-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.springframework\spring-tx\4.3.5.RELEASE\296d8ae21fadc0f115dae7d1e8d5c4f8c7de2c3e\spring-tx-4.3.5.RELEASE.jar;C:\Users\SYJ\.gradle\caches\modules-2\files-2.1\org.aspectj\aspectjweaver\1.8.9\db28774f477f07220eac18d5ec9c4e01f48589d7\aspectjweaver-1.8.9.jar;D:\java\IDEA-14.1.4\lib\idea_rt.jar com.intellij.rt.execution.application.AppMain com.zhaopin.AppMain
[17/03/24 20:46:36:702][org.springframework.context.support.AbstractApplicationContext-prepareRefresh] Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@6bc168e5: startup date [Fri Mar 24 20:46:36 CST 2017]; root of context hierarchy
[17/03/24 20:46:36:971][org.springframework.beans.factory.xml.XmlBeanDefinitionReader-loadBeanDefinitions] Loading XML bean definitions from file [D:\zp\git\zp-kafka\zp-consumer-friend\build\resources\main\spring\applicationContext-consumer.xml]
[17/03/24 20:46:37:472][org.springframework.beans.factory.xml.XmlBeanDefinitionReader-loadBeanDefinitions] Loading XML bean definitions from file [D:\zp\git\zp-kafka\zp-consumer-friend\build\resources\main\spring\applicationContext-dao.xml]
[17/03/24 20:46:37:871][org.springframework.beans.factory.xml.XmlBeanDefinitionReader-loadBeanDefinitions] Loading XML bean definitions from file [D:\zp\git\zp-kafka\zp-consumer-friend\build\resources\main\spring\applicationContext-service.xml]
[17/03/24 20:46:39:580][org.springframework.core.io.support.PropertiesLoaderSupport-loadProperties] Loading properties file from file [D:\zp\git\zp-kafka\zp-consumer-friend\build\resources\main\resource\db.properties]
[17/03/24 20:46:39:580][org.springframework.core.io.support.PropertiesLoaderSupport-loadProperties] Loading properties file from file [D:\zp\git\zp-kafka\zp-consumer-friend\build\resources\main\resource\resource.properties]
[17/03/24 20:46:41:058][org.apache.kafka.common.config.AbstractConfig-logAll] ConsumerConfig values:
auto.commit.interval.ms = 1000
auto.offset.reset = latest
bootstrap.servers = [192.168.71.11:9092, 192.168.71.12:9092, 192.168.71.13:9092]
check.crcs = true
client.id =
connections.max.idle.ms = 540000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = friend-group
heartbeat.interval.ms = 3000
interceptor.classes = null
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 1
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.ms = 50
request.timeout.ms = 305000
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 15000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer [17/03/24 20:46:41:066][org.apache.kafka.common.config.AbstractConfig-logAll] ConsumerConfig values:
auto.commit.interval.ms = 1000
auto.offset.reset = latest
bootstrap.servers = [192.168.71.11:9092, 192.168.71.12:9092, 192.168.71.13:9092]
check.crcs = true
client.id = consumer-1
connections.max.idle.ms = 540000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = friend-group
heartbeat.interval.ms = 3000
interceptor.classes = null
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 1
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.ms = 50
request.timeout.ms = 305000
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 15000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer [17/03/24 20:46:41:480][org.apache.kafka.common.utils.AppInfoParser$AppInfo-<init>] Kafka version : 0.10.1.1
[17/03/24 20:46:41:480][org.apache.kafka.common.utils.AppInfoParser$AppInfo-<init>] Kafka commitId : f10ef2720b03b247
[17/03/24 20:46:41:625][org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup-start] Starting beans in phase 0
[17/03/24 20:46:41:630][org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup-start] Starting beans in phase 0
[17/03/24 20:46:58:908][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:46:58:911][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator-onJoinPrepare] Revoking previously assigned partitions [] for group friend-group
[17/03/24 20:46:58:912][org.springframework.kafka.listener.AbstractMessageListenerContainer$2-onPartitionsRevoked] partitions revoked:[]
[17/03/24 20:46:58:912][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:46:59:240][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:46:59:558][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:46:59:559][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:46:59:566][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:46:59:688][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:46:59:689][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:46:59:692][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:46:59:807][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:46:59:808][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:46:59:814][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:46:59:941][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:46:59:942][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:00:520][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:00:828][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:00:829][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:00:841][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:01:182][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:01:183][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:02:911][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:03:352][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:03:354][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:03:946][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:04:394][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:04:398][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:04:450][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:04:676][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:04:678][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:04:687][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:05:307][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:05:308][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:05:655][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:06:182][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:06:183][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:06:190][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:08:076][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:08:077][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:11:343][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:11:514][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:11:516][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:11:548][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:11:698][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:11:700][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:11:714][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:13:814][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:13:816][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:47:17:025][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1-onSuccess] Successfully joined group friend-group with generation 1
[17/03/24 20:47:17:026][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator-onJoinComplete] Setting newly assigned partitions [] for group friend-group
[17/03/24 20:47:17:027][org.springframework.kafka.listener.AbstractMessageListenerContainer$2-onPartitionsAssigned] partitions assigned:[]
[17/03/24 20:47:23:055][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:23:237][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.12:9092 (id: 2147483646 rack: null) for group friend-group.
[17/03/24 20:47:35:384][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.12:9092 (id: 2147483646 rack: null) dead for group friend-group
[17/03/24 20:47:35:407][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:47:39:366][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-coordinatorDead] Marking the coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) dead for group friend-group
[17/03/24 20:47:39:408][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$GroupCoordinatorResponseHandler-onSuccess] Discovered coordinator 192.168.71.11:9092 (id: 2147483647 rack: null) for group friend-group.
[17/03/24 20:55:43:272][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator-onJoinPrepare] Revoking previously assigned partitions [] for group friend-group ====发送一条消息到生产者之后，消费者输出日志如下=========================================
[17/03/24 20:55:43:273][org.springframework.kafka.listener.AbstractMessageListenerContainer$2-onPartitionsRevoked] partitions revoked:[]
[17/03/24 20:55:43:273][org.apache.kafka.clients.consumer.internals.AbstractCoordinator-sendJoinGroupRequest] (Re-)joining group friend-group
[17/03/24 20:55:43:420][org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1-onSuccess] Successfully joined group friend-group with generation 2
[17/03/24 20:55:43:424][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator-onJoinComplete] Setting newly assigned partitions [friend-1, friend-0, friend-2] for group friend-group
[17/03/24 20:55:43:425][org.springframework.kafka.listener.AbstractMessageListenerContainer$2-onPartitionsAssigned] partitions assigned:[friend-1, friend-0, friend-2]
====发送第二条消息到生产者后，消费者输出日志如下======================================================
[17/03/24 21:05:05:408][com.zhaopin.consumer.ConsumerService-onMessage] ====onMessage====ConsumerRecord(topic = friend, partition = 0, offset = 0, CreateTime = 1490360705152, checksum = 2832597311, serialized key size = 48, serialized value size = 172, key = 5f974fa3-676c-4474-a260-b4417c36bd34||zhaopin123, value = {"data":{"friends":[{"friendName":"55445","friendPhone":"55445"},{"friendName":"55445","friendPhone":"55445"}],"userId":55555,"userPhone":"55445"},"requestId":"zhaopin123"})
[17/03/24 21:05:06:137][com.alibaba.druid.pool.DruidDataSource-validationQueryCheck] testWhileIdle is true, validationQuery not set
[17/03/24 21:05:06:181][com.alibaba.druid.pool.DruidDataSource-init] {dataSource-1} inited =====数据库friend表中增加了2条数据===============================================
287835 55555 55445 1 0 55445 0 55445
287836 55555 55445 1 0 55445 0 55445
解决kafka集群由于默认的__consumer_offsets这个topic的默认的副本数为1而存在的单点故障问题的更多相关文章
- Docker实战之Kafka集群
1. 概述 Apache Kafka 是一个快速.可扩展的.高吞吐.可容错的分布式发布订阅消息系统.其具有高吞吐量.内置分区.支持数据副本和容错的特性,适合在大规模消息处理场景中使用. 笔者之前在物联 ...
- zookeeper与Kafka集群搭建及python代码测试
Kafka初识 1.Kafka使用背景 在我们大量使用分布式数据库.分布式计算集群的时候,是否会遇到这样的一些问题: 我们想分析下用户行为(pageviews),以便我们设计出更好的广告位 我想对用户 ...
- Kafka集群管理工具kafka-manager的安装使用
一.kafka-manager简介 kafka-manager是目前最受欢迎的kafka集群管理工具,最早由雅虎开源,用户可以在Web界面执行一些简单的集群管理操作.具体支持以下内容: 管理多个集群 ...
- Linux Kafka集群管理工具kafka-manager的安装使用
一.kafka-manager简介 kafka-manager是目前最受欢迎的kafka集群管理工具,最早由雅虎开源,用户可以在Web界面执行一些简单的集群管理操作.具体支持以下内容: 管理多个集群 ...
- zookeeper+kafka集群的安装部署
准备工作 上传 zookeeper-3.4.6.tar.gz.scala-2.11.4.tgz.kafka_2.9.2-0.8.1.1.tgz.slf4j-1.7.6.zip 至/usr/local目 ...
- (Linux环境Kafka集群安装配置及常用命令
Linux环境Kafka集群安装配置及常用命令 Kafka 消息队列内部实现原理 Kafka架构 一.下载Kafka安装包 二.Kafka安装包的解压 三.设置环境变量 四.配置kafka文件 4.1 ...
- docker-compose部署zk集群、kafka集群以及kafka-manager，及其遇到的问题和解决
zk集群docker-compose.yml 1.新建网络 docker network create --driver bridge --subnet --gateway 172.23.0.1 zo ...
- kafka集群在消息消费出现无法找到topic分区的处理解决
最近几天在做spark数据同步过程中,中间通过kafka集群处理消息,每次同步到一半就会出现同步不了 查看日志如下: 最开始看到这个问题很懵逼,完全找不到解决问题的切入口,期间也询问了架构师-因为这个 ...
- kafka集群及监控部署
1. kafka的定义 kafka是一个分布式消息系统,由linkedin使用scala编写,用作LinkedIn的活动流(Activity Stream)和运营数据处理管道(Pipeline)的基础 ...
随机推荐
- ios中一级导航
效果图 第一步先下载文件 http://pan.baidu.com/share/link?shareid=2148445651&uk=9237761877 将目录引入到xcode中 在xcod ...
- oracle中生成大批量数据的方法-下
方法五:使用PLSQL的数据生成器 首先测试环境建立:dept表 CREATE TABLE dept(deptno NUMBER(6),dname VARCHAR2(20),loc VARCHAR2( ...
- codeM 2018 资格赛
比赛链接:https://www.nowcoder.com/activity/2018codem/index?from=meituan 1.下单 给定若干商品,可以选择打折.满减两种方式. #incl ...
- 我的第一个Raspberry PI装置
好吧,我先承认是这是从书上学习的第一个示例. 我入手的是Raspberry红版,版本2,看到这么小巧的卡片电脑可以做这么多事情,真是让人惊喜! 安装系统等都很顺利,启动.安装程序.远程桌面.SSH登录 ...
- 【LeetCode】215. Kth Largest Element in an Array (2 solutions)
Kth Largest Element in an Array Find the kth largest element in an unsorted array. Note that it is t ...
- List<String> bikeList = Arrays.asList(bikeuuids);
最近项目中 List<String> bikeList = Arrays.asList(bikeuuids);报错 而且console里面没有特别有用的 bikeList.add(&qu ...
- HDU 3018 Ant Trip （欧拉回路）
Ant Trip Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 32768/32768 K (Java/Others)Total Su ...
- iOS中的copy
原文:http://www.jianshu.com/p/5254f1277dba 内存的栈区 : 由编译器自动分配释放, 存放函数的参数值, 局部变量的值等. 其操作方式类似于数据结构中的栈. 内存的 ...
- Xcode 常用插件
1.Xcode 插件 从 Xcode 8 起 Apple 禁用 Xcode 插件. 1)Xcode 插件安装目录 ~/library/Application Support/Developer/Sha ...
- centos yum源
#remi的源 rpm -ivh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm rpm --import /etc/pki/r ...