3 Kafka两个版本在Zookeeper的元数据存储

0.8

[consumers, admin, config, controller, brokers, controller_epoch]

这些节点都归于谁管理

consumers: 0.8版本的客户端Consumer
admin:管理命令的脚本:kafka-preferred-replica-election.sh,
config:管理命令的脚本:kafka-topics.sh
controller:所有的broker监控这个。
brokers:kafka-topics.sh建立topic
controller_epoch:

1.consumers

1.1 ids 代表消费者线程

/consumers/[groupId]/ids/[consumerIdString]:每个consumer都有一个唯一的ID(consumerId可以通过配置文件指定,也可以由系统生成),此id用来标记消费者信息.

[zk: 10.39.1.66:22181(CONNECTED) 17] ls /consumers/guest_tblog_expo/ids

[guest_tblog_expo_yz4834.hadoop.data.sina.com.cn-1472452954131-13232511, guest_tblog_expo_yz4879.hadoop.data.sina.com.cn-1472452955003-3ff3d939]

这两个节点是临时的znode。

1.2 Consumer owner 表示partition和consumer线程的对应关系

/consumers/[groupId]/owners/[topic]/[partitionId] -> consumerIdString + threadId 索引编号

[zk: 10.39.1.66:22181(CONNECTED) 23] get /consumers/guest_tblog_expo/owners/openapi_exposure/0

guest_tblog_expo_yz4845.hadoop.data.sina.com.cn-1472452954464-2f40ad23-0

1.3 Consumer offset 表示partition被消费到的Offset位置

用来跟踪每个consumer目前所消费的partition中最大的offset

此znode为持久节点,可以看出offset跟group_id有关,以表明当消费者组(consumer group)中一个消费者失效,

重新触发balance,其他consumer可以继续消费.

/consumers/[groupId]/offsets/[topic]/[partitionId] -> long (offset)

[zk: 10.39.1.66:22181(CONNECTED) 26] get  /consumers/guest_tblog_expo/offsets/openapi_exposure/0

50793695368

2 admin 表示Controller管理的命令

常有的目录结构

/admin/reassign_partitions 重新分配partition
/admin/preferred_replica_election 重新preffed partition leader
/admin/delete_topics 删除topic

3 config 配置

主要包括三个的配置

topic : 存储topic集群属性之外的属性，比如修改了一个新的topic属性
clients：Client and Topic configuration overrides: The content of both znodes has the same structure（consumer或者Producer的配置地方）
changes: Config Change notification (Topic and Client config修改通知的地方)

clients 和 changes 在0.8 版本中也许没有。

[zk: localhost:2181(CONNECTED) 3] get /kafka10/config/topics/test2

{"version":1,"config":{"retention.ms":"3600000"}}

4 controller

/controller -> int (broker id of the controller) 存储center controller中央控制器所在kafka broker的信息

0.8 版本的存储内容

[zk: 10.39.1.66:22181(CONNECTED) 34] get /controller

3

0.10 版本的存储内容

[zk: localhost:2181(CONNECTED) 12] get /kafka10/controller

{"version":1,"brokerid":40107,"timestamp":"1479456552014"}

两个版本的存储内容有差别了。

6 controller_epoch 表示controller选举的次数

/controller_epoch -> int (epoch) 此值为一个数字,kafka集群中第一个broker第一次启动时为1，以后只要集群中center controller中央控制器所在broker变更或挂掉，就会重新选举新的center controller，每次center controller变更controller_epoch值就会 + 1;

[zk: localhost:2181(CONNECTED) 13] get /kafka10/controller_epoch

9

什么时候会重新选举controller？有broker与zk失去联系的时候。

5 brokers

存集群broker和topic的元信息:

/brokers/topics/[topic] :存储某个topic的partitions所有分配信息
``

5.1 topic注册信息

/brokers/topics/[topic]: 存储某个topic的partitions所有分配信息
/brokers/topics/[topic]/partitions/[0...N] : partition的个数
/brokers/topics/[topic]/partitions/[partitionId]/state ： partition的信息

[zk: localhost:2181(CONNECTED) 20] get /kafka10/brokers/topics/test2/partitions/0/state

{"controller_epoch":9,"leader":40106,"version":1,"leader_epoch":13,"isr":[40106,40107]}

[zk: 10.39.1.66:22181(CONNECTED) 38] get /brokers/topics/openapi_advertise/partitions/0/state

{ "controller_epoch":33, "isr":[ 13, 7 ], "leader":13, "leader_epoch":29, "version":1 }

5.2 Broker注册信息

/brokers/ids/[0...N] : 每个broker的配置文件中都需要指定一个数字类型的id(全局不可重复),此节点为临时znode(EPHEMERAL)

0.10 和 0.8 信息不一样。

[zk: 10.39.1.66:22181(CONNECTED) 39] get /brokers/ids/48152

{ "host":"yz48152.hadoop.data.sina.com.cn", "jmx_port":9999, "port":19092, "version":1 }

[zk: localhost:2181(CONNECTED) 22] get /kafka10/brokers/ids/4099

{"jmx_port":9999,"timestamp":"1479456552292","endpoints":["PLAINTEXT://10.39.40.99:9110"],"host":"10.39.40.99","version":3,"port":9110}

5.3 seqs 0.10 版本特有

未知？？？？

0.10

[consumers, admin, config, controller, brokers, controller_epoch, isr_change_notification, kafka-acl, kafka-acl-changes]

10.1 isr_change_notification

0.10版本多了3组znode

/isr_change_notification/isr_change_x: Gets created when ISR is changed at any broker, controller watches for these notifications and sends MetadataUpdateRequest to all brokers. (ISR改变的时候，通知Controller，发送更新请求给all brokers.)

10.2 kafka-acl

关于授权，一共有三种：

Cluster:Create集群权限（官方写的是“Create on Cluster”）
Group: ConsumerGroup权限
Topic：读写权限

[zk: localhost:2181(CONNECTED) 10] get /kafka10/kafka-acl/Cluster/kafka-cluster

{"version":1,"acls":[{"principal":"User:mapiRsyslogForwardKafka","permissionType":"Allow","operation":"Create","host":"*"},,{"principal":"User:mapiRsyslogForwardKafka","permissionType":"Allow","operation":"Create","host":"10.77.*"}]}

[zk: localhost:2181(CONNECTED) 12] get /kafka10/kafka-acl/Group/superConsumer

{"version":1,"acls":[{"principal":"User:fengchao1","permissionType":"Allow","operation":"Read","host":"*"}]}

[zk: localhost:2181(CONNECTED) 13] get /kafka10/kafka-acl/Topic/client_ad_realread

{"version":1,"acls":[{"principal":"User:fengchao1","permissionType":"Allow","operation":"Read","host":"*"},{"principal":"User:fengchao1","permissionType":"Allow","operation":"Describe","host":"*"}]}

10.3 kafka-acl-changes

存储acl的变更信息。由授权的脚本触发。

[zk: localhost:2181(CONNECTED) 17] get /kafka10/kafka-acl-changes/acl_changes_0000000053

Topic:test2

关于这些节点里的内容格式，请查看kafka zk wiki