kafka3 本地目录结构以及在在zk上的znode
一 kafka本地目录结构
[root@hadoop ~]# cd /tmp/kafka-logs1 [root@hadoop kafka-logs1]# find . |
搭建单节点多broker的kafka后，启动zk和kafka。
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# zookeeper-server-start.sh config/zookeeper.properties
...
[root@hadoop kafka]# kafka-server-start.sh config/server0.properties &
...
[root@hadoop kafka]# kafka-server-start.sh config/server1.properties &
...
[root@hadoop kafka]# kafka-server-start.sh config/server2.properties &
... [root@hadoop ~]# jps
QuorumPeerMain
Kafka
Kafka
Kafka
Jps
创建kafka集群时我已经创建了一个主题test02,现在我们再创建一个主题mytest（2个分区）
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# kafka-topics.sh --create --zookeeper localhost: --replication-factor --partitions --topic mytest
Created topic "mytest".
查看日志目录可以发现，3个日志目录几乎是一致的（__consumer_offsets-0是个什么鬼？）
test02和mytest副本数均是3，所以3个broker的log下面均有；
test02只有一个分区，mytest有2个分区，所以有一个test02目录，和2个mytest目录。
[root@hadoop ~]# ll /tmp/kafka-logs0
总用量
-rw-r--r-- root root 8月 : cleaner-offset-checkpoint
drwxr-xr-x root root 8月 : __consumer_offsets-
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 8月 : meta.properties
drwxr-xr-x root root 8月 : mytest- #命名格式：主题-分区
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint
drwxr-xr-x root root 8月 : test02- [root@hadoop ~]# ll /tmp/kafka-logs1
总用量
-rw-r--r-- root root 7月 : cleaner-offset-checkpoint
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 7月 : meta.properties
drwxr-xr-x root root 8月 : mytest-
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint
drwxr-xr-x root root 8月 : test02- [root@hadoop ~]# ll /tmp/kafka-logs2
总用量
-rw-r--r-- root root 7月 : cleaner-offset-checkpoint
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 7月 : meta.properties
drwxr-xr-x root root 8月 : mytest-
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint
drwxr-xr-x root root 8月 : test02-
查看主题目录
[root@hadoop ~]# ll /tmp/kafka-logs0/test02-/
总用量
-rw-r--r-- root root 8月 : .index
-rw-r--r-- root root 8月 : .log
-rw-r--r-- root root 8月 : .timeindex
-rw-r--r-- root root 8月 : .snapshot
-rw-r--r-- root root 8月 : leader-epoch-checkpoint
查看元数据信息
[root@hadoop ~]# cat /tmp/kafka-logs0/meta.properties
version=
broker.id=
[root@hadoop ~]# cat /tmp/kafka-logs1/meta.properties
version=
broker.id=
[root@hadoop ~]# cat /tmp/kafka-logs2/meta.properties
version=
broker.id=
向 mytest主题 生产消息并消费消息，可以看出消息分别保存在了不同分区的log里
#生产消息
[root@hadoop kafka]# kafka-console-producer.sh --broker-list localhost:,localhost:,localhost: --topic mytest
>hello kafka
>hello world #消费消息
[root@hadoop kafka]# kafka-console-consumer.sh --bootstrap-server localhost:,localhost:,localhost: --topic mytest --from-beginning
hello kafka
hello world #可以看出，消息分别保存在了不同分区的log里
[root@hadoop ~]# cat /tmp/kafka-logs0/mytest-/.log
C}_Me Ye Yÿÿÿÿÿÿÿÿÿÿÿÿÿÿ"hello kafka[root@hadoop ~]#
[root@hadoop ~]# cat /tmp/kafka-logs0/mytest-/.log
e ꛁe ꞿÿÿÿÿÿÿÿÿÿÿÿÿÿ"hello world[root@hadoop ~]#
二 kafka在zk上的znode
|
1./controller //data = {"version":1,"brokerid":0,"timestamp":"1533396512695"} 2./controller_epoch //data=27#不清楚什么意思，貌似第一次启动时是1，难道是kafka启动过的次数？ 3./brokers/ids //实时维护active的broker /brokers/topics /brokers/seqid 4./admin/delete_topics 5./isr_change_notification 6./consumers 7./config 注意：productor不在zk注册 |
启动zk客户端
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# zkCli.sh -server hadoop: #启动zk客户端
...
查看根目录
[zk: hadoop:(CONNECTED) ] ls / #查看znode
[cluster, controller_epoch, controller, brokers, zookeeper, admin, isr_change_notification, consumers,
log_dir_event_notification, latest_producer_id_block, config]
查看/controller
[zk: hadoop:(CONNECTED) ] ls /controller
[]
[zk: hadoop:(CONNECTED) ] get /controller
#这里的brokerid为0意思是kafka集群的leader为0
#如果集群中有多个broker，将leader杀死后会发现这里的brokerid变化。
{"version":,"brokerid":,"timestamp":""}
cZxid = 0x513
ctime = Sat Aug :: CST
mZxid = 0x513
mtime = Sat Aug :: CST
pZxid = 0x513
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x10000711d710001
dataLength =
numChildren =
查看/controller_epoch
[zk: hadoop:(CONNECTED) ] ls /controller_epoch
[]
[zk: hadoop:(CONNECTED) ] get /controller_epoch cZxid = 0x1c
ctime = Tue Jul :: CST
mZxid = 0x514
mtime = Sat Aug :: CST
pZxid = 0x1c
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/brokers
[zk: hadoop:(CONNECTED) ] get /brokers
null
cZxid = 0x4
ctime = Tue Jul :: CST
mZxid = 0x4
mtime = Tue Jul :: CST
pZxid = 0xd
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren = #3个孩子分别是ids, topics, seqid [zk: hadoop:(CONNECTED) ] ls /brokers
[ids, topics, seqid]
查看/brokers/ids
[zk: hadoop:(CONNECTED) ] ls /brokers/ids
#显示kafka集群中的所有active的brokerid
#如果杀死broker ，这里将会显示[, ]
[, , ] [zk: hadoop:(CONNECTED) 8] get /brokers/ids/
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://hadoop:9092"],
"jmx_port":-,"host":"hadoop","timestamp":"","port":,"version":}
cZxid = 0x55f
ctime = Sun Aug :: CST
mZxid = 0x55f
mtime = Sun Aug :: CST
pZxid = 0x55f
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x10000711d710005
dataLength =
numChildren =
查看/brokers/topics
[zk: hadoop:(CONNECTED) 9] ls /brokers/topics
[mytest, test02, __consumer_offsets] [zk: hadoop:(CONNECTED) ] ls /brokers/topics/mytest
[partitions]
[zk: hadoop:(CONNECTED) ] ls /brokers/topics/mytest/partitions #显示分区个数：mytest主题有2个分区，分别为0和1
[, ]
[zk: hadoop:(CONNECTED) 12] ls /brokers/topics/mytest/partitions/
[state]
[zk: hadoop:(CONNECTED) 13] ls /brokers/topics/mytest/partitions//state
[] #以下可以看出每个分区拥有不同的leader
[zk: hadoop:(CONNECTED) 14] get /brokers/topics/mytest/partitions//state
{"controller_epoch":,"leader":,"version":,"leader_epoch":,"isr":[,,]}
cZxid = 0x5a2
ctime = Sun Aug :: CST
mZxid = 0x5a2
mtime = Sun Aug :: CST
pZxid = 0x5a2
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
[zk: hadoop:(CONNECTED) ] get /brokers/topics/mytest/partitions//state
{"controller_epoch":,"leader":,"version":,"leader_epoch":,"isr":[,,]}
cZxid = 0x5a1
ctime = Sun Aug :: CST
mZxid = 0x5a1
mtime = Sun Aug :: CST
pZxid = 0x5a1
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/brokers/seqid
[zk: hadoop:(CONNECTED) 16] ls /brokers/seqid
[]
[zk: hadoop:(CONNECTED) ] get /brokers/seqid
null
cZxid = 0xd
ctime = Tue Jul :: CST
mZxid = 0xd
mtime = Tue Jul :: CST
pZxid = 0xd
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/admin/delete_topics
[zk: hadoop:(CONNECTED) ] ls /admin
[delete_topics]
[zk: hadoop:(CONNECTED) ] ls /admin/delete_topics
[]
查看/isr_change_notification
[zk: hadoop:(CONNECTED) 20] ls /isr_change_notification
[]
[zk: hadoop:(CONNECTED) ] get /isr_change_notification
null
cZxid = 0xe
ctime = Tue Jul :: CST
mZxid = 0xe
mtime = Tue Jul :: CST
pZxid = 0x544
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/consumers
#不知为何，我这里一直显示为空，按理说启动消费者之后这里应该显示相应信息的
[zk: hadoop:(CONNECTED) ] ls /consumers
[]
[zk: hadoop:(CONNECTED) 23] get /consumers
null
cZxid = 0x2
ctime = Tue Jul :: CST
mZxid = 0x2
mtime = Tue Jul :: CST
pZxid = 0x2
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/config
[zk: hadoop:(CONNECTED) 24] ls /config
[changes, clients, brokers, topics, users]
[zk: hadoop:(CONNECTED) 25] ls /config/changes
[]
[zk: hadoop:(CONNECTED) 26] ls /config/clients
[]
[zk: hadoop:(CONNECTED) ] ls /config/brokers
[]
[zk: hadoop:(CONNECTED) ] ls /config/topics #与ls /brokers/topics结果一致
[mytest, test02, __consumer_offsets]
[zk: hadoop:(CONNECTED) ] ls /config/users
[]
尝试杀死集群的leader后，查看相对应的znode变化
#.杀死leader
[root@hadoop ~]# ps -ef|grep server0.properties #找到server0的进程号为4791
root 8月04 pts/ :: ...信息太多，忽略.../server0.properties
root : pts/ :: grep --color=auto server0.properties
[root@hadoop ~]# kill - #杀死进程
[root@hadoop ~]# ps -ef|grep server0.properties #再次查看
root : pts/ :: grep --color=auto server0.properties #.查看/controller：brokerid由0变为1
[zk: hadoop:(CONNECTED) 30] get /controller
{"version":,"brokerid":,"timestamp":""}
cZxid = 0x54c
ctime = Sun Aug :: CST
mZxid = 0x54c
mtime = Sun Aug :: CST
pZxid = 0x54c
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x10000711d710002
dataLength =
numChildren = #.查看/brokers/ids：active的broker只剩下1和2了
[zk: hadoop:(CONNECTED) ] ls /brokers/ids
[, ] #.重新启动broker
[root@hadoop kafka]# kafka-server-start.sh config/server0.properties &
尝试删除一个主题test02，查看相对应的znode变化
#.删除主题
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# kafka-topics.sh --zookeeper localhost: --delete --topic test02
Topic test02 is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true. #.查看主题：没有结果（应该有结果？）
[root@hadoop kafka]# kafka-topics.sh --describe --zookeeper localhost: --topic test02 #.查看log目录下test02-0目录是否存在：不存在了
[root@hadoop kafka]# ll /tmp/kafka-logs0 #test02目录没有了
总用量
-rw-r--r-- root root 8月 : cleaner-offset-checkpoint
drwxr-xr-x root root 8月 : __consumer_offsets-
...
drwxr-xr-x root root 8月 : __consumer_offsets-
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 8月 : meta.properties
drwxr-xr-x root root 8月 : mytest-
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint #.查看/admin/delete_topics：为空。按理说这里应该显示删除的主题test02
[zk: hadoop:(CONNECTED) 32] ls /admin/delete_topics
[] #.查看/brokers/topics：这里显示test02确实被删除了
[zk: hadoop:(CONNECTED) 33] ls /brokers/topics
[mytest, __consumer_offsets] #.查看/config/topics：
[zk: hadoop:(CONNECTED) 34] ls /config/topics
[mytest, __consumer_offsets] #以上均是我的实际操作结果，与老师演示的有一些出入，暂时无法解释原因
kafka3 本地目录结构以及在在zk上的znode的更多相关文章
- svn本地目录结构for window
演示内容: 使用svn目录结构来进行备份正式版和修复版本,最终合并修复版本.主干上的版本. 使用工具: visualSVN server 服务器软件工具 TortoiseSVN客户端工具 1.服务器的 ...
- Twitter Storm源代码分析之Nimbus/Supervisor本地目录结构
storm集群里面工作机器分为两种一种是nimbus, 一种是supervisor, 他们通过zookeeper来进行交互,nimbus通过zookeeper来发布一些指令,supervisor去读z ...
- 浅谈android中的目录结构
之前在android游戏开发中就遇到本地数据存储的问题:一般情形之下就将动态数据写入SD中存储,在没有SD卡的手机上就需另作处理了;再有在开发android应用的过程中,总要去调试APP,安装时又想去 ...
- Linux文件系统的目录结构
Linux下的文件系统为树形结构,入口为/ 树形结构下的文件目录: 无论哪个版本的Linux系统,都有这些目录,这些目录应该是标准的.各个Linux发行版本会存在一些小小的差异,但总体来说,还是大体差 ...
- Linux 文件系统的目录结构
http://www.jb51.net/LINUXjishu/151820.htmlLinux下的文件系统为树形结构,入口为/ 树形结构下的文件目录: 无论哪个版本的Linux系统,都有这些目录,这些 ...
- chromiun 学习《二》 目录结构 +启动流程
1.chromium的目录结构. 2.先上分析图一张.主要是从BrowserMain进程进行分析的.
- thinkphp学习笔记1—目录结构和命名规则
原文:thinkphp学习笔记1-目录结构和命名规则 最近开始学习thinkphp,在下不才,很多的问题看不明白所以想拿出来,恕我大胆发在首页上,希望看到的人能为我答疑解惑,这样大家有个互动,学起来快 ...
- CentOS版本区别及 Linux目录结构及其详解
CentOS 7.0体验与之前版本的不同http://www.linuxidc.com/Linux/2014-07/104196.htm CentOS版本选择http://www.centoscn.c ...
- mybatis学习笔记（六）使用generator生成mybatis基础配置代码和目录结构
原文:http://blog.csdn.net/oh_mourinho/article/details/51463413 创建maven项目 <span style="font-siz ...
随机推荐
- 【QT】第一个QT程序（点击按钮，显示特定文本）
1.基类选 QWidget 2.添加UI Btnshowhello 按钮 labelhello 文本标签 要注意:文本字样和对象名的区别! 对象名和UI部件是一一对应的关系. 修改对象名和修改文本字样 ...
- [Converge] Batch Normalisation
参考:https://www.zhihu.com/question/38102762 参考:CNN和RNN中如何引入BatchNorm 论文:Batch Normalization: Accelera ...
- web java -- 连接池 -- 概述
1. 连接池的实现原理 1. 创建连接池 首先要创建一个静态的连接池.这里的“静态”是指池中的连接时在系统初始化时就分配好的,并且不能够随意关闭.Java 提供了很多容器类可用来构建连接池,例如Vec ...
- VS Code非英语版本连接TFS错误解决方案
使用VS Code连接TFS时,提示以下错误: (team) It appears you have configured a non-English version of the TF execut ...
- ubuntu上安装docker
author : headsen chen date : 2019-03-06 16:36:12 apt-get remove docker docker-engine docker-ce dock ...
- C++设计实现一个不能被继承的类
C++不同于Java,Java中被final关键字修饰的类不能被继承,C++能实现不被继承的类,但是需要自己实现. 为了使类不被继承,最好的办法是使子类不能构造父类的部分,此时子类就无法实例化整个子类 ...
- ERP项目实施记录01
工厂8月中与某ERP服务商签约,至今已经过去4个月,顾问服务了13人天,进行了10次培训. 内部产生项目文档1个:物料编码方案 制度:0个 流程:无 因ERP服务商不是针对本行业的,在BOM和生产计划 ...
- eclipse搭建j2ee
Tomcat环境变量设置,分别添加3个系统变量 CATALINA_BASE E:/tomcat7 CATALINA_HOME E:/tomcat7 CATALINA_TMPDIR Etomcat7/t ...
- codeforces#525 Div2---ABC
A---Ehab and another constriction problem https://codeforc.es/contest/1088/problem/A 题意:给定一个数$x$找两个在 ...
- okvis源码解读
为相机图像提取brisk特征点,并设置描述子的提取方向为重力加速度在世界坐标系中的方向(0,0,1)