kafka3 本地目录结构以及在在zk上的znode
一 kafka本地目录结构
[root@hadoop ~]# cd /tmp/kafka-logs1 [root@hadoop kafka-logs1]# find . |
搭建单节点多broker的kafka后，启动zk和kafka。
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# zookeeper-server-start.sh config/zookeeper.properties
...
[root@hadoop kafka]# kafka-server-start.sh config/server0.properties &
...
[root@hadoop kafka]# kafka-server-start.sh config/server1.properties &
...
[root@hadoop kafka]# kafka-server-start.sh config/server2.properties &
... [root@hadoop ~]# jps
QuorumPeerMain
Kafka
Kafka
Kafka
Jps
创建kafka集群时我已经创建了一个主题test02,现在我们再创建一个主题mytest（2个分区）
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# kafka-topics.sh --create --zookeeper localhost: --replication-factor --partitions --topic mytest
Created topic "mytest".
查看日志目录可以发现，3个日志目录几乎是一致的（__consumer_offsets-0是个什么鬼？）
test02和mytest副本数均是3，所以3个broker的log下面均有；
test02只有一个分区，mytest有2个分区，所以有一个test02目录，和2个mytest目录。
[root@hadoop ~]# ll /tmp/kafka-logs0
总用量
-rw-r--r-- root root 8月 : cleaner-offset-checkpoint
drwxr-xr-x root root 8月 : __consumer_offsets-
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 8月 : meta.properties
drwxr-xr-x root root 8月 : mytest- #命名格式：主题-分区
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint
drwxr-xr-x root root 8月 : test02- [root@hadoop ~]# ll /tmp/kafka-logs1
总用量
-rw-r--r-- root root 7月 : cleaner-offset-checkpoint
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 7月 : meta.properties
drwxr-xr-x root root 8月 : mytest-
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint
drwxr-xr-x root root 8月 : test02- [root@hadoop ~]# ll /tmp/kafka-logs2
总用量
-rw-r--r-- root root 7月 : cleaner-offset-checkpoint
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 7月 : meta.properties
drwxr-xr-x root root 8月 : mytest-
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint
drwxr-xr-x root root 8月 : test02-
查看主题目录
[root@hadoop ~]# ll /tmp/kafka-logs0/test02-/
总用量
-rw-r--r-- root root 8月 : .index
-rw-r--r-- root root 8月 : .log
-rw-r--r-- root root 8月 : .timeindex
-rw-r--r-- root root 8月 : .snapshot
-rw-r--r-- root root 8月 : leader-epoch-checkpoint
查看元数据信息
[root@hadoop ~]# cat /tmp/kafka-logs0/meta.properties
version=
broker.id=
[root@hadoop ~]# cat /tmp/kafka-logs1/meta.properties
version=
broker.id=
[root@hadoop ~]# cat /tmp/kafka-logs2/meta.properties
version=
broker.id=
向 mytest主题 生产消息并消费消息，可以看出消息分别保存在了不同分区的log里
#生产消息
[root@hadoop kafka]# kafka-console-producer.sh --broker-list localhost:,localhost:,localhost: --topic mytest
>hello kafka
>hello world #消费消息
[root@hadoop kafka]# kafka-console-consumer.sh --bootstrap-server localhost:,localhost:,localhost: --topic mytest --from-beginning
hello kafka
hello world #可以看出，消息分别保存在了不同分区的log里
[root@hadoop ~]# cat /tmp/kafka-logs0/mytest-/.log
C}_Me Ye Yÿÿÿÿÿÿÿÿÿÿÿÿÿÿ"hello kafka[root@hadoop ~]#
[root@hadoop ~]# cat /tmp/kafka-logs0/mytest-/.log
e ꛁe ꞿÿÿÿÿÿÿÿÿÿÿÿÿÿ"hello world[root@hadoop ~]#
二 kafka在zk上的znode
|
1./controller //data = {"version":1,"brokerid":0,"timestamp":"1533396512695"} 2./controller_epoch //data=27#不清楚什么意思，貌似第一次启动时是1，难道是kafka启动过的次数？ 3./brokers/ids //实时维护active的broker /brokers/topics /brokers/seqid 4./admin/delete_topics 5./isr_change_notification 6./consumers 7./config 注意：productor不在zk注册 |
启动zk客户端
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# zkCli.sh -server hadoop: #启动zk客户端
...
查看根目录
[zk: hadoop:(CONNECTED) ] ls / #查看znode
[cluster, controller_epoch, controller, brokers, zookeeper, admin, isr_change_notification, consumers,
log_dir_event_notification, latest_producer_id_block, config]
查看/controller
[zk: hadoop:(CONNECTED) ] ls /controller
[]
[zk: hadoop:(CONNECTED) ] get /controller
#这里的brokerid为0意思是kafka集群的leader为0
#如果集群中有多个broker，将leader杀死后会发现这里的brokerid变化。
{"version":,"brokerid":,"timestamp":""}
cZxid = 0x513
ctime = Sat Aug :: CST
mZxid = 0x513
mtime = Sat Aug :: CST
pZxid = 0x513
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x10000711d710001
dataLength =
numChildren =
查看/controller_epoch
[zk: hadoop:(CONNECTED) ] ls /controller_epoch
[]
[zk: hadoop:(CONNECTED) ] get /controller_epoch cZxid = 0x1c
ctime = Tue Jul :: CST
mZxid = 0x514
mtime = Sat Aug :: CST
pZxid = 0x1c
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/brokers
[zk: hadoop:(CONNECTED) ] get /brokers
null
cZxid = 0x4
ctime = Tue Jul :: CST
mZxid = 0x4
mtime = Tue Jul :: CST
pZxid = 0xd
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren = #3个孩子分别是ids, topics, seqid [zk: hadoop:(CONNECTED) ] ls /brokers
[ids, topics, seqid]
查看/brokers/ids
[zk: hadoop:(CONNECTED) ] ls /brokers/ids
#显示kafka集群中的所有active的brokerid
#如果杀死broker ，这里将会显示[, ]
[, , ] [zk: hadoop:(CONNECTED) 8] get /brokers/ids/
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://hadoop:9092"],
"jmx_port":-,"host":"hadoop","timestamp":"","port":,"version":}
cZxid = 0x55f
ctime = Sun Aug :: CST
mZxid = 0x55f
mtime = Sun Aug :: CST
pZxid = 0x55f
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x10000711d710005
dataLength =
numChildren =
查看/brokers/topics
[zk: hadoop:(CONNECTED) 9] ls /brokers/topics
[mytest, test02, __consumer_offsets] [zk: hadoop:(CONNECTED) ] ls /brokers/topics/mytest
[partitions]
[zk: hadoop:(CONNECTED) ] ls /brokers/topics/mytest/partitions #显示分区个数：mytest主题有2个分区，分别为0和1
[, ]
[zk: hadoop:(CONNECTED) 12] ls /brokers/topics/mytest/partitions/
[state]
[zk: hadoop:(CONNECTED) 13] ls /brokers/topics/mytest/partitions//state
[] #以下可以看出每个分区拥有不同的leader
[zk: hadoop:(CONNECTED) 14] get /brokers/topics/mytest/partitions//state
{"controller_epoch":,"leader":,"version":,"leader_epoch":,"isr":[,,]}
cZxid = 0x5a2
ctime = Sun Aug :: CST
mZxid = 0x5a2
mtime = Sun Aug :: CST
pZxid = 0x5a2
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
[zk: hadoop:(CONNECTED) ] get /brokers/topics/mytest/partitions//state
{"controller_epoch":,"leader":,"version":,"leader_epoch":,"isr":[,,]}
cZxid = 0x5a1
ctime = Sun Aug :: CST
mZxid = 0x5a1
mtime = Sun Aug :: CST
pZxid = 0x5a1
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/brokers/seqid
[zk: hadoop:(CONNECTED) 16] ls /brokers/seqid
[]
[zk: hadoop:(CONNECTED) ] get /brokers/seqid
null
cZxid = 0xd
ctime = Tue Jul :: CST
mZxid = 0xd
mtime = Tue Jul :: CST
pZxid = 0xd
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/admin/delete_topics
[zk: hadoop:(CONNECTED) ] ls /admin
[delete_topics]
[zk: hadoop:(CONNECTED) ] ls /admin/delete_topics
[]
查看/isr_change_notification
[zk: hadoop:(CONNECTED) 20] ls /isr_change_notification
[]
[zk: hadoop:(CONNECTED) ] get /isr_change_notification
null
cZxid = 0xe
ctime = Tue Jul :: CST
mZxid = 0xe
mtime = Tue Jul :: CST
pZxid = 0x544
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/consumers
#不知为何，我这里一直显示为空，按理说启动消费者之后这里应该显示相应信息的
[zk: hadoop:(CONNECTED) ] ls /consumers
[]
[zk: hadoop:(CONNECTED) 23] get /consumers
null
cZxid = 0x2
ctime = Tue Jul :: CST
mZxid = 0x2
mtime = Tue Jul :: CST
pZxid = 0x2
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/config
[zk: hadoop:(CONNECTED) 24] ls /config
[changes, clients, brokers, topics, users]
[zk: hadoop:(CONNECTED) 25] ls /config/changes
[]
[zk: hadoop:(CONNECTED) 26] ls /config/clients
[]
[zk: hadoop:(CONNECTED) ] ls /config/brokers
[]
[zk: hadoop:(CONNECTED) ] ls /config/topics #与ls /brokers/topics结果一致
[mytest, test02, __consumer_offsets]
[zk: hadoop:(CONNECTED) ] ls /config/users
[]
尝试杀死集群的leader后，查看相对应的znode变化
#.杀死leader
[root@hadoop ~]# ps -ef|grep server0.properties #找到server0的进程号为4791
root 8月04 pts/ :: ...信息太多，忽略.../server0.properties
root : pts/ :: grep --color=auto server0.properties
[root@hadoop ~]# kill - #杀死进程
[root@hadoop ~]# ps -ef|grep server0.properties #再次查看
root : pts/ :: grep --color=auto server0.properties #.查看/controller：brokerid由0变为1
[zk: hadoop:(CONNECTED) 30] get /controller
{"version":,"brokerid":,"timestamp":""}
cZxid = 0x54c
ctime = Sun Aug :: CST
mZxid = 0x54c
mtime = Sun Aug :: CST
pZxid = 0x54c
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x10000711d710002
dataLength =
numChildren = #.查看/brokers/ids：active的broker只剩下1和2了
[zk: hadoop:(CONNECTED) ] ls /brokers/ids
[, ] #.重新启动broker
[root@hadoop kafka]# kafka-server-start.sh config/server0.properties &
尝试删除一个主题test02，查看相对应的znode变化
#.删除主题
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# kafka-topics.sh --zookeeper localhost: --delete --topic test02
Topic test02 is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true. #.查看主题：没有结果（应该有结果？）
[root@hadoop kafka]# kafka-topics.sh --describe --zookeeper localhost: --topic test02 #.查看log目录下test02-0目录是否存在：不存在了
[root@hadoop kafka]# ll /tmp/kafka-logs0 #test02目录没有了
总用量
-rw-r--r-- root root 8月 : cleaner-offset-checkpoint
drwxr-xr-x root root 8月 : __consumer_offsets-
...
drwxr-xr-x root root 8月 : __consumer_offsets-
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 8月 : meta.properties
drwxr-xr-x root root 8月 : mytest-
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint #.查看/admin/delete_topics：为空。按理说这里应该显示删除的主题test02
[zk: hadoop:(CONNECTED) 32] ls /admin/delete_topics
[] #.查看/brokers/topics：这里显示test02确实被删除了
[zk: hadoop:(CONNECTED) 33] ls /brokers/topics
[mytest, __consumer_offsets] #.查看/config/topics：
[zk: hadoop:(CONNECTED) 34] ls /config/topics
[mytest, __consumer_offsets] #以上均是我的实际操作结果，与老师演示的有一些出入，暂时无法解释原因
kafka3 本地目录结构以及在在zk上的znode的更多相关文章
- svn本地目录结构for window
演示内容: 使用svn目录结构来进行备份正式版和修复版本,最终合并修复版本.主干上的版本. 使用工具: visualSVN server 服务器软件工具 TortoiseSVN客户端工具 1.服务器的 ...
- Twitter Storm源代码分析之Nimbus/Supervisor本地目录结构
storm集群里面工作机器分为两种一种是nimbus, 一种是supervisor, 他们通过zookeeper来进行交互,nimbus通过zookeeper来发布一些指令,supervisor去读z ...
- 浅谈android中的目录结构
之前在android游戏开发中就遇到本地数据存储的问题:一般情形之下就将动态数据写入SD中存储,在没有SD卡的手机上就需另作处理了;再有在开发android应用的过程中,总要去调试APP,安装时又想去 ...
- Linux文件系统的目录结构
Linux下的文件系统为树形结构,入口为/ 树形结构下的文件目录: 无论哪个版本的Linux系统,都有这些目录,这些目录应该是标准的.各个Linux发行版本会存在一些小小的差异,但总体来说,还是大体差 ...
- Linux 文件系统的目录结构
http://www.jb51.net/LINUXjishu/151820.htmlLinux下的文件系统为树形结构,入口为/ 树形结构下的文件目录: 无论哪个版本的Linux系统,都有这些目录,这些 ...
- chromiun 学习《二》 目录结构 +启动流程
1.chromium的目录结构. 2.先上分析图一张.主要是从BrowserMain进程进行分析的.
- thinkphp学习笔记1—目录结构和命名规则
原文:thinkphp学习笔记1-目录结构和命名规则 最近开始学习thinkphp,在下不才,很多的问题看不明白所以想拿出来,恕我大胆发在首页上,希望看到的人能为我答疑解惑,这样大家有个互动,学起来快 ...
- CentOS版本区别及 Linux目录结构及其详解
CentOS 7.0体验与之前版本的不同http://www.linuxidc.com/Linux/2014-07/104196.htm CentOS版本选择http://www.centoscn.c ...
- mybatis学习笔记（六）使用generator生成mybatis基础配置代码和目录结构
原文:http://blog.csdn.net/oh_mourinho/article/details/51463413 创建maven项目 <span style="font-siz ...
随机推荐
- [Linux] 设置系统时区
1. 检查当前时区 以 root 身份登录. # date Fri Sep :: UTC 其中 UTC 是指当前使用的时间系统为世界标准时间,也称世界协调时间.英文名称为 Coordinated Un ...
- 【IBM-WALA】Step by Step : use WALA to generate System Dependency Graph PDF and Dot File (Mac)
Preparations: 1. IDE : eclipse (my version is luna) 2. maven (my version is 3.5.0) 3. git 4. JAVA 1. ...
- c++11 auto 与 decltype 详解
转自: here 一. auto简介 编程时候常常需要把表达式的值付给变量,需要在声明变量的时候清楚的知道变量是什么类型.然而做到这一点并非那么容易(特别是模板中),有时候根本做不到.为了解决这个问题 ...
- Qt编写GIF录屏工具（开源）
在平时的写作过程中,经常需要将一些操作动作和效果图截图成gif格式,使得涵盖的信息更全面更生动,有时候可以将整个操作过程和运行效果录制成MP4,但是文件体积比较大,而且很多网站不便于上传,基本上都支持 ...
- echarts - 使用echarts过程中遇到的问题(pending...)
1. 配合tab切换时,被display:none的元素init设置echarts失败 2018-11-09 18:09:35 现象描述:有一个tabs选项卡,每个切换项A.B中都有使用echart ...
- Log4j2 中format增加自定义的参数
首先收藏一下log4j2的官网:http://logging.apache.org/log4j/2.x/manual/plugins.html#TypeConverters 对于这种需要,可以使用lo ...
- 转载->CPU的内部架构和工作原理
CPU的内部架构和工作原理 本片博客转自:http://www.cnblogs.com/onepixel/p/8724526.html 感谢博主分享! 内部架构 CPU 的根本任务就是执行指令,对计 ...
- H - Painter
杂货店出售一种由N(3<=N<=12)种不同颜色的颜料,每种一瓶(50ML),组成的颜料套装.你现在需要使用这N种颜料:不但如此,你还需要一定数量的灰色颜料.杂货店从来不出售灰色颜料--也 ...
- hdu5157 Harry and magic string【manacher】
Harry and magic string Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 32768/32768 K (Java/O ...
- ROS中.launch文件的remap标签详解
https://www.cnblogs.com/LiuQiang921202/p/7679943.html