kafka3 本地目录结构以及在在zk上的znode
一 kafka本地目录结构
[root@hadoop ~]# cd /tmp/kafka-logs1 [root@hadoop kafka-logs1]# find . |
搭建单节点多broker的kafka后，启动zk和kafka。
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# zookeeper-server-start.sh config/zookeeper.properties
...
[root@hadoop kafka]# kafka-server-start.sh config/server0.properties &
...
[root@hadoop kafka]# kafka-server-start.sh config/server1.properties &
...
[root@hadoop kafka]# kafka-server-start.sh config/server2.properties &
... [root@hadoop ~]# jps
QuorumPeerMain
Kafka
Kafka
Kafka
Jps
创建kafka集群时我已经创建了一个主题test02,现在我们再创建一个主题mytest（2个分区）
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# kafka-topics.sh --create --zookeeper localhost: --replication-factor --partitions --topic mytest
Created topic "mytest".
查看日志目录可以发现，3个日志目录几乎是一致的（__consumer_offsets-0是个什么鬼？）
test02和mytest副本数均是3，所以3个broker的log下面均有；
test02只有一个分区，mytest有2个分区，所以有一个test02目录，和2个mytest目录。
[root@hadoop ~]# ll /tmp/kafka-logs0
总用量
-rw-r--r-- root root 8月 : cleaner-offset-checkpoint
drwxr-xr-x root root 8月 : __consumer_offsets-
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 8月 : meta.properties
drwxr-xr-x root root 8月 : mytest- #命名格式：主题-分区
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint
drwxr-xr-x root root 8月 : test02- [root@hadoop ~]# ll /tmp/kafka-logs1
总用量
-rw-r--r-- root root 7月 : cleaner-offset-checkpoint
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 7月 : meta.properties
drwxr-xr-x root root 8月 : mytest-
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint
drwxr-xr-x root root 8月 : test02- [root@hadoop ~]# ll /tmp/kafka-logs2
总用量
-rw-r--r-- root root 7月 : cleaner-offset-checkpoint
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 7月 : meta.properties
drwxr-xr-x root root 8月 : mytest-
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint
drwxr-xr-x root root 8月 : test02-
查看主题目录
[root@hadoop ~]# ll /tmp/kafka-logs0/test02-/
总用量
-rw-r--r-- root root 8月 : .index
-rw-r--r-- root root 8月 : .log
-rw-r--r-- root root 8月 : .timeindex
-rw-r--r-- root root 8月 : .snapshot
-rw-r--r-- root root 8月 : leader-epoch-checkpoint
查看元数据信息
[root@hadoop ~]# cat /tmp/kafka-logs0/meta.properties
version=
broker.id=
[root@hadoop ~]# cat /tmp/kafka-logs1/meta.properties
version=
broker.id=
[root@hadoop ~]# cat /tmp/kafka-logs2/meta.properties
version=
broker.id=
向 mytest主题 生产消息并消费消息，可以看出消息分别保存在了不同分区的log里
#生产消息
[root@hadoop kafka]# kafka-console-producer.sh --broker-list localhost:,localhost:,localhost: --topic mytest
>hello kafka
>hello world #消费消息
[root@hadoop kafka]# kafka-console-consumer.sh --bootstrap-server localhost:,localhost:,localhost: --topic mytest --from-beginning
hello kafka
hello world #可以看出，消息分别保存在了不同分区的log里
[root@hadoop ~]# cat /tmp/kafka-logs0/mytest-/.log
C}_Me Ye Yÿÿÿÿÿÿÿÿÿÿÿÿÿÿ"hello kafka[root@hadoop ~]#
[root@hadoop ~]# cat /tmp/kafka-logs0/mytest-/.log
e ꛁe ꞿÿÿÿÿÿÿÿÿÿÿÿÿÿ"hello world[root@hadoop ~]#
二 kafka在zk上的znode
|
1./controller //data = {"version":1,"brokerid":0,"timestamp":"1533396512695"} 2./controller_epoch //data=27#不清楚什么意思，貌似第一次启动时是1，难道是kafka启动过的次数？ 3./brokers/ids //实时维护active的broker /brokers/topics /brokers/seqid 4./admin/delete_topics 5./isr_change_notification 6./consumers 7./config 注意：productor不在zk注册 |
启动zk客户端
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# zkCli.sh -server hadoop: #启动zk客户端
...
查看根目录
[zk: hadoop:(CONNECTED) ] ls / #查看znode
[cluster, controller_epoch, controller, brokers, zookeeper, admin, isr_change_notification, consumers,
log_dir_event_notification, latest_producer_id_block, config]
查看/controller
[zk: hadoop:(CONNECTED) ] ls /controller
[]
[zk: hadoop:(CONNECTED) ] get /controller
#这里的brokerid为0意思是kafka集群的leader为0
#如果集群中有多个broker，将leader杀死后会发现这里的brokerid变化。
{"version":,"brokerid":,"timestamp":""}
cZxid = 0x513
ctime = Sat Aug :: CST
mZxid = 0x513
mtime = Sat Aug :: CST
pZxid = 0x513
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x10000711d710001
dataLength =
numChildren =
查看/controller_epoch
[zk: hadoop:(CONNECTED) ] ls /controller_epoch
[]
[zk: hadoop:(CONNECTED) ] get /controller_epoch cZxid = 0x1c
ctime = Tue Jul :: CST
mZxid = 0x514
mtime = Sat Aug :: CST
pZxid = 0x1c
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/brokers
[zk: hadoop:(CONNECTED) ] get /brokers
null
cZxid = 0x4
ctime = Tue Jul :: CST
mZxid = 0x4
mtime = Tue Jul :: CST
pZxid = 0xd
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren = #3个孩子分别是ids, topics, seqid [zk: hadoop:(CONNECTED) ] ls /brokers
[ids, topics, seqid]
查看/brokers/ids
[zk: hadoop:(CONNECTED) ] ls /brokers/ids
#显示kafka集群中的所有active的brokerid
#如果杀死broker ，这里将会显示[, ]
[, , ] [zk: hadoop:(CONNECTED) 8] get /brokers/ids/
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://hadoop:9092"],
"jmx_port":-,"host":"hadoop","timestamp":"","port":,"version":}
cZxid = 0x55f
ctime = Sun Aug :: CST
mZxid = 0x55f
mtime = Sun Aug :: CST
pZxid = 0x55f
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x10000711d710005
dataLength =
numChildren =
查看/brokers/topics
[zk: hadoop:(CONNECTED) 9] ls /brokers/topics
[mytest, test02, __consumer_offsets] [zk: hadoop:(CONNECTED) ] ls /brokers/topics/mytest
[partitions]
[zk: hadoop:(CONNECTED) ] ls /brokers/topics/mytest/partitions #显示分区个数：mytest主题有2个分区，分别为0和1
[, ]
[zk: hadoop:(CONNECTED) 12] ls /brokers/topics/mytest/partitions/
[state]
[zk: hadoop:(CONNECTED) 13] ls /brokers/topics/mytest/partitions//state
[] #以下可以看出每个分区拥有不同的leader
[zk: hadoop:(CONNECTED) 14] get /brokers/topics/mytest/partitions//state
{"controller_epoch":,"leader":,"version":,"leader_epoch":,"isr":[,,]}
cZxid = 0x5a2
ctime = Sun Aug :: CST
mZxid = 0x5a2
mtime = Sun Aug :: CST
pZxid = 0x5a2
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
[zk: hadoop:(CONNECTED) ] get /brokers/topics/mytest/partitions//state
{"controller_epoch":,"leader":,"version":,"leader_epoch":,"isr":[,,]}
cZxid = 0x5a1
ctime = Sun Aug :: CST
mZxid = 0x5a1
mtime = Sun Aug :: CST
pZxid = 0x5a1
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/brokers/seqid
[zk: hadoop:(CONNECTED) 16] ls /brokers/seqid
[]
[zk: hadoop:(CONNECTED) ] get /brokers/seqid
null
cZxid = 0xd
ctime = Tue Jul :: CST
mZxid = 0xd
mtime = Tue Jul :: CST
pZxid = 0xd
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/admin/delete_topics
[zk: hadoop:(CONNECTED) ] ls /admin
[delete_topics]
[zk: hadoop:(CONNECTED) ] ls /admin/delete_topics
[]
查看/isr_change_notification
[zk: hadoop:(CONNECTED) 20] ls /isr_change_notification
[]
[zk: hadoop:(CONNECTED) ] get /isr_change_notification
null
cZxid = 0xe
ctime = Tue Jul :: CST
mZxid = 0xe
mtime = Tue Jul :: CST
pZxid = 0x544
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/consumers
#不知为何，我这里一直显示为空，按理说启动消费者之后这里应该显示相应信息的
[zk: hadoop:(CONNECTED) ] ls /consumers
[]
[zk: hadoop:(CONNECTED) 23] get /consumers
null
cZxid = 0x2
ctime = Tue Jul :: CST
mZxid = 0x2
mtime = Tue Jul :: CST
pZxid = 0x2
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
查看/config
[zk: hadoop:(CONNECTED) 24] ls /config
[changes, clients, brokers, topics, users]
[zk: hadoop:(CONNECTED) 25] ls /config/changes
[]
[zk: hadoop:(CONNECTED) 26] ls /config/clients
[]
[zk: hadoop:(CONNECTED) ] ls /config/brokers
[]
[zk: hadoop:(CONNECTED) ] ls /config/topics #与ls /brokers/topics结果一致
[mytest, test02, __consumer_offsets]
[zk: hadoop:(CONNECTED) ] ls /config/users
[]
尝试杀死集群的leader后，查看相对应的znode变化
#.杀死leader
[root@hadoop ~]# ps -ef|grep server0.properties #找到server0的进程号为4791
root 8月04 pts/ :: ...信息太多，忽略.../server0.properties
root : pts/ :: grep --color=auto server0.properties
[root@hadoop ~]# kill - #杀死进程
[root@hadoop ~]# ps -ef|grep server0.properties #再次查看
root : pts/ :: grep --color=auto server0.properties #.查看/controller：brokerid由0变为1
[zk: hadoop:(CONNECTED) 30] get /controller
{"version":,"brokerid":,"timestamp":""}
cZxid = 0x54c
ctime = Sun Aug :: CST
mZxid = 0x54c
mtime = Sun Aug :: CST
pZxid = 0x54c
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x10000711d710002
dataLength =
numChildren = #.查看/brokers/ids：active的broker只剩下1和2了
[zk: hadoop:(CONNECTED) ] ls /brokers/ids
[, ] #.重新启动broker
[root@hadoop kafka]# kafka-server-start.sh config/server0.properties &
尝试删除一个主题test02，查看相对应的znode变化
#.删除主题
[root@hadoop ~]# cd /usr/local/kafka
[root@hadoop kafka]# kafka-topics.sh --zookeeper localhost: --delete --topic test02
Topic test02 is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true. #.查看主题：没有结果（应该有结果？）
[root@hadoop kafka]# kafka-topics.sh --describe --zookeeper localhost: --topic test02 #.查看log目录下test02-0目录是否存在：不存在了
[root@hadoop kafka]# ll /tmp/kafka-logs0 #test02目录没有了
总用量
-rw-r--r-- root root 8月 : cleaner-offset-checkpoint
drwxr-xr-x root root 8月 : __consumer_offsets-
...
drwxr-xr-x root root 8月 : __consumer_offsets-
-rw-r--r-- root root 8月 : log-start-offset-checkpoint
-rw-r--r-- root root 8月 : meta.properties
drwxr-xr-x root root 8月 : mytest-
drwxr-xr-x root root 8月 : mytest-
-rw-r--r-- root root 8月 : recovery-point-offset-checkpoint
-rw-r--r-- root root 8月 : replication-offset-checkpoint #.查看/admin/delete_topics：为空。按理说这里应该显示删除的主题test02
[zk: hadoop:(CONNECTED) 32] ls /admin/delete_topics
[] #.查看/brokers/topics：这里显示test02确实被删除了
[zk: hadoop:(CONNECTED) 33] ls /brokers/topics
[mytest, __consumer_offsets] #.查看/config/topics：
[zk: hadoop:(CONNECTED) 34] ls /config/topics
[mytest, __consumer_offsets] #以上均是我的实际操作结果，与老师演示的有一些出入，暂时无法解释原因
kafka3 本地目录结构以及在在zk上的znode的更多相关文章
- svn本地目录结构for window
演示内容: 使用svn目录结构来进行备份正式版和修复版本,最终合并修复版本.主干上的版本. 使用工具: visualSVN server 服务器软件工具 TortoiseSVN客户端工具 1.服务器的 ...
- Twitter Storm源代码分析之Nimbus/Supervisor本地目录结构
storm集群里面工作机器分为两种一种是nimbus, 一种是supervisor, 他们通过zookeeper来进行交互,nimbus通过zookeeper来发布一些指令,supervisor去读z ...
- 浅谈android中的目录结构
之前在android游戏开发中就遇到本地数据存储的问题:一般情形之下就将动态数据写入SD中存储,在没有SD卡的手机上就需另作处理了;再有在开发android应用的过程中,总要去调试APP,安装时又想去 ...
- Linux文件系统的目录结构
Linux下的文件系统为树形结构,入口为/ 树形结构下的文件目录: 无论哪个版本的Linux系统,都有这些目录,这些目录应该是标准的.各个Linux发行版本会存在一些小小的差异,但总体来说,还是大体差 ...
- Linux 文件系统的目录结构
http://www.jb51.net/LINUXjishu/151820.htmlLinux下的文件系统为树形结构,入口为/ 树形结构下的文件目录: 无论哪个版本的Linux系统,都有这些目录,这些 ...
- chromiun 学习《二》 目录结构 +启动流程
1.chromium的目录结构. 2.先上分析图一张.主要是从BrowserMain进程进行分析的.
- thinkphp学习笔记1—目录结构和命名规则
原文:thinkphp学习笔记1-目录结构和命名规则 最近开始学习thinkphp,在下不才,很多的问题看不明白所以想拿出来,恕我大胆发在首页上,希望看到的人能为我答疑解惑,这样大家有个互动,学起来快 ...
- CentOS版本区别及 Linux目录结构及其详解
CentOS 7.0体验与之前版本的不同http://www.linuxidc.com/Linux/2014-07/104196.htm CentOS版本选择http://www.centoscn.c ...
- mybatis学习笔记（六）使用generator生成mybatis基础配置代码和目录结构
原文:http://blog.csdn.net/oh_mourinho/article/details/51463413 创建maven项目 <span style="font-siz ...
随机推荐
- docker中，如何将镜像保存为tar文件或者将镜像保存为文件，将tar文件导入到docker中
需求说明: 在实际使用docker的过程中,比如某些机器不能上网,恰巧需要某些docker镜像,一般可以采用在能上网的机器上,pull下来基础镜像,然后将docker 镜像保存为文件,在不能上网的机器 ...
- 自己开发chrome插件生成二维码
摘要: 最近在开发微信项目时,需要在微信调试,所以经常会在微信中输入本地服务地址,输入起来特别麻烦,所以自己就想了想微信中的扫一扫,然后开发了这款chrome插件,将当前url生成二维码,用微信扫一扫 ...
- [OpenCV] Install OpenCV 3.3 with DNN
OpenCV 3.3 Aug 3, 2017 OpenCV 3.3 has been released with greatly improved Deep Learning module and l ...
- 4、一、Introduction(入门)：3、System Permissions(系统权限)
3.System Permissions(系统权限) Android is a privilege-separated operating system, in which each applic ...
- 07装饰模式Decorator
一.什么是装饰模式 装饰( Decorator )模式又叫做包装模式.通 过一种对客户端透明的方式来扩展对象的功能, 是继承关系的一个替换方案. 二.装饰模式的结构 三.装饰模式的角色和职责 抽象组件 ...
- HttpWatch的Result中出现Aborted的原因分析
我们在使用HttpWatch进行Web调试的过程中有时候会看到非HTTP Status Code(状态码)的值,例如:(Aborted). (Aborted)是HttpWatch中定义的三种非HTTP ...
- IIS 6的日志time-taken字段没有值的解决方案
1.IIS 6网站的活动日志格式选择“W3C 扩展日志文件格式”. 2.日志记录属性的高级选项卡,扩展属性列表勾选time-taken.
- TIScript 代码Demo
var filelist = null; function alert(msg) { view.msgbox(null,msg); } self.on("click", " ...
- Android Studio 工具窗口浮动与布局恢复【申明：来源于网络】
Android Studio 工具窗口浮动与布局恢复[申明:来源于网络] http://bbs.chinaunix.net/thread-4182438-1-1.html
- 启动weblogic报错：string value '2.4' is not a valid enumeration value for web-app-versionType in namespace http://java.sun.com/xml/ns/javaee
启动报错: 原因:有人改动了web.xml的头 解决方法: 在web.xml中修改抬头为: <?xml version="1.0" encoding="UTF-8& ...