Problem of Creating Topics in Kafka with Kerberos
Hi,
After enabled Kerberos using Ambari, I got problem creating topics in Kafka using the kafka-topics.sh script. The topic was created, but its status is wrong without leader. It seems the topic is created with PLAINTEXT, while there is only PLAINTEXTSASL broker in the cluster after enabled Kerberos. The only configuration change I made is to chagne broker listener from 'PLAINTEXT://localhost:6667' to 'PLAINTEXTSASL://localhost:6667'. As posted in this question, I also changed the kafka-topics.sh to make it work with Kerberos. I am using HDP2.5.3.
- $ ./kafka-topics.sh --zookeeper ip-10-0-0-149.ap-northeast-1.compute.internal --create --partitions 1 --replication-factor 1 --topic mytopic
- Created topic "mytopic".
- $ ./kafka-topics.sh --zookeeper ip-10-0-0-149.ap-northeast-1.compute.internal --describe --topic mytopic
- Topic:mytopic PartitionCount:1 ReplicationFactor:1 Configs:
- Topic: mytopic Partition: 0 Leader: none Replicas: 1001 Isr:
个解答,截止mthiele · 2017年01月23日 02:24
When you use a script, command, or API to create a topic, an entry is created under ZooKeeper. The only user with access to ZooKeeper is the service account running Kafka (by default, kafka
). Therefore, the first step toward creating a Kafka topic on a secure cluster is to run kinit
, specifying the Kafka service keytab. The second step is to create the topic.
- Run
kinit
, specifying the Kafka service keytab. For example:kinit -k -t /etc/security/keytabs/kafka.service.keytab kafka/c6401.ambari.apache.org@EXAMPLE.COM
- Next, create the topic. Run the
kafka-topics.sh
command-line tool with the following options:/bin/kafka-topics.sh --zookeeper <hostname>:<port> --create --topic <topic-name> --partitions <number-of-partitions> --replication-factor <number-of-replicating-servers>
For example:
- /bin/kafka-topics.sh --zookeeper c6401.ambari.apache.org:2181 --create --topic test_topic --partitions 2 --replication-factor 2
- Created topic "test_topic".
个解答,截止yjiang · 2017年01月26日 10:23
@dbains, @mthiele, @Daniel Kozlowski,
Thank you. It works when created as kafka user.
To summarize all the steps -- from Ambari configs, to creating topic, granting permission and testing on kafka console producer/consumer scripts, I created this article Step by Step Recipe for Securing Kafka with Kerberos. Hope it saves others' time :-)
个解答,截止amankumbare · 2017年03月07日 16:59
@yjiang
If you try to create a topic as a non kafka user, it creates a topic but with no Leader and ISR. This is a known issue. According to me, the reason behind this could be the zookeeper acl's. Once topic is created in zookeeper, its acl's will not allow kafka to read details about it.
If you want to create a topic as a non kafka user you need to workaround by following below steps :
If you are not using Ranger :
1. Make sure "auto.create.topic.enable = true"
2. Give acl's for the user from which you want to create a topic, for ex :
# bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --producer --topic Test-topic
3. Do a kinit as a user from which you want to create topic.
4. Now try to produce messages to topic as that user :
# ./kafka-console-producer.sh --broker-list <hostname-broker>:6667 --topic Test-topic --security-protocol PLAINTEXTSASL
If you are using Ranger :
Instead of point 2 in above steps you will need to add a policy for the topic in ranger. Allow permissions for that user to produce, create, consumer. Restart kafka service. Then follow step 3 and 4 as mentioned above.
Hope this helps !!
个解答,截止dbains · 2017年01月23日 18:21
@yjiang Yes, please try creating the topic as kafka user because when we use kafka-topics.sh to create a test topic, what this script does is to create a node in zookeeper path - /broker/topics/test
then brokers thread gets notified that a new node is created and broker then creates actual data for topic test that is the metadata and physical data. But notice that Brokers are kafka/host@REALM, so if a user other than kafka creates a topic it gets permission for example world:anyone:r sasl:xyz:crdwa
So the new node that is created in zookeeper path will have these permissions. Now when the broker gets alerted and tries to create metadata and physical data for this new topic, it wont be able to because broker principal is kafka but topic's is xyz
个解答,截止Daniel Kozlowski · 2017年01月23日 08:47
To answer your question -
---------------
When you use a script, command, or API to create a topic, an entry is created under ZooKeeper. The only user with access to ZooKeeper is the service account running Kafka (by default, kafka
). Therefore, the first step toward creating a Kafka topic on a secure cluster is to runkinit
, specifying the Kafka service keytab. The second step is to create the topic.
----------------
See more in here: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_secure-kafka-ambari/content/ch_secure-kafka-create-topics.html
I hope this helps.
Problem of Creating Topics in Kafka with Kerberos的更多相关文章
- ERROR:"org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/topics/test" when creating or deleting Kafka operations authorized through the Ranger policies
PROBLEM DESCRIPTION When creating or deleting topics in Kafka, they cannot be authorized through the ...
- Step by Step Recipe for Securing Kafka with Kerberos
Short Description: Step by Step Recipe for Securing Kafka with Kerberos. Article I found it is a lit ...
- kafka Enabling Kerberos Authentication
CDK 2.0 and higher Powered By Apache Kafka supports Kerberos authentication, but it is supported onl ...
- flume集成kafka(kafka开启kerberos)配置
根据flume官网:当kafka涉及kerberos认证: 涉及两点配置,如下: 配置一:见下实例中红色部分 配置conf实例: [root@gz237-107 conf]# cat flume_sl ...
- 进行Spark,Kafka针对Kerberos相关配置
1. 提交任务的命令 spark-submit \--class <classname> \--master yarn \--deploy-mode client \--executor- ...
- Kafka集成Kerberos之后如何使用生产者消费者命令
1.生产者1.1.准备jaas.conf并添加到环境变量(使用以下方式的其中一种)1.1.1.使用Kinit方式前提是手动kinit 配置内容为: KafkaClient { com.sun.secu ...
- kafka实战kerberos
more /etc/krb5.conf [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log a ...
- kafka 配置kerberos校验以及开启acl实践
转载请注明原创地址:http://www.cnblogs.com/dongxiao-yang/p/7131626.html kafka从0.9版本以后引入了集群安全机制,由于最近需要新搭建一套kafk ...
- kafka Authentication using SASL/Kerberos
Authentication using SASL/Kerberos Prerequisites KerberosIf your organization is already using a Ker ...
随机推荐
- SqlServer中的系统数据库
SqlServer中的系统数据库有五个,平时写代码不太关注,今天一时兴起研究了一下. 1. master 记录SQL Server系统的所有系统级信息,例如:登陆账户信息.链接服务器和系统配置设置.记 ...
- 分布式理论 之 CAP 定理
-----------------------------------------------------入巷间吃汤面 笑看窗边飞雪. 目录: 什么是 CAP 定理 为什么只能 3 选 2 能不能解决 ...
- JavaScript中的 NaN 与 isNaN
NaN NaN 即 Not a Number ,不是一个数字. 在 JavaScript 中,整数和浮点数都统称为 Number 类型 .除此之外,Number 类型还有一个很特殊的值,即 NaN . ...
- Linux 桌面玩家指南:05. 发博客必备的图片处理和视频录制神器
特别说明:要在我的随笔后写评论的小伙伴们请注意了,我的博客开启了 MathJax 数学公式支持,MathJax 使用$标记数学公式的开始和结束.如果某条评论中出现了两个$,MathJax 会将两个$之 ...
- 暖春许愿季丨i春秋给你送福利
没有一点点防备 也没有一丝顾虑 就这样出现——暖春许愿季 纳尼?这不是我的歌声里 是i春秋在搞活动 这次准备搞个大的 多大呢 看这里 你许下心愿 我帮你实现 这是一棵神奇的心愿树 是一个畅所欲言之地 ...
- 在Linux(Centos7)上使用Docker运行.NetCore
在上一篇中我们写了如何在windows中使用docker运行.netcore,既然我们了解了windows下的运行发布,我们也可以试试linux下使用docker运行.netcore项目,那么今天我们 ...
- 【Android Studio安装部署系列】二、Android Studio开发环境搭建
版权声明:本文为HaiyuKing原创文章,转载请注明出处! 概述 Android Studio开发环境安装步骤 下载Android Studio 下载地址: http://www.wanandroi ...
- EscapeAndUnescapeUtil【java模拟js的escape和unescape函数】
版权声明:本文为HaiyuKing原创文章,转载请注明出处! 前言 在这里做一个记录,基本代码同参考资料<java模拟js的escape和unescape函数>一样. 效果图 代码 ...
- 『集群』004 Slithice 集群分布式(多个客户端,基于中央服务器的集群服务)
Slithice 集群分布式(多个客户端,基于中央服务器的多个集群服务端) 案例Demo展示: 集群架构图 如下: 如上图,上图 展示了 这个集群 的 结构: >一个中央服务器(可以有多个),负 ...
- 使用 ASP.NET Core MVC 创建 Web API(四)
使用 ASP.NET Core MVC 创建 Web API 使用 ASP.NET Core MVC 创建 Web API(一) 使用 ASP.NET Core MVC 创建 Web API(二) 使 ...