Quick Start

Step 1: Download the code

Download the
0.8 release.

> tar xzf kafka-<VERSION>.tgz
> cd kafka-<VERSION>
> ./sbt update
> ./sbt package
> ./sbt assembly-package-dependency

This tutorial assumes you are starting on a fresh zookeeper instance with no pre-existing data. If you want to migrate from an existing 0.7 installation you
will need to follow the migration instructions.

Step 2: Start the server

Kafka uses zookeeper so you need to first start a zookeeper server if you don't already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node zookeeper instance.

> bin/zookeeper-server-start.sh config/zookeeper.properties
[2013-04-22 15:01:37,495] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
...

Now start the Kafka server:

> bin/kafka-server-start.sh config/server.properties
[2013-04-22 15:01:47,028] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2013-04-22 15:01:47,051] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
...

Step 3: Create a topic

Let's create a topic named "test" with a single partition and only one replica:

> bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 1 --partition 1 --topic test

We can now see that topic if we run the list topic command:

> bin/kafka-list-topic.sh --zookeeper localhost:2181

Alternatively, you can also configure your brokers to auto-create topics when a non-existent topic is published to.

Step 4: Send some messages

Kafka comes with a command line client that will take input from a file or standard in and send it out as messages to the Kafka cluster. By default each line
will be sent as a separate message.

Run the producer and then type a few messages to send to the server.

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message

Step 5: Start a consumer

Kafka also has a command line consumer that will dump out messages to standard out.

> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
This is a message
This is another message

If you have each of the above commands running in a different terminal then you should now be able to type messages into the producer terminal and see them appear in the consumer terminal.

All the command line tools have additional options; running the command with no arguments will display usage information documenting them in more detail.

Step 6: Setting up a multi-broker cluster

So far we have been running against a single broker, but that's no fun. For Kafka, a single broker is just a cluster of size one, so nothing much changes other
than starting a few more broker instances. But just to get feel for it, let's expand our cluster to three nodes (still all on our local machine).

First we make a config file for each of the brokers:

> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties

Now edit these new files and set the following properties:

config/server-1.properties:
broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1 config/server-2.properties:
broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2

The broker.id property
is the unique and permanent name of each node in the cluster. We have to override the port and log directory only because we are running these all on the same machine and we want to keep the brokers from trying to all register on the same port or overwrite
each others data.

We already have Zookeeper and our single node started, so we just need to start the two new nodes. However, this time we have to override the JMX port used by java too to avoid clashes with the running node:

> JMX_PORT=9997 bin/kafka-server-start.sh config/server-1.properties &
...
> JMX_PORT=9998 bin/kafka-server-start.sh config/server-2.properties &
...

Now create a new topic with a replication factor of three:

> bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 3 --partition 1 --topic my-replicated-topic

Okay but now that we have a cluster how can we know which broker is doing what? To see that run the "list topics" command:

> bin/kafka-list-topic.sh --zookeeper localhost:2181
topic: my-replicated-topic partition: 0 leader: 1 replicas: 1,2,0 isr: 1,2,0
topic: test partition: 0 leader: 0 replicas: 0 isr: 0

Here is an explanation of output:

  • "leader" is the node responsible for all reads and writes for the given partition. Each node would be the leader for a randomly selected portion of the partitions.
  • "replicas" is the list of nodes that are supposed to server the log for this partition regardless of whether they are currently alive.
  • "isr" is the set of "in-sync" replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.

Note that both topics we created have only a single partition (partition 0). The original topic has no replicas and so it is only present on the leader (node
0), the replicated topic is present on all three nodes with node 1 currently acting as leader and all replicas in sync.

As before let's publish a few messages message:

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
...
my test message 1
my test message 2
^C

Now consume this message:

> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic
...
my test message 1
my test message 2
^C

Now let's test out fault-tolerance. Kill the broker acting as leader for this topic's only partition:

> pkill -9 -f server-1.properties

Leadership should switch to one of the slaves:

> bin/kafka-list-topic.sh --zookeeper localhost:2181
...
topic: my-replicated-topic partition: 0 leader: 2 replicas: 1,2,0 isr: 2
topic: test partition: 0 leader: 0 replicas: 0 isr: 0

And the messages should still be available for consumption even though the leader that took the writes originally is down:

> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic
...
my test message 1
my test message 2
^C

转载自:http://kafka.apache.org/08/quickstart.html


kafka第四篇--快速入门(如何使用kafka)的更多相关文章

  1. Spring Boot 揭秘与实战(二) 数据缓存篇 - 快速入门

    文章目录 1. 声明式缓存 2. Spring Boot默认集成CacheManager 3. 默认的 ConcurrenMapCacheManager 4. 实战演练5. 扩展阅读 4.1. Mav ...

  2. net core体系-web应用程序-4asp.net core2.0 项目实战(CMS)-第二章 入门篇-快速入门ASP.NET Core看这篇就够了

    .NET Core实战项目之CMS 第二章 入门篇-快速入门ASP.NET Core看这篇就够了   原文链接:https://www.cnblogs.com/yilezhu/p/9985451.ht ...

  3. Kafka学习笔记2: 快速入门

    在开始Kafka环境搭建之前,首先要安装Linux系统,并在Linux系统上安装JDK1.8版本,关于linux虚拟机的安装和linux系统下jdk的安装可以参考我的博文: http://blog.c ...

  4. .NET Core实战项目之CMS 第二章 入门篇-快速入门ASP.NET Core看这篇就够了

    作者:依乐祝 原文链接:https://www.cnblogs.com/yilezhu/p/9985451.html 本来这篇只是想简单介绍下ASP.NET Core MVC项目的(毕竟要照顾到很多新 ...

  5. Spring Boot教程(十四)快速入门

    快速入门 本章主要目标完成Spring Boot基础项目的构建,并且实现一个简单的Http请求处理,通过这个例子对Spring Boot有一个初步的了解,并体验其结构简单.开发快速的特性. 系统要求: ...

  6. SpringBoot学习(四)-->SpringBoot快速入门,开山篇

    Spring Boot简介 Spring Boot的目的在于创建和启动新的基于Spring框架的项目.Spring Boot会选择最适合的Spring子项目和第三方开源库进行整合.大部分Spring ...

  7. CDH下集成spark2.2.0与kafka(四十一):在spark+kafka流处理程序中抛出错误java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V

    错误信息 19/01/15 19:36:40 WARN consumer.ConsumerConfig: The configuration max.poll.records = 1 was supp ...

  8. Realm for Android快速入门教程

    介绍 如果你关注安卓开发的最新趋势,你可能已经听说过Realm.Realm是一个可以替代SQLite以及ORMlibraries的轻量级数据库. 相比SQLite,Realm更快并且具有很多现代数据库 ...

  9. 【转】Robot Framework 快速入门

    目录 介绍 概述 安装 运行demo 介绍样例应用程序 测试用例 第一个测试用例 高级别测试用例 数据驱动测试用例 关键词keywords 内置关键词 库关键词 用户定义关键词 变量 定义变量 使用变 ...

随机推荐

  1. ios开发使用lipo命令合并真机库和模拟器库

    在开发ios时,我们经常会遇到编译两套库文件,使用模拟器时链接模拟器库,使用真机时使用真机库,这样操作会对后期的维护带来麻烦,所以Apple提供了一个把多个不同平台的.a库文件合并成一个适用于多平台的 ...

  2. Android OpenGL ES(四)关于EGL .

    OpenGL ES的javax.microedition.khronos.opengles 包定义了平台无关的GL绘图指令,EGL(javax.microedition.khronos.egl ) 则 ...

  3. Python中元素定位探讨

    以下以阿里云(10.10.1.11)系统中考试答题为例进行了python脚本的编写研究.模糊定位(定位一些动态值): ————————————————————————————————————————— ...

  4. C++的精髓——虚函数

    虚函数为了重载和多态的需要,在基类中是由定义的,即便定义是空,所以子类中可以重写也可以不写基类中的函数! 纯虚函数在基类中是没有定义的,必须在子类中加以实现,很像java中的接口函数! 虚函数 引入原 ...

  5. java 工作内存

    所谓线程的“工作内存”到底是个什么东西?有的人认为是线程的栈,其实这种理解是不正确的.看看JLS(java语言规范)对线程工作 内存的描述,线程的working memory只是cpu的寄存器和高速缓 ...

  6. UVALive 6887 Book Club

    最大流,有向环覆盖问题. #include<cstdio> #include<cstring> #include<string> #include<cmath ...

  7. filter by date in Sphinx

    Indexing: include the date column in the sql_query of source. Also set the date as sql_attr_unit PHP ...

  8. 去除移动端点击事件出现的背景框 tap-highlight-color

    -webkit-tap-highlight-color 这个属性只用于iOS (iPhone和iPad).当你点击一个链接或者通过Javascript定义的可点击元素的时候,它就会出现一个半透明的灰色 ...

  9. EtherChannel Cisco 端口聚合详解

    冗余连接及其实现 无论什么设备都无法保障运行的绝对稳定性,即使再优秀的产品也无法保证24×7不间断的工作.除去设备或模块损坏.传输线路中断等硬件故障原因以外,还可能由于网络流量过载.任务负荷过大而导致 ...

  10. SecureCRT 上传文件的两种方法 Zmodem、SFTP

    Zmodem: 无论有xshell还是secureCRT连接linux的时. 默认都用一个zmodem可以帮助window和linux之间传输文件 很方便和实用的工具. 不过默认是无法使用的 需要安装 ...