1,下载kafka 并进行解压 http://mirrors.cnnic.cn/apache/kafka/0.8.1.1/kafka_2.9.2-0.8.1.1.tgz

2,启动Zookeeper  bin/zookeeper-server-start.sh config/zookeeper.properties &

3,启动kafka   bin/kafka-server-start.sh config/server.properties 

4,创建 topic  bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testweixuan &

5,  查看创建的topic  bin/kafka-topics.sh --list --zookeeper localhost:2181

查看topic的详细信息  bin/kafka-topics.sh --describe --zookeeper localhost:2181

解决报错:

  1. kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.

需要改动config文件夹下的server.properties中的以下两个属性

zookeeper.connect=localhost:2181改成zookeeper.connect=192.168.1.116 (自己的服务器IP地址):2181

以及默认注释掉的 #host.name=localhost 改成 host.name=192.168.1.116 (自己的服务器IP地址)

java api 测试kafka

消费端:

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties; import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
import kafka.serializer.StringDecoder;
import kafka.utils.VerifiableProperties; public class KafkaConsumer { private final ConsumerConnector consumer;
public final static String TOPIC = "kafkaToptic2";
private KafkaConsumer() {
Properties props = new Properties();
//zookeeper 配置
props.put("zookeeper.connect", "ip:2181");
//group 代表一个消费组
props.put("group.id", "jd-group");
//zk连接超时
props.put("zookeeper.session.timeout.ms", "4000");
props.put("zookeeper.sync.time.ms", "200");
props.put("auto.commit.interval.ms", "1000");
props.put("auto.offset.reset", "smallest");
//序列化类
props.put("serializer.class", "kafka.serializer.StringEncoder");
ConsumerConfig config = new ConsumerConfig(props);
consumer = kafka.consumer.Consumer.createJavaConsumerConnector(config);
} void consume() {
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put(TOPIC, new Integer(1)); StringDecoder keyDecoder = new StringDecoder(new VerifiableProperties());
StringDecoder valueDecoder = new StringDecoder(new VerifiableProperties());
Map<String, List<KafkaStream<String, String>>> consumerMap =
consumer.createMessageStreams(topicCountMap,keyDecoder,valueDecoder);
KafkaStream<String, String> stream = consumerMap.get(TOPIC).get(0);
ConsumerIterator<String, String> it = stream.iterator();
while (it.hasNext())
System.out.println(it.next().message());
} public static void main(String[] args) {
new KafkaConsumer().consume();
}
}

生产端

import java.util.Properties;

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig; /**
* Hello world!
*
*/
public class KafkaProducer
{
private final Producer<String, String> producer;
public final static String TOPIC = "TEST-TOPIC"; private KafkaProducer(){
Properties props = new Properties();
//此处配置的是kafka的端口
props.put("metadata.broker.list", "ip:9092");
props.put("zookeeper.connect", "ip:2181");//声明zk
// props.put("metadata.broker.list", "localhsot:9092");
// props.put("zookeeper.connect", "localhsot:2181");//声明zk
//配置value的序列化类
props.put("serializer.class", "kafka.serializer.StringEncoder");
//配置key的序列化类
props.put("key.serializer.class", "kafka.serializer.StringEncoder"); //request.required.acks
//0, which means that the producer never waits for an acknowledgement from the broker (the same behavior as 0.7). This option provides the lowest latency but the weakest durability guarantees (some data will be lost when a server fails).
//1, which means that the producer gets an acknowledgement after the leader replica has received the data. This option provides better durability as the client waits until the server acknowledges the request as successful (only messages that were written to the now-dead leader but not yet replicated will be lost).
//-1, which means that the producer gets an acknowledgement after all in-sync replicas have received the data. This option provides the best durability, we guarantee that no messages will be lost as long as at least one in sync replica remains.
// props.put("request.required.acks","-1");
producer = new Producer<String, String>(new ProducerConfig(props));
} void produce() {
int messageNo = 1000;
final int COUNT = 10000;
while (messageNo < COUNT) {
String key = String.valueOf(messageNo);
String data = "hello kafka message " + key;
producer.send(new KeyedMessage<String, String>(TOPIC, key ,data));
System.out.println(data);
messageNo ++;
}
} public static void main( String[] args )
{
new KafkaProducer().produce();
}
}

需要修改的文件是 kafka下的config

    1. 修改配置文件config/server.properties
    2. broker.id=1 #(唯一)
    3. host.name=ip
    4. log.dirs=/usr/local/kafka_2.10-0.8.2.0/log
    5. zookeeper.connect=192.168.1.116 (自己的服务器IP地址):2181

kafka 安装以及测试的更多相关文章

  1. Kafka 安装和测试

    转载自:http://czj4451.iteye.com/blog/2041096 1. 简介 kafka (官网地址:http://kafka.apache.org)是一款分布式消息发布和订阅的系统 ...

  2. kafka安装与测试

    基于linux-Centos7.0环境先进行测试学习 Producer即生产者,向Kafka集群发送消息,在发送消息之前,会对消息进行分类,即Topic, Topic即主题,通过对消息指定主题可以将消 ...

  3. kafka安装和简单测试

    kafka安装和简单测试 # 安装zookeeper(apache-zookeeper-3.5.6-bin)https://archive.apache.org/dist/zookeeper/zook ...

  4. Kafka安装及部署

    安装及部署 一.环境配置 操作系统:Cent OS 7 Kafka版本:0.9.0.0 Kafka官网下载:请点击 JDK版本:1.7.0_51 SSH Secure Shell版本:XShell 5 ...

  5. Kafka安装与配置(windows)

    作者:灬花儿灬 出处:http://www.cnblogs.com/flower1990/ 本文版权归作者和博客园共有,欢迎转载,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文连接,否则 ...

  6. kafka系列一、kafka安装及部署、集群搭建

    一.环境准备 操作系统:Cent OS 7 Kafka版本:kafka_2.10 Kafka官网下载:请点击 JDK版本:1.8.0_171 zookeeper-3.4.10 二.kafka安装配置 ...

  7. kafka安装和部署

    阅读目录 一.环境配置 二.操作过程 Kafka介绍 安装及部署 回到顶部 一.环境配置 操作系统:Cent OS 7 Kafka版本:0.9.0.0 Kafka官网下载:请点击 JDK版本:1.7. ...

  8. kafka安装和使用

    kafka安装和启动 kafka的背景知识已经讲了很多了,让我们现在开始实践吧,假设你现在没有Kafka和ZooKeeper环境. Step 1: 下载代码 下载0.10.0.0版本并且解压它. &g ...

  9. 【Apache Kafka】二、Kafka安装及简单示例

    (一)Apache Kafka安装 1.安装环境与前提条件   安装环境:Ubuntu16.04   前提条件: ubuntu系统下安装好jdk 1.8以上版本,正确配置环境变量 ubuntu系统下安 ...

随机推荐

  1. 【Linux学习】Linux文件系统1--文件系统的目录结构

    Linux文件系统1--文件系统的目录结构 一.linux文件系统的类型 LINUX有四种基本文件系统类型:普通文件.目录文件.连接文件和特殊文件,可用file命令来识别. 1.普通文件:如文本文件. ...

  2. HDU - 3345 War Chess 广搜+优先队列

    War chess is hh's favorite game: In this game, there is an N * M battle map, and every player has hi ...

  3. Attributes.Add用途与用法

    Attributes.Add("javascript事件","javascript语句"); 如: this.TextBox1.Attributes.add(& ...

  4. ES6之Promise对象学习——8个例子学会Promise

    目录 Promise 立即执行 Promise 三种状态 Promise 不可逆性 链式调用 Promise.then()回调异步性 Promise中的异常 Promise.resolve() res ...

  5. Unity动画事件

    动画事件添加 var clip = new AnimationClip();//clip,动画剪辑  储存基于动画的关键帧.这里新建动画剪辑 clip.SetCurve ("", ...

  6. iOS开发 - 多线程实现方案之Pthread篇

    pthread基础 pthread是POSIX thread的简写,一套通用的多线程API,适用于Unix.Linux.Windows等系统,跨平台.可移植,使用难度大,C语言框架,线程生命周期由程序 ...

  7. SpringBoot | Thymeleaf | 局部更新

    建立一个实体类: public class Fruit { int id; String name; public Fruit() { } public Fruit(int id, String na ...

  8. 根据select中选定option触发不同事件

    做公司项目,要实现一个小需求,根据下拉框选定的数据,绑定不同的事件.我的思路是获取select下面的option值,然后给它们定义不同的点击事件,但是在使用中option不能添加类,同时没有click ...

  9. 错误之Illegal mix of collations for operation 'like'

    内容来自博客:https://www.cnblogs.com/install/p/4417527.html MySQL Illegal mix of collations for operation ...

  10. 18113 Secret Book of Kungfu 按位DFS

    http://acm.scau.edu.cn:8000/uoj/mainMenu.html 18113 Secret Book of Kungfu 该题有题解 时间限制:1000MS  内存限制:65 ...