kafka集群安装和使用

kafka
(1)kafka是一个分布式的消息缓存系统
(2)kafka集群中的服务器都叫做broker
(3)kafka有两类客户端,一个叫做producer(消息生产者),一类叫做consumer(消息消费者),客户端和broker服务器之间采用TCP协议连接
(4)kafka中的消息可以通过topic进行区分,而且每一个消息topic都会被分区,以分担消息服务器的负载
(5)每一个分区都可以有多个副本,以防止数据的丢失
(6)某一个分区中的数据如果需要更新,都必须通知该分区所有副本中的leader来更新
(7)消费者可以分组,比如有两个消费者组A和B,共同消费一个topic:order_info,A和B所消费的消息不会重复,如order_info中有100个消息,每个消息都有一个id,编号从1-99,那么如果A组消费从0-49,B组消费就从50-99,当然不一定都是连续的
(8)消费者在具体消费某个topic中的消息时,可以制定起始偏移量

集群安装
官网教程
http://kafka.apache.org/22/documentation.html#introduction
1.解压
2.修改server.properties
broker.id=1
zookeeper.connect=hadoop01:2182,hadoop02:2182,hadoop03:2182
3.将zookeeper集群启动
4.在每一台节点上启动broker
bin/kafka-server-start.sh config/server.properties
#5.自带zookeeper,用于单节点,一般集群不用
#bin/zookeeper-server-start.sh config/zookeeper.proterties

开始安装
1.下载解压

[linyouyi@hadoop01 software]$ wget https://mirrors.aliyun.com/apache/kafka/2.2.0/kafka_2.11-2.2.0.tgz
[linyouyi@hadoop01 software]$ tar -zxvf kafka_2.-2.2..tgz -C /hadoop/module/

2.修改配置文件

[linyouyi@hadoop01 software]$ cd /hadoop/module/
[linyouyi@hadoop01 module]$ ll
total
drwxrwxr-x  linyouyi linyouyi  Aug  : apache-storm-2.0.
drwxr-xr-x  linyouyi linyouyi  Aug   : hadoop-2.7.
drwxrwxr-x   linyouyi linyouyi  Aug  : hbase-2.0.
drwxr-xr-x   linyouyi linyouyi  Jul    jdk1..0_144
drwxr-xr-x   linyouyi linyouyi  Mar  : kafka_2.-2.2.
drwxr-xr-x  linyouyi linyouyi  Aug   : zookeeper-3.4.
[linyouyi@hadoop01 kafka_2.-2.2.]$ vim config/server.properties
broker.id=
log.dirs=/hadoop/kafka_2.-2.2./log/kafka-logs
zookeeper.connect=hadoop01:,hadoop02:,hadoop03:

3.拷贝到其他节点

[linyouyi@hadoop01 kafka_2.-2.2.]$ cd ../
[linyouyi@hadoop01 module]$ scp -r kafka_2.-2.2./ linyouyi@hadoop02:/hadoop/module/
[linyouyi@hadoop01 module]$ scp -r kafka_2.-2.2./ linyouyi@hadoop03:/hadoop/module/

4.修改另外两台的broker.id

[linyouyi@hadoop02 kafka_2.-2.2.]$ vim config/server.properties
broker.id=
[linyouyi@hadoop03 kafka_2.-2.2.]$ vim config/server.properties
broker.id=

5.启动kafka

[linyouyi@hadoop01 kafka_2.-2.2.]$ bin/kafka-server-start.sh -deamon config/server.properties
[linyouyi@hadoop01 kafka_2.-2.2.]$ jps
 Jps
 QuorumPeerMain
 Kafka
[linyouyi@hadoop02 kafka_2.-2.2.]$ bin/kafka-server-start.sh -deamon config/server.properties
[linyouyi@hadoop02 kafka_2.-2.2.]$ jps
 Jps
 Kafka
 QuorumPeerMain
[linyouyi@hadoop03 kafka_2.-2.2.]$ bin/kafka-server-start.sh -deamon config/server.properties
[linyouyi@hadoop03 kafka_2.-2.2.]$ jps
 QuorumPeerMain
 Jps
 Kafka

使用
1.创建话题
//hadoop01:9092,hadoop02:9092,hadoop03:9092都行

[linyouyi@hadoop03 kafka_2.-2.2.]$ bin/kafka-topics.sh --create --bootstrap-server hadoop01: --replication-factor  --partitions  --topic linyouyi
[-- ::,] INFO [ReplicaFetcherManager on broker ] Removed fetcher for partitions Set(linyouyi-) (kafka.server.ReplicaFetcherManager)
[-- ::,] INFO [Log partition=linyouyi-, dir=/tmp/kafka-logs] Loading producer state till offset  with message format version  (kafka.log.Log)
[-- ::,] INFO [Log partition=linyouyi-, dir=/tmp/kafka-logs] Completed load of log with  segments, log start offset  and log end offset  in  ms (kafka.log.Log)
[-- ::,] INFO Created log for partition linyouyi- in /tmp/kafka-logs with properties {compression.type -> producer, message.format.version -> 2.2-IV1, file.delete.delay.ms -> , max.message.bytes -> , min.compaction.lag.ms -> , message.timestamp.type -> CreateTime, message.downconversion.enable -> true, min.insync.replicas -> , segment.jitter.ms -> , preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> , unclean.leader.election.enable -> false, retention.bytes -> -, delete.retention.ms -> , cleanup.policy -> [delete], flush.ms -> , segment.ms -> , segment.bytes -> , retention.ms -> , message.timestamp.difference.max.ms -> , segment.index.bytes -> , flush.messages -> }. (kafka.log.LogManager)
[-- ::,] INFO [Partition linyouyi- broker=] No checkpointed highwatermark is found for partition linyouyi- (kafka.cluster.Partition)
[-- ::,] INFO Replica loaded for partition linyouyi- with initial high watermark  (kafka.cluster.Replica)
[-- ::,] INFO Replica loaded for partition linyouyi- with initial high watermark  (kafka.cluster.Replica)
[-- ::,] INFO Replica loaded for partition linyouyi- with initial high watermark  (kafka.cluster.Replica)
[-- ::,] INFO [Partition linyouyi- broker=] linyouyi- starts at Leader Epoch  from offset . Previous Leader Epoch was: - (kafka.cluster.Partition)

[linyouyi@hadoop03 kafka_2.-2.2.]$ bin/kafka-topics.sh --list --bootstrap-server localhost:
linyouyi
[linyouyi@hadoop03 kafka_2.-2.2.]$ bin/kafka-topics.sh --list --bootstrap-server hadoop01:
linyouyi
[linyouyi@hadoop03 kafka_2.-2.2.]$ bin/kafka-topics.sh --list --zookeeper hadoop01:
linyouyi
[linyouyi@hadoop03 kafka_2.-2.2.]$ bin/kafka-topics.sh --create --bootstrap-server hadoop01: --replication-factor  --partitions  --topic youyi
[linyouyi@hadoop03 kafka_2.-2.2.]$ bin/kafka-topics.sh --list --zookeeper hadoop01:
linyouyi
youyi

2.生产者往话题里面写消息

[linyouyi@hadoop03 kafka_2.-2.2.]$ bin/kafka-console-producer.sh --broker-list localhost: --topic linyouyi>This is a message
>This is another message

3.消费者立马消费消息

[linyouyi@hadoop02 kafka_2.-2.2.]$ bin/kafka-console-consumer.sh --bootstrap-server localhost: --topic linyouyi --from-beginning
This is a message
This is another message

4.接着在生产者继续写话题,消费者立马就消费了

5.查看topic linyouyi总体情况

[linyouyi@hadoop01 kafka_2.-2.2.]$ bin/kafka-topics.sh --describe --bootstrap-server localhost: --topic linyouyi
Topic:linyouyi    PartitionCount:    ReplicationFactor:    Configs:segment.bytes=
    Topic: linyouyi    Partition:     Leader:     Replicas: ,,    Isr: ,,

[linyouyi@hadoop01 kafka_2.-2.2.]$ bin/kafka-topics.sh --describe --bootstrap-server localhost: --topic youyi
Topic:youyi    PartitionCount:    ReplicationFactor:    Configs:segment.bytes=
    Topic: youyi    Partition:     Leader:     Replicas: ,,    Isr: ,,