Kafka 1.0.0集群安装

环境

主机名	IP 地址	安装路径	系统
sht-sgmhadoopdn-01	172.16.101.58	/opt/kafka_2.12-1.0.0 /opt/kafka（软连接）	CentOS Linux release 7.3.1611 (Core)
sht-sgmhadoopdn-02	172.16.101.59
sht-sgmhadoopdn-03	172.16.101.60

软件下载

Kafka下载

http://archive.apache.org/dist/kafka/1.0.0/kafka_2.12-1.0.0.tgz

JDK下载

https://download.oracle.com/otn/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz

软件安装

一. 安装jdk

# wget https://download.oracle.com/otn/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz
# tar -zxf jdk-8u45-linux-x64.tar.gz -C /usr/

二.设置.bash_profile环境变量

JAVA_HOME=/usr/java/jdk1..0_45
KAFKA_HOME=/opt/kafka
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JRE_HOME/lib
PATH=$KAFKA_HOME/bin:$JAVA_HOME/bin:$JRE_HOME/bin:$PATH:$HOME/bin
export PATH

三. 安装kafka

# wget http://archive.apache.org/dist/kafka/1.0.0/kafka_2.12-1.0.0.tgz -P /opt
# tar -zxf kafka_2.-1.0..tgz
# ln -s kafka_2.-1.0. kafka
# mkdir /opt/kafka/data

软件配置

一. zookeeper配置

1. 文件zookeeper.properties，各节点一致

tickTime=
initLimit=
syncLimit=
dataDir=/opt/kafka/data
clientPort=
server.=sht-sgmhadoopdn-::
server.=sht-sgmhadoopdn-::
server.=sht-sgmhadoopdn-::

2. 各节点分别创建server-id

sht-sgmhadoopdn-01

# echo  > /opt/kafka/data/myid

sht-sgmhadoopdn-02

# echo  > /opt/kafka/data/myid

sht-sgmhadoopdn-03

# echo  > /opt/kafka/data/myid

3.启动zookeeper

# /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties

4.查看zookeeper集群状态

# echo stat | nc sht-sgmhadoopdn-  | grep Mode
Mode: follower
# echo stat | nc sht-sgmhadoopdn-  | grep Mode
Mode: leader
# echo stat | nc sht-sgmhadoopdn-  | grep Mode
Mode: follower

二. kafka配置

1.各节点配置文件server.properties

sht-sgmhadoopdn-01

broker.id=
listeners=PLAINTEXT://172.16.101.58:9092
advertised.listeners=PLAINTEXT://172.16.101.58:9092
log.dirs=/opt/kafka/data
zookeeper.connect=sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-:

sht-sgmhadoopdn-02

broker.id=
listeners=PLAINTEXT://172.16.101.59:9092
advertised.listeners=PLAINTEXT://172.16.101.59:9092
log.dirs=/opt/kafka/data
zookeeper.connect=sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-:

sht-sgmhadoopdn-03

broker.id=
listeners=PLAINTEXT://172.16.101.60:9092
advertised.listeners=PLAINTEXT://172.16.101.60:9092
log.dirs=/opt/kafka/data
zookeeper.connect=sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-:

2. 启动kafka

# /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties

3. 查看集群状态

# echo dump | nc sht-sgmhadoopdn-  | grep broker
    /brokers/ids/
    /brokers/ids/
    /brokers/ids/

三 测试

1. 创建topic

# kafka-topics.sh --zookeeper sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-: --create --partitions  --replication-factor  --topic test-topic
Created topic "test-topic".
# kafka-topics.sh --zookeeper sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-: --describe --topic test-topic
Topic:test-topic    PartitionCount:    ReplicationFactor:    Configs:
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,
    Topic: test-topic    Partition:     Leader:     Replicas: ,,    Isr: ,,

2. 生产者生产数据

# kafka-console-producer.sh --broker-list sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-: --topic test-topic
>Hello
>This is my fisrt Kafka !

3. 消费者消费数据

# kafka-console-consumer.sh --bootstrap-server bootstrap-servers=sht-sgmhadoopdn-:,sht-sgmhadoopdn-:,sht-sgmhadoopdn-: --topic test-topic --from-beginning
Hello
This is my fisrt Kafka !