Install and Configure Apache Kafka

I. Installation

The installation environment must have JDK, verify that you enter:

java -version

1. down

Install server-side versions based on the jar downloaded by maven, such as my "C:\Users\xiaobin\.m2\repository\org\apache\kafka\kafka_2.11\1.0.2"

kafka_2.11-1.0.2

2. unzip

$-.tgz

II. Configuration

server.properties

1. Necessary settings

1) basic

(1) listener

listeners=PLAINTEXT://your_IP:9092

(2) logs

$mkdir HOME/kafka_2.11-1.0.2/logs

2) cluster

Sets the natural number of the broker ID to non-zero.

Host1:

broker.id=1

Host2:

broker.id=2

Host3:

broker.id=3

2. Optional settings

1) Zookeeper

default

## Zookeeper ##
zookeeper.connect: The ZooKeeper address (can list multiple addresses comma-separated for the ZooKeeper cluster).

zookeeper.connection.timeout.ms: Time to wait before going down if, for some reason, the broker is not able to connect.

2) Socket Server Settings

default

## Socket Server Settings ##
socket.send.buffer.bytes: The send buffer used by the socket server.

socket.receive.buffer.bytes: The socket server receives a buffer for network requests.

socket.request.max.bytes: The maximum request size the server will allow. This prevents the server from running out of memory.

3) Log Flush Policy

default

## Log Flush Policy ##
log.flush.interval.messages: Threshold for message count that is once reached all messages are flushed to the disk.

log.flush.interval.ms: Periodic time interval after which all messages will be flushed into the disk.

4) Log Retention Policy

default

## Log Retention Policy ##

log.retention.hours: The minimum age of the segment file to be eligible for deletion due to age.

log.retention.bytes: A size-based retention policy for logs. Segments are pruned from the log unless the remaining segments drop below log.retention.bytes.

log.segment.bytes: Size of the segment after which a new segment will be created.

log.retention.check.interval.ms: Periodic time interval after which log segments are checked for deletion as per the retention policy. If both retention policies are set, then segments are deleted when either criterion is met.

III. Run

1. start

./bin/kafka-server-start.sh config/server.properties

2. stop

./bin/kafka-server-stop.sh

Reference:

1. How to Set Up Kafka

2. How to Setup Kafka Cluster