Install and Configure Apache Kafka on Ubuntu 16.04
https://devops.profitbricks.com/tutorials/install-and-configure-apache-kafka-on-ubuntu-1604-1/
Table of Contents
- Introduction
- Features
- Requirements
- Getting Started
- Installing Java
- Install ZooKeeper
- Install and Start Kafka Server
- Testing Kafka Server
- Summary
Introduction
Apache Kafka is an open-source scalable and high-throughput messaging system developed by the Apache Software Foundation written in Scala. Apache Kafka is specially designed to allow a single cluster to serve as the central data backbone for a large environment. It has a much higher throughput compared to other message brokers systems like ActiveMQ and RabbitMQ. It is capable of handling large volumes of real-time data efficiently. You can deploy Kafka on single Apache server or in a distributed clustered environment.
Features
The general features of Kafka are as follows :
- Persist message on disk that provide constant time performance.
- High throughput with disk structures that supporting hundreds of thousands of messages per second.
- Distributed system scales easily with no downtime.
- Supports multi-subscribers and automatically balances the consumers during failure.
This tutorial shows how to install and configure Apache Kafka on a Ubuntu 16.04 server.
Requirements
- A Ubuntu 16.04 server.
- Non-root user account with sudo privilege set up on your server.
Getting Started
Let's start making sure that your Ubuntu 16.04 server is fully up to date.
You can update your server by running the following command:
sudo apt-get update -y
sudo apt-get upgrade -y
Installing Java
Before installing Kafka, you will need to install Java on your system. You can install Oracle JDK 8 using the Webupd8 team PPA repository.
To add the repository, run the following command:
sudo add-apt-repository -y ppa:webupd8team/java
You should see the following output:
gpg: keyring `/tmp/tmpkjrm4mnm/secring.gpg' created
gpg: keyring `/tmp/tmpkjrm4mnm/pubring.gpg' created
gpg: requesting key EEA14886 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpkjrm4mnm/trustdb.gpg: trustdb created
gpg: key EEA14886: public key "Launchpad VLC" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
Next, update the metadata of the new repository by running the following command:
sudo apt-get update
Once you have finished, run the following command to install JDK 8:
sudo apt-get install oracle-java8-installer -y
You can also verify that JDK 8 is installed properly by running the following command:
sudo java -version
You should see the output something like this:
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
Install ZooKeeper
Before installing Apache Kafka, you will need to have zookeeper available and running. ZooKeeper is an open source service for maintaining configuration information, providing distributed synchronization, naming and providing group services.
By default ZooKeeper package is available in Ubuntu's default repository, you can install it by running the following command:
sudo apt-get install zookeeperd
Once installation is finished, it will be started as a daemon automatically. By default ZooKeeper will run on port 2181.
You can test it by running the following command:
netstat -ant | grep :2181
If everything's fine, you should see the following Output:
tcp6 0 0 :::2181 :::* LISTEN
Install and Start Kafka Server
Now that Java and ZooKeeper are installed, it is time to download and extract Kafka from Apache website. You can use wget to download Kafka:
wget http://mirror.fibergrid.in/apache/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz
Next, create a directory for Kafka installation:
sudo mkdir /opt/Kafka
cd /opt/Kafka
Extract the downloaded archive using tar command in /opt/Kafka:
sudo tar -xvf kafka_2.10-0.10.0.1.tgz -C /opt/Kafka/
The next step is to start Kafka server, you can start it by running kafka-server-start.sh script located at /opt/Kafka/kafka_2.10-0.10.0.1/bin/ directory.
sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-server-start.sh /opt/Kafka/kafka_2.10-0.10.0.1/config/server.properties
You should see the following output, if the server has started successfully:
[2016-08-22 21:43:48,279] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2016-08-22 21:43:48,516] INFO Kafka version : 0.10.0.1 (org.apache.kafka.common.utils.AppInfoParser)
[2016-08-22 21:43:48,525] INFO Kafka commitId : a7a17cdec9eaa6c5 (org.apache.kafka.common.utils.AppInfoParser)
[2016-08-22 21:43:48,527] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-08-22 21:43:48,555] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
You can use nohup with script to start the Kafka server as a background process:
sudo nohup /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-server-start.sh /opt/Kafka/kafka_2.10-0.10.0.1/config/server.properties /tmp/kafka.log 2>&1 &
You now have a Kafka server running and listening on port 9092.
Testing Kafka Server
Now, it is time to verify the Kafka server is operating correctly.
To test Kafka, create a sample topic with name "testing" in Apache Kafka using the following command:
sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testing
You should see the following output:
Created topic "testing".
Now, ask Zookeeper to list available topics on Apache Kafka by running the following command:
sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-topics.sh --list --zookeeper localhost:2181
You should see the following output:
testing
Now, publish a sample messages to Apache Kafka topic called testing by using the following producer command:
sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testing
After running above command, enter some messages like "Hi how are you?" press enter, then enter another message like "Where are you?"
Now, use consumer command to check for messages on Apache Kafka Topic called testing by running the following command:
sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic testing --from-beginning
You should see the following output:
Hi how are you?
Where are you?
With this above testing you have successfully verified that you have a valid Apache Kafka setup with Apache Zookeeper.
Summary
At this point, we have installed, configured, and tested Kafka on a Ubuntu 16.04 server. You can adapt the setup to make use of it in your production environment. To learn more about Kafka check out the Kafka documentation.
Install and Configure Apache Kafka on Ubuntu 16.04的更多相关文章
- Ubuntu 16.04安装ROS Kinetic详细教程 | Tutorial to Install and Configure ROS Kinetic on Ubuntu 16.04
本文首发于个人博客https://kezunlin.me/post/e2780b93/,欢迎阅读! Tutorial to Install and Configure ROS Kinetic on U ...
- How To Install Apache Kafka on Ubuntu 14.04
打算学习kafka ,接触一些新的知识.加油!!! 参考:https://www.digitalocean.com/community/tutorials/how-to-install-apache- ...
- Installing Apache Spark on Ubuntu 16.04
Santosh Srinivas on 07 Nov 2016, tagged onApache Spark, Analytics, Data Minin I've finally got to a ...
- Install .NET Core Runtime on Linux Ubuntu 16.04 x64
原文链接https://www.microsoft.com/net/download/linux-package-manager/ubuntu16-04/runtime-current nstall ...
- Install and Configure Apache Kafka
I. Installation The installation environment must have JDK, verify that you enter: java -version 1. ...
- ubuntu 16.04源码编译和配置caffe详细教程 | Install and Configure Caffe on ubuntu 16.04
本文首发于个人博客https://kezunlin.me/post/b90033a9/,欢迎阅读! Install and Configure Caffe on ubuntu 16.04 Series ...
- Install LAMP Stack On Ubuntu 16.04
原文:http://www.unixmen.com/how-to-install-lamp-stack-on-ubuntu-16-04/ LAMP is a combination of operat ...
- Ubuntu 16.04环境布署小记
本系列文章记录了升级Ubuntu 16.04的布署过程 回到目录 10. 安装Mono, Xsp 当前版本16.04.1的系统源的Mono版本为4.2.1,如需使用最新版本(本文书写时稳定版本为4.6 ...
- Installation Guide Ubuntu 16.04
Beside the installation guide on the main page, here is a guide to install GenieACS off a freshly in ...
随机推荐
- (NO.00003)iOS游戏简单的机器人投射游戏成形记(二)
打开Ai按需求依次绘制机器人身体,手臂和篮框: 因为是实验性质的游戏所以没必要在这上面花太多功夫,画出意思即可.虽然是2D游戏,但实际游戏中可以表现出伪3D的图形效果;尽管本猫这次画的游戏元素都是满满 ...
- 《java入门第一季》之Random类和获取随机数案例
/* * Random:产生随机数的类 * * 构造方法: * public Random():没有给种子,用的是默认种子,是当前时间的毫秒值下的随机数,所以会一直变化 * public ...
- ISLR系列:(3)重采样方法 Cross-Validation & Bootstrap
Resampling Methods 此博文是 An Introduction to Statistical Learning with Applications in R 的系列读书笔记,作为本人的 ...
- JavaScript进阶(九)JS实现本地文件上传至阿里云服务器
JS实现本地文件上传至阿里云服务器 前言 在前面的博客< JavaScript进阶(八)JS实现图片预览并导入服务器功能>(点击查看详情)中,实现了JS将本地图片文件预览并上传至阿里云服务 ...
- SparseArray到底哪点比HashMap好
SparseArray是android里为<Interger,Object>这样的Hashmap而专门写的class,目的是提高效率,其核心是折半查找函数(binarySearch). H ...
- R.java文件介绍
HelloWorld工程中的R.java文件 package com.android.hellworld; public final class R { public static final ...
- batch gradient descent(批量梯度下降) 和 stochastic gradient descent(随机梯度下降)
批量梯度下降是一种对参数的update进行累积,然后批量更新的一种方式.用于在已知整个训练集时的一种训练方式,但对于大规模数据并不合适. 随机梯度下降是一种对参数随着样本训练,一个一个的及时updat ...
- 【freeradius2.x】 安装和学习
虚拟机中centos 安装和学习 radius2 版本是2.2.x 的使用等知识 安装 为了测试方面,yum安装 yum -y install freeradius* 配置文件的位置是 /etc/ra ...
- cocos2d-x升级到3.4与创建android项目
cocos2d-x升级到3.4与创建android项目 1 升级安装cocos2d-x windows7 64位机器, 到官网下载cocos2d-x-3.4: http://www.cocos2d-x ...
- 【Matlab编程】matlab 画图
1. 不用截图工具就可以将图保存成图像格式,并且没有背景颜色:saveas(gcf ,'outputname','png/jpg'),第三项省略时默认为fig.m文件 2. 计算形如(-1)^2/ ...