Install and Configure Apache Kafka on Ubuntu 16.04
https://devops.profitbricks.com/tutorials/install-and-configure-apache-kafka-on-ubuntu-1604-1/
Table of Contents
- Introduction
- Features
- Requirements
- Getting Started
- Installing Java
- Install ZooKeeper
- Install and Start Kafka Server
- Testing Kafka Server
- Summary
Introduction
Apache Kafka is an open-source scalable and high-throughput messaging system developed by the Apache Software Foundation written in Scala. Apache Kafka is specially designed to allow a single cluster to serve as the central data backbone for a large environment. It has a much higher throughput compared to other message brokers systems like ActiveMQ and RabbitMQ. It is capable of handling large volumes of real-time data efficiently. You can deploy Kafka on single Apache server or in a distributed clustered environment.
Features
The general features of Kafka are as follows :
- Persist message on disk that provide constant time performance.
- High throughput with disk structures that supporting hundreds of thousands of messages per second.
- Distributed system scales easily with no downtime.
- Supports multi-subscribers and automatically balances the consumers during failure.
This tutorial shows how to install and configure Apache Kafka on a Ubuntu 16.04 server.
Requirements
- A Ubuntu 16.04 server.
- Non-root user account with sudo privilege set up on your server.
Getting Started
Let's start making sure that your Ubuntu 16.04 server is fully up to date.
You can update your server by running the following command:
sudo apt-get update -y
sudo apt-get upgrade -y
Installing Java
Before installing Kafka, you will need to install Java on your system. You can install Oracle JDK 8 using the Webupd8 team PPA repository.
To add the repository, run the following command:
sudo add-apt-repository -y ppa:webupd8team/java
You should see the following output:
gpg: keyring `/tmp/tmpkjrm4mnm/secring.gpg' created
gpg: keyring `/tmp/tmpkjrm4mnm/pubring.gpg' created
gpg: requesting key EEA14886 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpkjrm4mnm/trustdb.gpg: trustdb created
gpg: key EEA14886: public key "Launchpad VLC" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
Next, update the metadata of the new repository by running the following command:
sudo apt-get update
Once you have finished, run the following command to install JDK 8:
sudo apt-get install oracle-java8-installer -y
You can also verify that JDK 8 is installed properly by running the following command:
sudo java -version
You should see the output something like this:
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
Install ZooKeeper
Before installing Apache Kafka, you will need to have zookeeper available and running. ZooKeeper is an open source service for maintaining configuration information, providing distributed synchronization, naming and providing group services.
By default ZooKeeper package is available in Ubuntu's default repository, you can install it by running the following command:
sudo apt-get install zookeeperd
Once installation is finished, it will be started as a daemon automatically. By default ZooKeeper will run on port 2181.
You can test it by running the following command:
netstat -ant | grep :2181
If everything's fine, you should see the following Output:
tcp6 0 0 :::2181 :::* LISTEN
Install and Start Kafka Server
Now that Java and ZooKeeper are installed, it is time to download and extract Kafka from Apache website. You can use wget to download Kafka:
wget http://mirror.fibergrid.in/apache/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz
Next, create a directory for Kafka installation:
sudo mkdir /opt/Kafka
cd /opt/Kafka
Extract the downloaded archive using tar command in /opt/Kafka:
sudo tar -xvf kafka_2.10-0.10.0.1.tgz -C /opt/Kafka/
The next step is to start Kafka server, you can start it by running kafka-server-start.sh script located at /opt/Kafka/kafka_2.10-0.10.0.1/bin/ directory.
sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-server-start.sh /opt/Kafka/kafka_2.10-0.10.0.1/config/server.properties
You should see the following output, if the server has started successfully:
[2016-08-22 21:43:48,279] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2016-08-22 21:43:48,516] INFO Kafka version : 0.10.0.1 (org.apache.kafka.common.utils.AppInfoParser)
[2016-08-22 21:43:48,525] INFO Kafka commitId : a7a17cdec9eaa6c5 (org.apache.kafka.common.utils.AppInfoParser)
[2016-08-22 21:43:48,527] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-08-22 21:43:48,555] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
You can use nohup with script to start the Kafka server as a background process:
sudo nohup /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-server-start.sh /opt/Kafka/kafka_2.10-0.10.0.1/config/server.properties /tmp/kafka.log 2>&1 &
You now have a Kafka server running and listening on port 9092.
Testing Kafka Server
Now, it is time to verify the Kafka server is operating correctly.
To test Kafka, create a sample topic with name "testing" in Apache Kafka using the following command:
sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testing
You should see the following output:
Created topic "testing".
Now, ask Zookeeper to list available topics on Apache Kafka by running the following command:
sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-topics.sh --list --zookeeper localhost:2181
You should see the following output:
testing
Now, publish a sample messages to Apache Kafka topic called testing by using the following producer command:
sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testing
After running above command, enter some messages like "Hi how are you?" press enter, then enter another message like "Where are you?"
Now, use consumer command to check for messages on Apache Kafka Topic called testing by running the following command:
sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic testing --from-beginning
You should see the following output:
Hi how are you?
Where are you?
With this above testing you have successfully verified that you have a valid Apache Kafka setup with Apache Zookeeper.
Summary
At this point, we have installed, configured, and tested Kafka on a Ubuntu 16.04 server. You can adapt the setup to make use of it in your production environment. To learn more about Kafka check out the Kafka documentation.
Install and Configure Apache Kafka on Ubuntu 16.04的更多相关文章
- Ubuntu 16.04安装ROS Kinetic详细教程 | Tutorial to Install and Configure ROS Kinetic on Ubuntu 16.04
本文首发于个人博客https://kezunlin.me/post/e2780b93/,欢迎阅读! Tutorial to Install and Configure ROS Kinetic on U ...
- How To Install Apache Kafka on Ubuntu 14.04
打算学习kafka ,接触一些新的知识.加油!!! 参考:https://www.digitalocean.com/community/tutorials/how-to-install-apache- ...
- Installing Apache Spark on Ubuntu 16.04
Santosh Srinivas on 07 Nov 2016, tagged onApache Spark, Analytics, Data Minin I've finally got to a ...
- Install .NET Core Runtime on Linux Ubuntu 16.04 x64
原文链接https://www.microsoft.com/net/download/linux-package-manager/ubuntu16-04/runtime-current nstall ...
- Install and Configure Apache Kafka
I. Installation The installation environment must have JDK, verify that you enter: java -version 1. ...
- ubuntu 16.04源码编译和配置caffe详细教程 | Install and Configure Caffe on ubuntu 16.04
本文首发于个人博客https://kezunlin.me/post/b90033a9/,欢迎阅读! Install and Configure Caffe on ubuntu 16.04 Series ...
- Install LAMP Stack On Ubuntu 16.04
原文:http://www.unixmen.com/how-to-install-lamp-stack-on-ubuntu-16-04/ LAMP is a combination of operat ...
- Ubuntu 16.04环境布署小记
本系列文章记录了升级Ubuntu 16.04的布署过程 回到目录 10. 安装Mono, Xsp 当前版本16.04.1的系统源的Mono版本为4.2.1,如需使用最新版本(本文书写时稳定版本为4.6 ...
- Installation Guide Ubuntu 16.04
Beside the installation guide on the main page, here is a guide to install GenieACS off a freshly in ...
随机推荐
- Device tree customization
Step 1: OEMs can create their own device tree by adding "qcom,msm-id/qcom,board-id" entry ...
- HTML移动开发参考
小强的HTML5移动开发之路 http://blog.csdn.net/dawanganban/article/details/17591373 其他: http://blog.csdn.net/gf ...
- python调用数据库并查询
http://blog.csdn.net/pipisorry/article/details/48024795 python调用数据库命令 conn = sqlite3.connect(". ...
- mysql进阶(四)mysql中select
mysql中select * for update 注: FOR UPDATE 仅适用于InnoDB,且必须在事务区块(BEGIN/COMMIT)中才能生效. 作用 锁定该语句所选择到的对象.防止在 ...
- my project 中git使用过程(基本操作流程)
1.g it clone git@name:server/BM/APPS.git 则BM_APPS.git项目被下载到当前目录下了,这时git@name:server/BM/APPS.git就是自己 ...
- JSP判断移动设备
日期: 2014年3月6日 人员::铁锚 看到了一篇很好的文章, <在天猫,前端做什么?>,里面有天猫php判断移动设备的正则(个人猜测),觉得很好用,于是就决定移植到JSP里面. jsp ...
- CUDA Cuts: Fast Graph Cuts on the GPU
原文出处: http://lincccc.blogspot.tw/2011/03/cuda-cuts-fast-graph-cuts-on-gpu_03.html 现在需要代理才能访问,所以就转载了. ...
- LeetCode之“散列表”:Two Sum && 3Sum && 3Sum Closest && 4Sum
1. Two Sum 题目链接 题目要求: Given an array of integers, find two numbers such that they add up to a specif ...
- VT控制码
VT100 是一个终端类型定义,VT100 控制码是用来在终端扩展显示的代码.比如果终端上任意坐标用 不同的颜色显示字符. 所有的控制符是 \033 打头(即 ESC 的 ASCII 码)用输出字符语 ...
- iOS9 ReplayKit录制视频
猴子原创,欢迎转载.转载请注明: 转载自Cocos2Der-CSDN,谢谢! 原文地址: http://blog.csdn.net/cocos2der/article/details/50260873 ...