Flume和Kafka整合安装
版本号：
1.flume安装
2.kafka安装
3.Flume和Kafka整合
- touch /usr/local/flume/apache-flume-1.6.0-bin/conf/flume-kafka.conf
- sudo gedit /usr/local/flume/apache-flume-1.6.0-bin/conf/flume-kafka.conf
- # 指定Agent的组件名称
- agent1.sources = source1
- agent1.sinks = sink1
- agent1.channels = channel1
- # 指定Flume source(要监听的路径)
- agent1.sources.source1.type = spooldir
- agent1.sources.source1.spoolDir = /usr/local/flume/logtest
- # 指定Flume sink
- #agent1.sinks.sink1.type = logger
- agent1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
- agent1.sinks.sink1.topic = test
- agent1.sinks.sink1.brokerList = 192.168.168.200:9092
- agent1.sinks.sink1.requiredAcks = 1
- agent1.sinks.sink1.batchSize = 100
- # 指定Flume channel
- agent1.channels.channel1.type = memory
- agent1.channels.channel1.capacity = 1000
- agent1.channels.channel1.transactionCapacity = 100
- # 绑定source和sink到channel上
- agent1.sources.source1.channels = channel1
- agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.topic = test 代表flume监听路径下发生变化时，会把消息发送到localhost机器上的test主题。
启动flume-kafka.conf：
- cd /usr/local/flume/apache-flume-1.6.0-bin
- bin/flume-ng agent --conf conf --conf-file conf/flume-kafka.conf --name agent1 -Dflume.root.logger=INFO,console
运行成功日志如下：
- 2017-07-07 22:22:02,270 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SINK, name: sink1: Successfully registered new MBean.
- 2017-07-07 22:22:02,270 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SINK, name: sink1 started
启动kafka的消费者，监听topic主题：
- kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
testKafka.log ：
- touch /usr/local/flume/testKafka.log
- sudo gedit /usr/local/flume/testKafka.log
- cp /usr/local/flume/testKafka.log /usr/local/flume/logtest
- [root@master kafka_2.11-0.9.0.0]# kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
- [2017-07-07 22:36:38,687] INFO [Group Metadata Manager on Broker 200]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
- Flume connect Kafka success!
---------------------------------flume------------------------------
- 2017-07-07 22:41:32,602 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:348)] Preparing to move file /usr/local/flume/logtest/testKafka.log to /usr/local/flume/logtest/testKafka.log.COMPLETED
- 2017-07-07 22:41:35,669 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - kafka.utils.Logging$class.info(Logging.scala:68)] Fetching metadata from broker id:0,host:localhost,port:9092 with correlation id 0 for 1 topic(s) Set(test)
- 2017-07-07 22:41:35,728 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - kafka.utils.Logging$class.info(Logging.scala:68)] Connected to localhost:9092 for producing
- 2017-07-07 22:41:35,757 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - kafka.utils.Logging$class.info(Logging.scala:68)] Disconnecting from localhost:9092
- 2017-07-07 22:41:35,791 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - kafka.utils.Logging$class.info(Logging.scala:68)] Connected to slave2:9092 for producing
Flume和Kafka整合安装的更多相关文章
- flume与kafka整合
flume与kafka整合 前提: flume安装和测试通过,可参考:http://www.cnblogs.com/rwxwsblog/p/5800300.html kafka安装和测试通过,可参考: ...
- ambari下的flume和kafka整合
1.配置flume #扫描指定文件配置 agent.sources = s1 agent.channels = c1 agent.sinks = k1 agent.sources.s1.type=ex ...
- flume和kafka整合（转）
原文链接:Kafka flume 整合 前提 前提是要先把flume和kafka独立的部分先搭建好. 下载插件包 下载flume-kafka-plus:https://github.com/beyon ...
- Flume 学习笔记之 Flume NG+Kafka整合
Flume NG集群+Kafka集群整合: 修改Flume配置文件(flume-kafka-server.conf),让Sink连上Kafka hadoop1: #set Agent name a1. ...
- CentOS7搭建Flume与Kafka整合及基础操作与测试
前提 已完成Kafka的搭建,具体步骤参照CentOS7搭建Kafka单机环境及基础操作 Flume安装 下载 wget http://mirrors.tuna.tsinghua.edu.cn/apa ...
- 大数据入门第二十四天——SparkStreaming（二）与flume、kafka整合
前一篇中数据源采用的是从一个socket中拿数据,有点属于“旁门左道”,正经的是从kafka等消息队列中拿数据! 主要支持的source,由官网得知如下: 获取数据的形式包括推送push和拉取pull ...
- hadoop 之 kafka 安装与 flume -> kafka 整合
62-kafka 安装 : flume 整合 kafka 一.kafka 安装 1.下载 http://kafka.apache.org/downloads.html 2. 解压 tar -zxvf ...
- Flume+Kafka整合
脚本生产数据---->flume采集数据----->kafka消费数据------->storm集群处理数据 日志文件使用log4j生成,滚动生成! 当前正在写入的文件在满足一定的数 ...
- 大数据入门第十八天——kafka整合flume、storm
一.实时业务指标分析 1.业务 业务: 订单系统---->MQ---->Kakfa--->Storm 数据:订单编号.订单时间.支付编号.支付时间.商品编号.商家名称.商品价格.优惠 ...
随机推荐
- wsgi&nginx-理解
WSGI协议 首先弄清下面几个概念:WSGI:全称是Web Server Gateway Interface,WSGI不是服务器,python模块,框架,API或者任何软件,只是一种规范,描述web ...
- python 异常处理函数--raise
Python 异常处理--raise函数用法 在Python中,要想引发异常,最简单的形式就是输入关键字raise,后跟要引发的异常的名称.异常名称标识出具体的类: Python异常处理是那些类的对象 ...
- HDU 6038 17多校1 Function（找循环节/环）
Problem Description You are given a permutation a from 0 to n−1 and a permutation b from 0 to m−1. D ...
- HDU 6049 17多校2 Sdjpx Is Happy（思维题difficult）
Problem Description Sdjpx is a powful man,he controls a big country.There are n soldiers numbered 1~ ...
- HDU 4704 Sum（隔板原理+组合数求和公式+费马小定理+快速幂）
题目传送:http://acm.hdu.edu.cn/showproblem.php?pid=4704 Problem Description Sample Input 2 Sample Outp ...
- 生成式模型 VS 判别式模型
1 定义 1.1 生成式模型 生成式模型(Generative Model)会对x和y的联合分布p(x,y)建模,然后通过贝叶斯公式来求得 p(yi|x),然后选取使得p(yi|x) 最大的 yi,即 ...
- Windows编程___创建窗口
创建Windows窗口不难,可以简要的概括为: 1,# 注册一个窗口类 填充WNDCLASS结构 书写窗口消息处理函数WinProc 2,# 创建一个窗口 填写基本的窗口信息 3,# 显示窗口 4,# ...
- CodeForces - 1101G :(Zero XOR Subset)-less(线性基)
You are given an array a1,a2,…,an of integer numbers. Your task is to divide the array into the maxi ...
- lesson6-图像分割-小象c
显著性检测:1)显著性物体检测-最能引起视觉注意的物体区域2)注视点预测:人类视觉注意机制 视觉注意机制的两种机制:1)自底而上基于数据驱动的注意机制,如颜色.边缘 2)自上而下基于任务驱动的目标的注 ...
- Mac下常用工具
MySql--Sequel Pro