Flume配置Load balancing Sink Processor
1 官网内容
2 找一个图来理解一目了然
3 详细配置
配置文件load_source_case.conf
配置数据入口 source到channel 配置了两个sink用来做负载均衡
#配置文件:
a1.sources= r1
a1.sinks= k1 k2
a1.channels= c1 #负载平衡
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type =load_balance
a1.sinkgroups.g1.processor.backoff =true
a1.sinkgroups.g1.processor.selector=round_robin #Describe/configure the source
a1.sources.r1.type= exec
a1.sources.r1.command= tail -F /tmp/logs/test.log #Describe the sink
a1.sinks.k1.type= avro
a1.sinks.k1.hostname= 127.0.0.1
a1.sinks.k1.port= 50001 a1.sinks.k2.type= avro
a1.sinks.k2.hostname= 127.0.0.1
a1.sinks.k2.port= 50002 # Usea channel which buffers events in memory
a1.channels.c1.type= memory
a1.channels.c1.capacity= 1000
a1.channels.c1.transactionCapacity= 100 # set channel
a1.sinks.k1.channel= c1
a1.sinks.k2.channel= c1
a1.sources.r1.channels= c1
sink1配置
# Name the components on this agent
a2.sources = r1
a2.sinks = k1
a2.channels = c1 # Describe/configure the source
a2.sources.r1.type = avro
a2.sources.r1.channels = c1
a2.sources.r1.bind = 127.0.0.1
a2.sources.r1.port = 50001 # Describe the sink
a2.sinks.k1.type = logger
a2.sinks.k1.channel = c1 # Use a channel which buffers events inmemory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100
sink2配置
# Name the components on this agent
a3.sources = r1
a3.sinks = k1
a3.channels = c1 # Describe/configure the source
a3.sources.r1.type = avro
a3.sources.r1.channels = c1
a3.sources.r1.bind = 127.0.0.1
a3.sources.r1.port = 50002 # Describe the sink
a3.sinks.k1.type = logger
a3.sinks.k1.channel = c1 # Use a channel which buffers events inmemory
a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100
4启动服务
先启动两个sink flume-ng agent -c conf -f /mnt/software/flume-1.6.0/flume-conf/loadBalance/sink1.conf -n a2 -Dflume.root.logger=DEBUG,console
flume-ng agent -c conf -f /mnt/software/flume-1.6.0/flume-conf/loadBalance/sink2.conf -n a3 -Dflume.root.logger=DEBUG,console 再启动source flume-ng agent -c conf -f /mnt/software/flume-1.6.0/flume-conf/loadBalance/load_source_case.conf -n a1 -Dflume.root.logger=DEBUG,console
5 查看效果
第一次启动走了sink1 9/02/21 23:20:55 INFO ipc.NettyServer: [id: 0x617271c1, /127.0.0.1:47138 => /127.0.0.1:50001] CONNECTED: /127.0.0.1:47138
19/02/21 23:21:01 INFO sink.LoggerSink: Event: { headers:{} body: 7A 68 61 6E 67 6A 69 6E zhangjin }
19/02/21 23:21:01 INFO sink.LoggerSink: Event: { headers:{} body: 78 78 78 78 xxxx }
19/02/21 23:21:01 INFO sink.LoggerSink: Event: { headers:{} body: 79 79 79 79 yyyy }
19/02/21 23:21:01 INFO sink.LoggerSink: Event: { headers:{} body: 7A 68 61 6E 67 6A 69 6E zhangjin }
19/02/21 23:21:01 INFO sink.LoggerSink: Event: { headers:{} body: 78 78 78 78 xxxx }
19/02/21 23:21:01 INFO sink.LoggerSink: Event: { headers:{} body: 79 79 79 79 yyyy }
19/02/21 23:21:01 INFO sink.LoggerSink: Event: { headers:{} body: 7A 68 61 6E 67 6A 69 6E zhangjin }
19/02/21 23:21:01 INFO sink.LoggerSink: Event: { headers:{} body: 78 78 78 78 xxxx }
19/02/21 23:21:01 INFO sink.LoggerSink: Event: { headers:{} body: 79 79 79 79 yyyy }
19/02/21 23:21:01 INFO sink.LoggerSink: Event: { headers:{} body: 5B 7B 20 22 68 65 61 64 65 72 73 22 20 3A 7B 22 [{ "headers" :{" } 19/02/21 23:23:47 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F hello } 第二次追加数据走了sink2 19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 77 6F 72 6C 64 world }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 6A 61 76 61 java }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 73 63 61 6C 61 scala }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 68 61 64 6F 6F 70 hadoop }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 7A 68 61 6E 67 6A 69 6E zhangjin }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 78 78 78 78 xxxx }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 79 79 79 79 yyyy }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 7A 68 61 6E 67 6A 69 6E zhangjin }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 78 78 78 78 xxxx }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 79 79 79 79 yyyy }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 7A 68 61 6E 67 6A 69 6E zhangjin }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 78 78 78 78 xxxx }
19/02/21 23:23:49 INFO sink.LoggerSink: Event: { headers:{} body: 79 79 79 79 yyyy } 数据文件 ello
world
java
scala
hadoop
zhangjin
xxxx
yyyy
zhangjin
xxxx
yyyy
zhangjin
xxxx
yyyy
[{ "headers" :{"state" : "CZ"},"body" : "TEST1"}]
hello
world
java
scala
hadoop
zhangjin
xxxx
yyyy
zhangjin
xxxx
yyyy
zhangjin
xxxx
yyyy
~
~
6 总结,从效果上来看实现了负载,选择的是轮询算法,其他的大家可以多测试一下
Flume配置Load balancing Sink Processor的更多相关文章
- Flume-Failover Sink Processor 故障转移与 Load balancing Sink 负载均衡
接上一篇:https://www.cnblogs.com/jhxxb/p/11579518.html 使用 Flume1 监控一个端口,其 sink 组中的 sink 分别对接 Flume2 和 Fl ...
- flume配置和说明(转)
Flume是什么 收集.聚合事件流数据的分布式框架 通常用于log数据 采用ad-hoc方案,明显优点如下: 可靠的.可伸缩.可管理.可定制.高性能 声明式配置,可以动态更新配置 提供上下文路由功能 ...
- Windows Server 2008配置Network Load Balancing(服务群集)
最近配置SharePoint 2013 WFE 时,客户提到要让多台WFE能load balance,于是研究了下Network Load Balancing. 当把一台服务器 ...
- Flume监听文件目录sink至hdfs配置
一:flume介绍 Flume是一个分布式.可靠.和高可用的海量日志聚合的系统,支持在系统中定制各类数据发送方,用于收集数据:同时,Flume提供对数据进行简单处理,并写到各种数据接受方(可定制)的能 ...
- Flume中的HDFS Sink配置参数说明【转】
转:http://lxw1234.com/archives/2015/10/527.htm 关键字:flume.hdfs.sink.配置参数 Flume中的HDFS Sink应该是非常常用的,其中的配 ...
- 大数据学习day36-----flume02--------1.avro source和kafka source 2. 拦截器(Interceptor) 3. channel详解 4 sink 5 slector(选择器)6 sink processor
1.avro source和kafka source 1.1 avro source avro source是通过监听一个网络端口来收数据,而且接受的数据必须是使用avro序列化框架序列化后的数据.a ...
- Network Load Balancing Technical Overview--reference
http://technet.microsoft.com/en-us/library/bb742455.aspx Abstract Network Load Balancing, a clusteri ...
- 【原创】大数据基础之Flume(2)Sink代码解析
flume sink核心类结构 1 核心接口Sink org.apache.flume.Sink /** * <p>Requests the sink to attempt to cons ...
- 负载均衡(Load Balancing)学习笔记(二)
概述 文章负载均衡(Load Balancing)学习笔记(一) 讲述了负载均衡的一般性原理,本文继续介绍常见的实现负载均衡的方法. HTTP重定向 HTTP重定向服务器是一台普通的Web服务器,用户 ...
随机推荐
- react-native中的TextInput
TextInput是一个允许用户输入文本的基础组件.它有一个名为onChangeText的属性,此属性接受一个函数, 而此函数会在文本变化时被调用.另外还有一个名为onSubmitEditing的属性 ...
- 记录替换django的user模型出现的异常InconsistentMigrationHistory
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration admin.0001_initial is applie ...
- PHP冒泡排序算法
算法说明: 冒泡排序大概的意思是依次比较相邻的两个数,然后根据大小做出排序,直至最后两位数.由于在排序过程中总是小数往前放,大数往后放,相当于气泡往上升,所以称作冒泡排序.但其实在实际过程中也可以根据 ...
- C connect实现Timeout效果(Windows)
int SocketClient::connectTimeOutForWin(SOCKET &connect_fd, const int &timeout, const sockadd ...
- c#中序列化和反序列化的理解
using System.IO;using System.Runtime.Serialization.Formatters.Binary; 序列化:对象转化为文件的过程(字节流) 反序列化:文件(字节 ...
- Elastic 开发篇(3)
1.报错: java.lang.NoSuchFieldError: FAIL_ON_SYMBOL_HASH_OVERFLOW 原因: 系统中已引入jackson版本,但版本较低,缺少所需要的字段. 解 ...
- redis主从复制配置(1)
我们来配置一个一主两从的服务,根据前面写的已经配置好的redis基础上进行主从配置 一:进入redis的配置目录 cd /usr/local/redis 创建下面3个目录,命令为:make -p /u ...
- nginx根据cookie分流
转载互联网 nginx根据cookie分流众所周知,nginx可以根据url path进行分流,殊不知对于cookie分流也很强大,同时这也是我上篇提到的小流量实验的基础. 二话不说,先看需求,两台服 ...
- 10款 Mac 经典原型设计开发软件推荐
在Mac上有大量强大的开发和设计工具,今天和大家推荐10款Mac上的经典原型设计开发工具,原型设计工具是开发者必备的一款工具,无论是网站开发还是移动APP开发,都需要在前期进行严格细致的原型设计,才能 ...
- python自动化开发-[第十天]-线程、协程、socketserver
今日概要 1.线程 2.协程 3.socketserver 4.基于udp的socket(见第八节) 一.线程 1.threading模块 第一种方法:实例化 import threading imp ...