1. 写在前面

flume-ng高可用长在大数据处理环节第一个出现，对于处理日志文件有很好的作用，本篇博客将详细介绍flume-ng的高可用负载均衡搭建

2. flume-ng高可用负载均衡描述

在一般情况下，Flume-ng高可用采用server和client模式，client主要负责数据源source及数据流向端的sink指向配置，server主要负责数据流向sink详细配置，client需要将server的信息统一管理，server和sink之间数据连接通过channels

3. 配置server，这里配置三个server

flume-server1.properties

#set Agent name

agent.sources = r1

agent.channels = c1

agent.sinks = k1

#set channel

agent.channels.c1.type = memory

agent.channels.c1.capacity = 1024000

agent.channels.c1.transactionCapacity = 10000

agent.channels.c1.byteCapacity=134217728

agent.channels.c1.byteCapacityBufferPercentage=80

# other node,nna to nns

agent.sources.r1.type = avro

agent.sources.r1.bind = ynjz003

agent.sources.r1.port = 52020

agent.sources.r1.interceptors = i1

agent.sources.r1.interceptors.i1.type = static

agent.sources.r1.interceptors.i1.key = Collector

agent.sources.r1.interceptors.i1.value = ynjz003

agent.sources.r1.channels = c1

#set sink to hdfs

agent.sinks.k1.channel = c1

agent.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink

agent.sinks.k1.brokerList = ynjz003:9092,ynjz004:9092,ynjz005:9092,ynjz006:9092,ynjz007:9092,ynjz008:9092,ynjz009:9092

agent.sinks.k1.topic = flume-kafka-meijs33

agent.sinks.k1.serializer.class = kafka.serializer.StringEncoder

flume-server2.properties

#set Agent name

agent.sources = r1

agent.channels = c1

agent.sinks = k1

#set channel

agent.channels.c1.type = memory

agent.channels.c1.capacity = 1024000

agent.channels.c1.transactionCapacity = 10000

agent.channels.c1.byteCapacity=134217728

agent.channels.c1.byteCapacityBufferPercentage=80

# other node,nna to nns

agent.sources.r1.type = avro

agent.sources.r1.bind = ynjz004

agent.sources.r1.port = 52020

agent.sources.r1.interceptors = i1

agent.sources.r1.interceptors.i1.type = static

agent.sources.r1.interceptors.i1.key = Collector

agent.sources.r1.interceptors.i1.value = ynjz004

agent.sources.r1.channels = c1

#set sink to hdfs

agent.sinks.k1.channel = c1

agent.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink

agent.sinks.k1.brokerList = ynjz003:9092,ynjz004:9092,ynjz005:9092,ynjz006:9092,ynjz007:9092,ynjz008:9092,ynjz009:9092

agent.sinks.k1.topic = flume-kafka-meijs33

agent.sinks.k1.serializer.class = kafka.serializer.StringEncoder

flume-server3.properties

#set Agent name

agent.sources = r1

agent.channels = c1

agent.sinks = k1

#set channel

agent.channels.c1.type = memory

agent.channels.c1.capacity = 1024000

agent.channels.c1.transactionCapacity = 10000

agent.channels.c1.byteCapacity=134217728

agent.channels.c1.byteCapacityBufferPercentage=80

# other node,nna to nns

agent.sources.r1.type = avro

agent.sources.r1.bind = ynjz005

agent.sources.r1.port = 52020

agent.sources.r1.interceptors = i1

agent.sources.r1.interceptors.i1.type = static

agent.sources.r1.interceptors.i1.key = Collector

agent.sources.r1.interceptors.i1.value = ynjz005

agent.sources.r1.channels = c1

#set sink to hdfs

agent.sinks.k1.channel = c1

agent.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink

agent.sinks.k1.brokerList = ynjz003:9092,ynjz004:9092,ynjz005:9092,ynjz006:9092,ynjz007:9092,ynjz008:9092,ynjz009:9092

agent.sinks.k1.topic = flume-kafka-meijs33

agent.sinks.k1.serializer.class = kafka.serializer.StringEncoder

可以看出多个server配置的规律

3. 配置client，这里也配置一个client示例

flume-client.properties

#agent1 name

agent.channels = c1

agent.sources = r1

agent.sinks = k1 k2 k3 k4 k5 k6 k7

#set gruop

agent.sinkgroups = g1

#set channel

agent.channels.c1.type = memory

agent.channels.c1.capacity = 102400

agent.channels.c1.transactionCapacity = 1000

agent.channels.c1.byteCapacity=134217728

agent.channels.c1.byteCapacityBufferPercentage=80

agent.sources.r1.type = com.cbo.flume.source.zip.SpoolDirectorySource

agent.sources.r1.channels = c1

agent.sources.r1.spoolDir = /data/ynjz/workspace/zip

agent.sources.r1.fileHeader = true

agent.sources.r1.channels = c1

agent.sources.r1.flumeBatchSize=1000

agent.sources.r1.useFlumeEventFormat=false

agent.sources.r1.restart=true

agent.sources.r1.batchSize=1000

agent.sources.r1.batchTimeout=3000

agent.sources.r1.channels=c1

# set sink1

agent.sinks.k1.channel = c1

agent.sinks.k1.type = avro

agent.sinks.k1.hostname = ynjz003

agent.sinks.k1.port = 52020

# set sink2

agent.sinks.k2.channel = c1

agent.sinks.k2.type = avro

agent.sinks.k2.hostname = ynjz004

agent.sinks.k2.port = 52020

# set sink3

agent.sinks.k3.channel = c1

agent.sinks.k3.type = avro

agent.sinks.k3.hostname = ynjz005

agent.sinks.k3.port = 52020

# set sink4

agent.sinks.k1.channel = c1

agent.sinks.k1.type = avro

agent.sinks.k1.hostname = ynjz006

agent.sinks.k1.port = 52020

# set sink5

agent.sinks.k2.channel = c1

agent.sinks.k2.type = avro

agent.sinks.k2.hostname = ynjz007

agent.sinks.k2.port = 52020

# set sink6

agent.sinks.k3.channel = c1

agent.sinks.k3.type = avro

agent.sinks.k3.hostname = ynjz008

agent.sinks.k3.port = 52020

# set sink7

agent.sinks.k3.channel = c1

agent.sinks.k3.type = avro

agent.sinks.k3.hostname = ynjz009

agent.sinks.k3.port = 52020

#set sink group

agent.sinkgroups.g1.sinks = k1 k2 k3 k4 k5 k6 k7

#set failover

agent.sinkgroups.g1.processor.type = failover

agent.sinkgroups.g1.processor.priority.k1 = 10

agent.sinkgroups.g1.processor.priority.k2 = 10

agent.sinkgroups.g1.processor.priority.k3 = 10

agent.sinkgroups.g1.processor.priority.k4 = 10

agent.sinkgroups.g1.processor.priority.k5 = 10

agent.sinkgroups.g1.processor.priority.k6 = 10

agent.sinkgroups.g1.processor.priority.k7 = 10

agent.sinkgroups.g1.processor.maxpenalty = 10000

这里需要注意sinkgroups配置，flume sinkgroups在常用的应用中有两种方式failover和load_balance,failover可以理解为容错机制，在上面的配置中sink只会往一个kafka写入数据，但一个kafka挂了，failover机制会立马选举一个出来，所以这里的容错机制很完善，但是应对大数据量会影响数据写入的能力，所以建议在大数据量的时候采用load_balance配置，下面时配置示例

 #agent1 name

agent.channels = c1

agent.sources = r1

agent.sinks = k1 k2 k3 k4 k5 k6 k7

#set gruop

agent.sinkgroups = g1

#set channel

agent.channels.c1.type = memory

agent.channels.c1.capacity = 102400

agent.channels.c1.transactionCapacity = 24000

agent.channels.c1.byteCapacity=134217728

agent.channels.c1.byteCapacityBufferPercentage=80

agent.sources.r1.type = com.cbo.flume.source.zip.SpoolDirectorySource

agent.sources.r1.channels = c1

agent.sources.r1.spoolDir = /data/4G

agent.sources.r1.includePattern = ([^ ]*\.zip$)

agent.sources.r1.fileHeader = true

agent.sources.r1.channels = c1

agent.sources.r1.flumeBatchSize=10000

agent.sources.r1.useFlumeEventFormat=false

agent.sources.r1.restart=true

agent.sources.r1.batchSize=10000

agent.sources.r1.batchTimeout=3000

agent.sources.r1.channels=c1

# set sink1

agent.sinks.k1.channel = c1

agent.sinks.k1.type = avro

agent.sinks.k1.hostname = ynjz003

agent.sinks.k1.port = 52020

# set sink2

agent.sinks.k2.channel = c1

agent.sinks.k2.type = avro

agent.sinks.k2.hostname = ynjz004

agent.sinks.k2.port = 52020

# set sink3

agent.sinks.k3.channel = c1

agent.sinks.k3.type = avro

agent.sinks.k3.hostname = ynjz005

agent.sinks.k3.port = 52020

# set sink4

agent.sinks.k4.channel = c1

agent.sinks.k4.type = avro

agent.sinks.k4.hostname = ynjz006

agent.sinks.k4.port = 52020

# set sink5

agent.sinks.k5.channel = c1

agent.sinks.k5.type = avro

agent.sinks.k5.hostname = ynjz007

agent.sinks.k5.port = 52020

# set sink6

agent.sinks.k6.channel = c1

agent.sinks.k6.type = avro

agent.sinks.k6.hostname = ynjz008

agent.sinks.k6.port = 52020

# set sink7

agent.sinks.k7.channel = c1

agent.sinks.k7.type = avro

agent.sinks.k7.hostname = ynjz009

agent.sinks.k7.port = 52020

#set sink group

agent.sinkgroups.g1.sinks = k1 k2 k3 k4 k5 k6 k7

#set load_balance

agent.sinkgroups.g1.processor.type=load_balance

agent.sinkgroups.g1.processor.backoff=true

agent.sinkgroups.g1.processor.selector=random

在实际应用中多个client基本上一直，只有监控文件目录的配置不同即可agent.sources.r1.spoolDir = /data/4G

4. 启动flume-ng高可用集群

首先启动每个server,每个server只是配置文件flume-server-data.properties不同:

./bin/flume-ng agent --name agent --conf conf --conf-file conf/flume-server-data.properties -Dflume.root.logger=INFO,console > /data/ynjz/workspace/flume-server-data.log 2>&1 &

启动每个client,,每个server只是配置文件flume-client-data.properties不同:

./bin/flume-ng agent --name agent --conf conf --conf-file conf/flume-client-data.properties -Dflume.root.logger=INFO,console > /data/ynjz/workspace/flume-client-data.log 2>&1 &

在平时应用中，可以随时停止client，但停止了server没起而启动client会导致报错

Flume-ng高可用集群负载安装与配置的更多相关文章

  1. Flume 学习笔记之 Flume NG高可用集群搭建

    Flume NG高可用集群搭建: 架构总图: 架构分配: 角色 Host 端口 agent1 hadoop3 52020 collector1 hadoop1 52020 collector2 had ...

  2. 大数据高可用集群环境安装与配置（09）——安装Spark高可用集群

    1. 获取spark下载链接 登录官网:http://spark.apache.org/downloads.html 选择要下载的版本 2. 执行命令下载并安装 cd /usr/local/src/ ...

  3. Flume NG高可用集群搭建详解

    .Flume NG简述 Flume NG是一个分布式,高可用,可靠的系统,它能将不同的海量数据收集,移动并存储到一个数据存储系统中.轻量,配置简单,适用于各种日志收集,并支持 Failover和负载均 ...

  4. 大数据高可用集群环境安装与配置（06）——安装Hadoop高可用集群

    下载Hadoop安装包 登录 https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/ 镜像站,找到我们要安装的版本,点击进去复制下载链接 ...

  5. 大数据高可用集群环境安装与配置（07）——安装HBase高可用集群

    1. 下载安装包 登录官网获取HBase安装包下载地址 https://hbase.apache.org/downloads.html 2. 执行命令下载并安装 cd /usr/local/src/ ...

  6. 大数据高可用集群环境安装与配置（03）——设置SSH免密登录

    Hadoop的NameNode需要启动集群中所有机器的Hadoop守护进程,这个过程需要通过SSH登录来实现 Hadoop并没有提供SSH输入密码登录的形式,因此,为了能够顺利登录每台机器,需要将所有 ...

  7. 大数据高可用集群环境安装与配置（08）——安装Ganglia监控集群

    1. 安装依赖包和软件 在所有服务器上输入命令进行安装操作 yum install epel-release -y yum install ganglia-web ganglia-gmetad gan ...

  8. 大数据高可用集群环境安装与配置（02）——配置ntp服务

    NTP服务概述 NTP服务器[Network Time Protocol(NTP)]是用来使计算机时间同步化的一种协议,它可以使计算机对其服务器或时钟源(如石英钟,GPS等等)做同步化,它可以提供高精 ...

  9. 大数据高可用集群环境安装与配置（10）——安装Kafka高可用集群

    1. 获取安装包下载链接 访问https://kafka.apache.org/downloads 找到kafka对应版本 需要与服务器安装的scala版本一致(运行spark-shell可以看到当前 ...

随机推荐

  1. PAT 甲级真题题解（63-120）

    2019/4/3 1063 Set Similarity n个序列分别先放进集合里去重.在询问的时候,遍历A集合中每个数,判断下该数在B集合中是否存在,统计存在个数(分子),分母就是两个集合大小减去分 ...

  2. 牛客网 272B Xor Path（树上操作）

    题目链接:Xor Path 题意:每个顶点的点权为Ai,任意两点路径上点权异或和为Path(i,j),求所有Path(i,j)和. 题解:考虑每个顶点被用到的次数,分以下三种情况: 1.本身和其他顶点 ...

  3. Docker Dockerfile

    镜像的定制实际上就是定制每一层所添加的配置.文件.如果我们可以把每一层修改.安装.构建.操作的命令都写入一个脚本,用这个脚本来构建.定制镜像,那么之前提及的无法重复的问题.镜像构建透明性的问题.体积的 ...

  4. 小白在 Eclipse如何避免启动时自动building workspace和validating

    问题: Eclipse启动时会出现如下的情况(时间比较长): 原因所在: Validating 意为验证,validating... 逐个的检查每一个文件,Eclipse在启动时自动验证代码和创建wo ...

  5. ORCAL Merge into用法总结

    简单的说就是,判断表中有没有符合on()条件中的数据,有了就更新数据,没有就插入数据. 有一个表T,有两个字段a.b,我们想在表T中做Insert/Update,如果条件满足,则更新T中b的值,否则在 ...

  6. 实现select联动效果，数据从后台获取

    效果如下: 当type值选择完后,amount值会自动相应填入. 1. 从后台获取数据,为一个数组,里面包含多个对象. <select id="scholarshipTypeSelec ...

  7. vue路由实现多视图的单页应用

    多视图的单页应用:在一个页面中实现多个页面不同切换,url也发生相应变化. router-view结合this.$router.push("/pickUp")实现,效果如下: 当点 ...

  8. Modbus通讯数据格式:

    RTU&&TCP单个写入 RTU&&TCP多个写入 RTU&&TCP数据读取 ASCII码数据格式 Modbus功能码: 功能码 名称 作用 1 读取线 ...

  9. Tomcat系列(1)——Tomcat安装配置

    核心步骤 1. 安装JAVA(因为tomcat依赖于java) 配置:JAVA_HOME D:\Program Files (x86)\Java\jdk1.7.0 path  %JAVA_HOME%\ ...

  10. [面试]死锁-最简单的死锁demo

    死锁 什么是死锁 (百度百科) 死锁是指两个或两个以上的进程在执行过程中,由于竞争资源或者由于彼此通信而造成的一种阻塞的现象,若无外力作用,它们都将无法推进下去.此时称系统处于死锁状态或系统产生了死锁 ...