flume的安装和使用

1.下载

[linyouyi@hadoop01 software]$ wget https://mirrors.aliyun.com/apache/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
---- ::--  https://mirrors.aliyun.com/apache/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
Resolving mirrors.aliyun.com (mirrors.aliyun.com)... 27.148.180.227, 119.147.111.230, 119.147.111.231, ...
Connecting to mirrors.aliyun.com (mirrors.aliyun.com)|27.148.180.227|:... connected.
HTTP request sent, awaiting response...  OK
Length:  (65M) [application/gzip]
Saving to: ‘apache-flume-1.9.-bin.tar.gz’
 
%[=======================================================================>] ,,  .0MB/s   in .2s   
 
-- :: (30.0 MB/s) - ‘apache-flume-1.9.-bin.tar.gz’ saved [/]

2.解压

[linyouyi@hadoop01 software]$ tar -zxvf apache-flume-1.9.-bin.tar.gz -C /hadoop/module/
[linyouyi@hadoop01 software]$ cd /hadoop/module/
[linyouyi@hadoop01 module]$ cd apache-flume-1.9.-bin/
[linyouyi@hadoop01 apache-flume-1.9.-bin]$ ll
total
drwxr-xr-x   linyouyi linyouyi   Sep   : bin
-rw-rw-r--   linyouyi linyouyi  Nov    CHANGELOG
drwxr-xr-x   linyouyi linyouyi   Sep   : conf
-rw-r--r--   linyouyi linyouyi   Nov    DEVNOTES
-rw-r--r--   linyouyi linyouyi   Nov    doap_Flume.rdf
drwxrwxr-x  linyouyi linyouyi   Dec    docs
drwxrwxr-x   linyouyi linyouyi   Sep   : lib
-rw-rw-r--   linyouyi linyouyi  Dec    LICENSE
-rw-r--r--   linyouyi linyouyi    Nov    NOTICE
-rw-r--r--   linyouyi linyouyi   Nov    README.md
-rw-rw-r--   linyouyi linyouyi   Dec    RELEASE-NOTES
drwxrwxr-x   linyouyi linyouyi   Sep   : tools
[linyouyi@hadoop01 apache-flume-1.9.-bin]$ ll conf/
total 

3.启动agent

使用名为flume-ng的shell脚本启动代理程序，该脚本位于Flume发行版的bin目录中。您需要在命令行上指定代理名称，config目录和配置文件：

bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template
 
-n agent_name #取名称
-c conf              #配置文件夹
-f  conf/flume-conf.properties.template    #配置文件

4.一个简单的例子

http://flume.apache.org/FlumeUserGuide.html#netcat-tcp-source

在这里，我们给出一个示例配置文件，描述单节点Flume部署。此配置允许用户生成事件，然后将其记录到控制台。

[linyouyi@hadoop01 apache-flume-1.9.-bin]$ vim conf/example.conf
#example.conf：单节点Flume配置
 
＃在此代理上命名组件
a1.sources  =  r1
a1.sinks  =  k1
a1.channels  =  c1 
 
＃描述/配置源
a1.sources.r1.type  =  netcat
a1。 sources.r1.bind  =  localhost
a1.sources.r1.port  =   
 
＃描述接收器
a1.sinks.k1.type  =  logger 
 
＃使用一个缓冲内存中事件的通道
a1.channels.c1.type  =  memory
a1.channels .c1.capacity  =
a1.channels.c1.transactionCapacity  =  
 
＃将源和接收器绑定到通道
a1.sources.r1.channels  =  c1
a1.sinks.k1.channel  =  c1

此配置定义名为a1的单个代理。a1有一个侦听端口44444上的数据的源，一个缓冲内存中事件数据的通道，以及一个将事件数据记录到控制台的接收器。配置文件命名各种组件，然后描述其类型和配置参数。给定的配置文件可能会定义几个命名的代理 当一个给定的Flume进程启动时，会传递一个标志，告诉它要显示哪个命名代理。

鉴于此配置文件，我们可以按如下方式启动Flume：

[linyouyi@hadoop01 apache-flume-1.9.-bin]$ bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1 -Dflume.root.logger=INFO,console

请注意，在完整部署中，我们通常会包含一个选项： - conf=<conf-dir>。所述<CONF-DIR>目录将包括一个shell脚本flume-env.sh和潜在的一个log4j的属性文件。在这个例子中，我们传递一个Java选项来强制Flume登录到控制台，我们没有自定义环境脚本。

从一个单独的终端，我们可以telnet端口44444并向Flume发送一个事件：

$ telnet localhost
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
Hello world! <ENTER>
OK

原始的Flume终端将在日志消息中输出事件。

// :: INFO source.NetcatSource: Source starting
// :: INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:]
// :: INFO sink.LoggerSink: Event: { headers:{} body:   6C 6C 6F   6F  6C   0D          Hello world!. }

恭喜 - 您已成功配置并部署了Flume代理！后续部分更详细地介绍了代理配置。

5.exec采集

http://flume.apache.org/FlumeUserGuide.html#exec-source

[linyouyi@hadoop01 apache-flume-1.9.-bin]$ cp conf/example.conf conf/example-exec.conf
[linyouyi@hadoop01 apache-flume-1.9.-bin]$ vim conf/example-exec.conf
#example.conf：单节点Flume配置
 
#在此代理上命名组件
a1.sources  =  r1
a1.sinks  =  k1
a1.channels  =  c1 
 
#描述/配置源
a1.sources.r1.type  =  exec
a1。 sources.r1.command  =  tail -F /hadoop/module/text.log 
 
#描述接收器
a1.sinks.k1.type  =  logger 
 
#使用一个缓冲内存中事件的通道
a1.channels.c1.type  =  memory
a1.channels .c1.capacity  =
a1.channels.c1.transactionCapacity  =  
 
#将源和接收器绑定到通道
a1.sources.r1.channels  =  c1
a1.sinks.k1.channel  =  c1

启动

[linyouyi@hadoop01 apache-flume-1.9.-bin]$ bin/flume-ng agent --conf conf --conf-file conf/example-exec.conf --name a1 -Dflume.root.logger=INFO,console

打开另一个客户端往/hadoop/module/text.log不断写数据,发现原始的Flume终端消息中输出信息

[linyouyi@hadoop01 module]$ echo "flume" >> text.log
[linyouyi@hadoop01 module]$ echo "flume" >> text.log
[linyouyi@hadoop01 module]$ cat text.log
flume
flume
flume
[linyouyi@hadoop01 module]$ echo "flume" >> text.log
[linyouyi@hadoop01 module]$ echo "hello linyouyi" >> text.log

原始终端输出信息

-- ::, (lifecycleSupervisor--) [INFO - org.apache.flume.source.ExecSource.start(ExecSource.java:)] Exec source starting with command: tail -F /hadoop/module/text.log
-- ::, (lifecycleSupervisor--) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
-- ::, (lifecycleSupervisor--) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:)] Component type: SOURCE, name: r1 started
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:)] Event: { headers:{} body:  6C  6D                                   flume }
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:)] Event: { headers:{} body:  6C  6D                                   flume }
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:)] Event: { headers:{} body:  6C  6D                                   flume }
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:)] Event: { headers:{} body:  6C  6D                                   flume }
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:)] Event: { headers:{} body:   6C 6C 6F  6C  6E  6F          hello linyouyi }

在配置文件中使用环境变

http://flume.apache.org/FlumeUserGuide.html