Flume 1.9.0 的安装(比较简单, 操作也不像老版本那么繁琐了)

之前已经完成了Hadoop集群、Hbase集群、Hive的搭建, 这次来安装一下flume-1.9.0

安装过程

1. 将tar包上传并解压到指定目录, 并修改名称

   tar -zxvf apache-flume-1.9.0-bin.tar.gz -C /opt/ronnie
   cd /opt/ronnie/
   mv apache-flume-1.9.0-bin/ flume-1.9.0
   

2. 进入flume配置文件目录, 拷贝一份环境配置文件并修改

   cd flume-1.9.0/conf/
   cp flume-env.sh.template flume-env.sh
   vim flume-env.sh
   

   # 修改JAVA_HOME
   export JAVA_HOME=/usr/lib/jvm/jdk1.8
   

3. 配置环境变量

   • vim ~/.bashrc

     # Flume
     export FLUME_HOME=/opt/ronnie/flume-1.9.0
     export PATH=$FLUME_HOME/bin:$PATH
     

   • source ~/.bashrc

4. 查看flume版本

   flume-ng version
   
   # result
   Flume 1.9.0
   Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
   Revision: d4fcab4f501d41597bc616921329a4339f73585e
   Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018
   From source with checksum 35db629a3bda49d23e9b3690c80737f9
   

5. 启动flume

   • 配置flume运行文件flume-conf.properties

     cp flume-conf.properties.template flume-conf.properties
     vim flume-conf.properties
     

     
     # 1.Name the components on this agent
     a1.sources = r1
     a1.sinks = k1
     a1.channels = c1
     
     # 2.Describe/configure the source
     a1.sources.r1.type = netcat
     a1.sources.r1.bind = node02 # 我改了hostname文件
     a1.sources.r1.port = 44444
     
     # 3.Describe the sink
     a1.sinks.k1.type = logger
     
     # 4.Use a channel which buffers events in memory
     a1.channels.c1.type = memory
     a1.channels.c1.capacity = 1000
     a1.channels.c1.transactionCapacity = 100
     
     # 5.Bind the source and sink to the channel
     a1.sources.r1.channels = c1
     a1.sinks.k1.channel = c1
     

   • 运行flume

     flume-ng agent \
     --name a1 \
     --conf $FLUME_HOME/conf \
     --conf-file $FLUME_HOME/conf/flume-conf.properties \
     -Dflume.root.logger=INFO,console
     

   • 启动成功

     erGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
     2019-12-09 16:05:05,927 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started
     2019-12-09 16:05:05,927 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:196)] Starting Sink k1
     2019-12-09 16:05:05,927 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:207)] Starting Source r1
     2019-12-09 16:05:05,928 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:155)] Source starting
     2019-12-09 16:05:05,935 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:166)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/192.168.180.131:44444]