Flume使用

avro agent

配置文件

cd $FLUME_HOME/conf
vim avro.conf
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1

    a1.sources.r1.type = avro
    a1.sources.r1.channels = c1
    a1.sources.r1.bind = dev01
    a1.sources.r1.port = 4411

    a1.sinks.k1.type = logger

    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100

    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

启动agent

用以下脚本开启agent客户端，从以下信息中可以看到当前 agent 的 jmx 监控端口号是4154，且绑定4411端口。

flume-ng agent -c . -f /opt/bigdata/flume/conf/avro.conf -n a1 -Dflume.root.logger=INFO,console
    Info: Sourcing environment configuration script /opt/bigdata/flume/conf/flume-env.sh
    Info: Including Hadoop libraries found via (/opt/bigdata/hadoop/bin/hadoop) for HDFS access
    Info: Including HBASE libraries found via (/opt/bigdata/hbase/bin/hbase) for HBASE access
    Info: Including Hive libraries found via () for Hive access
    + exec /opt/soft/jdk1.8.0_144/bin/java -Xms100m -Xmx200m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=4154 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dflume.root.logger=INFO,console -cp '/opt/bigdata/flume/conf:/opt/bigdata/flume/lib/*:/opt/bigdata/hadoop/etc/hadoop:/opt/bigdata/hadoop/share/hadoop/common/lib/*:/opt/bigdata/hadoop/share/hadoop/common/*:/opt/bigdata/hadoop/share/hadoop/hdfs:/opt/bigdata/hadoop/share/hadoop/hdfs/lib/*:/opt/bigdata/hadoop/share/hadoop/hdfs/*:/opt/bigdata/hadoop/share/hadoop/yarn/lib/*:/opt/bigdata/hadoop/share/hadoop/yarn/*:/opt/bigdata/hadoop/share/hadoop/mapreduce/lib/*:/opt/bigdata/hadoop/share/hadoop/mapreduce/*:/opt/bigdat......
    org.apache.flume.source.AvroSource.start(AvroSource.java:234)] Starting Avro source r1: { bindAddress: dev01, port: 4411 }...
    2018-11-02 11:42:26,026 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
    2018-11-02 11:42:26,026 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: r1 started
    2018-11-02 11:42:26,027 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:259)] Avro source r1 started.

启动文本传输

新开一个终端进行以下动作。

1. 创建一个文件

   echo "hello world" > /home/dev/flume/log.00
   

2. 开启文本传输

   flume-ng avro-client --conf conf -H dev01 -p 4411 -F /home/dev/flume/log.00
   

3. 可以从agent客户端看到文件的内容

   2018-11-02 11:48:05,426 (New I/O server boss #9) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 => /192.168.20.10:4411] OPEN
   2018-11-02 11:48:05,428 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 => /192.168.20.10:4411] BOUND: /192.168.20.10:4411
   2018-11-02 11:48:05,428 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 => /192.168.20.10:4411] CONNECTED: /192.168.20.10:46114
   2018-11-02 11:48:05,705 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 :> /192.168.20.10:4411] DISCONNECTED
   2018-11-02 11:48:05,705 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 :> /192.168.20.10:4411] UNBOUND
   2018-11-02 11:48:05,706 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 :> /192.168.20.10:4411] CLOSED
   2018-11-02 11:48:05,706 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed(NettyServer.java:209)] Connection to /192.168.20.10:46114 disconnected.
   2018-11-02 11:48:07,804 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64                hello world }