Flume Hello World！

Flume 是 Cloudera 公司开源出来的一套日志收集系统。模型如下所示：

图中Source，Sink分别代表数据源和数据目的地，channel表示Source和Sink之间的通道。配置文件为/path/to/flume/conf/flume.conf

# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory

# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 41414

# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.log-sink1.channel = ch1
agent1.sinks.log-sink1.type = hdfs
agent1.sinks.log-sink1.hdfs.path = hdfs://qaserver:9001/hello/helloworld 

# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = log-sink1

启动flume

/path/to/flume/bin/flume-ng agent --conf /path/to/flume/conf/ -f /path/to/flume/conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent1

执行以下命令将./hello.txt内容发送到avro-source1，即本地端口41414。

/path/to/flume/bin/flume-ng avro-client -H localhost -p 41414 -F ./hello.txt

./hello.txt内容如下

Hello World!

查看HDFS文件内容
hadoop fs -cat /hello/helloworld/FlumeData.1394013090403
ƒšî‘ÓD‘§´Ùache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable…=>ûù7ög
           Hello World!