大数据学习（19）—— Flume环境搭建

系统要求

Java1.8或以上
内存要足够大
硬盘足够大
Agent对源和目的要有读写权限

Flume部署

我这8G内存的电脑之前搭建Hadoop、Hive和HBase已经苟延残喘了，怀疑会卡死，硬着头皮上吧。先解压缩，大数据的这些产品都是一个部署套路。

我准备在server01上部署flume，单节点就可以了。在公司生产环境部署要考虑高可用。

[root@server01 home]# tar -xvf apache-flume-1.9.0-bin.tar.gz -C /usr

[root@server01 home]# cd /usr

[root@server01 usr]# chown -R hadoop:hadoop apache-flume-1.9.0-bin/

[root@server01 usr]# mv apache-flume-1.9.0-bin/ apache-flume-1.9.0

在profile文件中添加配置

FLUME_HOME=/usr/apache-flume-1.9.0/

PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$HBASE_HOME/bin:$FLUME_HOME/bin

刷新配置文件

[root@server01 bin]# source /etc/profile

修改flume配置文件

[hadoop@server01 conf]$ pwd

/usr/apache-flume-1.9.0/conf

[hadoop@server01 conf]$ mv flume-env.sh.template flume-env.sh

[hadoop@server01 conf]$ vi flume-env.sh

把flume-env.sh里的JAVA_HOME修改为绝对路径

export JAVA_HOME=/usr/java/jdk1.8.0

Flume启动

我们试一下通过网络端口写入数据。新建一个配置文件。

[hadoop@server01 conf]$ vi config1

数据流向：telent -> source -> channel -> sink -> logger

具体配置内容如下。

[hadoop@server01 conf]$ cat config1

# Name the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = netcat

a1.sources.r1.bind = server01

a1.sources.r1.port = 44444

# Describe the sink

a1.sinks.k1.type = logger

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

启动flume。注意flume1.0以后叫ng（next generation），之前叫og（original generation）。

[hadoop@server01 apache-flume-1.9.0]$ flume-ng agent --conf conf --conf-file conf/config1  --name a1 -Dflume.root.logger=INFO,console

启动之后，另开server02对44444端口发送数据。

[hadoop@server02 ~]$ telnet server01 44444

Trying 182.182.0.8...

Connected to server01.

Escape character is '^]'.

hello

OK

thank you

OK

thank you very much

OK

how are you everyone

OK

看看server01控制台输出了啥。

2021-01-07 11:17:14,198 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 65 6C 6C 6F 0D                               hello. }

2021-01-07 11:18:24,209 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 74 68 61 6E 6B 20 79 6F 75 0D                   thank you. }

2021-01-07 11:18:34,088 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 74 68 61 6E 6B 20 79 6F 75 20 76 65 72 79 20 6D thank you very m }

2021-01-07 11:18:51,602 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 6F 77 20 61 72 65 20 79 6F 75 20 65 76 65 72 how are you ever }

我们可以看到，控制台只会输出前面几个字节的内容，但是信息已经获取到了。

再来一个例子

上面是一个最简单的例子，从网络端口获取数据，输出到控制台。再来一个复杂一点的，从日志文件获取增量数据，写入HDFS。

做过开发的都清楚用tail -f filename来查看最新的请求日志，配置文件新建config2，内容如下。

[hadoop@server01 conf]$ cat config2

# Name the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = exec

a1.sources.r1.command = tail -F /home/log.txt

# Describe the sink

a1.sinks.k1.type = hdfs

a1.sinks.k1.hdfs.path = hdfs://mycluster/flume

a1.sinks.k1.hdfs.writeFormat = Text

a1.sinks.k1.hdfs.fileType = DataStream

a1.sinks.k1.hdfs.rollInterval = 10

a1.sinks.k1.hdfs.rollSize = 0

a1.sinks.k1.hdfs.rollCount = 0

a1.sinks.k1.hdfs.filePrefix = %Y-%m-%d

a1.sinks.k1.hdfs.useLocalTimeStamp = true

# Use a channel which buffers events in file

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

启动hdfs，用上面的配置文件启动flume。

[hadoop@server01 apache-flume-1.9.0]$ flume-ng agent --name a1 --conf conf --conf-file conf/config2 -Dflume.root.logger=INFO,console

启动报错。

2021-01-07 19:19:51,905 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed

java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1380)

    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1361)

    at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1703)

    at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:221)

    at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:572)

    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:412)

    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)

    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)

    at java.lang.Thread.run(Thread.java:748)

Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1380)

    at org.apache.hadoop.conf.Configuration.set(Configuration.java:1361)

    at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1703)

    at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:221)

    at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:572)

    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:412)

    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)

    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)

    at java.lang.Thread.run(Thread.java:748)

这跟Hive启动错误是一样的，原因就是与Hadoop的guava包版本不一致。把Hadoop的jar包拷到Flume路径下，删除老的jar包。在Flume的lib目录执行如下命令。

[hadoop@server01 lib]$ cp /usr/hadoop-3.3.0/share/hadoop/common/lib/guava-27.0-jre.jar .

[hadoop@server01 lib]$ ll|grep guava

-rw-rw-r--. 1 hadoop hadoop 1648200 9月  13 2018 guava-11.0.2.jar

-rw-r--r--. 1 hadoop hadoop 2747878 1月  12 11:42 guava-27.0-jre.jar

[hadoop@server01 lib]$ rm guava-11.0.2.jar

[hadoop@server01 lib]$ ll|grep guava

-rw-r--r--. 1 hadoop hadoop 2747878 1月  12 11:42 guava-27.0-jre.jar

再次启动Flume。启动完毕后，模拟向/home/log.txt写入数据，中间间隔一段时间。

[root@server01 home]# echo "hello,thank you,thank you very much" >> log.txt

[root@server01 home]# echo "How are you Indian Mi fans?" >> log.txt

再去看看HDFS生成的文件里有什么内容。

打开下面的两个文件，看看内容。原谅我不厚道地用了雷总歌词。

这样就把日志收集到HDFS了，后续可以通过MR任务来处理HDFS文件，提取需要的内容。