flume ng 问题点

1. 启动Flume，出现脚本错误

错误如下：

bin/flume-ng: line 82: syntax error in conditional expression: unexpected token `('
bin/flume-ng: line 82: syntax error near `^java\.library\.path=(.'
bin/flume-ng: line 82: `      if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then'

将字符串的引用，加上双引号即可。如if [[ “$line” =~ “^java\.library\.path=(.*)$” ]]; then 。

flume-ng中有82、105、154行需要修改

2. flume sink 对接hdfs

29 Feb 2016 05:57:10,372 ERROR [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcher
Runnable.run:145)  - Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:38)
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:36)
        at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:108)
        at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:210)
        at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:553)
        at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:272)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)

它通过hadoop命令,自动去找到了hadoop的所有jra包并添加到了自己的classpath里面,所以就不存在这个错了.

那它到底依赖哪些jar包呢? 我也是快崩溃了,一个一个的找,一个一个的试,最终试出来了结果.

其实报错的那个类,在hadoop2的时候,是在hadoop-common当中. hadoop 1的话,应该是在hadoop-core中.

首先把这个hadoop-common包拷贝到flume的lib目录里面.

　　

总结一下,安装完flume之后,需要从hadoop中拷贝过来的jar包有以下

commons-configuration-1.6.jar
hadoop-auth-2.5.2.jar
hadoop-common-2.5.2.jar
hadoop-hdfs-2.5.2.jar
hadoop-mapreduce-client-core-2.5.2.jar