spark streaming集成flume

1、 安装flume

flume安装，解压后修改flume_env.sh配置文件，指定java_home即可。

cp hdfs jar包到flume lib目录下（否则无法抽取数据到hdfs上）：

$ cp /opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.6.jar

/opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6-bin/lib/

2、 spark streaming集成flume

2.1）编译spark，获得集成flume jar包：

参考文档：http://www.cnblogs.com/wcwen1990/p/7688027.html

说明：spark streaming集成flume或者kafka需要一些支持jar包，这些jar包在编译spark过程中会自动在external目录下生成相应的jar文件，因此，这里需要编译spark来获得这些jar包。

Spark streaming集成flume主要需要：spark-streaming-flume_2.10-1.3.0.jar包。

2.2）集成jar包

$mkdir –p /opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs

$cp spark-1.3.0/external/flume/target/spark-streaming-flume_2.10-1.3.0.jar

/opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/

$ pwd

/opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6-bin/lib

$ cp flume-avro-source-1.5.0-cdh5.3.6.jar flume-ng-sdk-1.5.0-cdh5.3.6.jar

/opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/

$ cd /opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/

$ ll

flume-avro-source-1.5.0-cdh5.3.6.jar

flume-ng-sdk-1.5.0-cdh5.3.6.jar

spark-streaming-flume_2.10-1.3.0.jar

3、 编译flume配置文件（配置sources、channel、sink）：

$ cat flume-spark-push.conf

ss.sources = sss

ss.channels = ssc

ss.sinks = ssk

ss.sources.sss.type = exec

ss.sources.sss.command = tail -f /opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6-bin/wctotal.log

ss.sources.sss.shell = /bin/bash -c

ss.channels.ssc.type = memory

ss.channels.ssc.capacity = 1000

ss.channels.ssc.transactionCapacity = 100

ss.sinks.ssk.type = avro

ss.sinks.ssk.hostname = chavin.king

ss.sinks.ssk.port = 9999

ss.sources.sss.channels = ssc

ss.sinks.ssk.channel = ssc

4、 编写spark streaming程序：

import org.apache.spark._

import org.apache.spark.streaming._

import org.apache.spark.streaming.StreamingContext._

import org.apache.spark.streaming.flume._

import org.apache.spark.storage.StorageLevel

val ssc = new StreamingContext(sc, Seconds(5))

// read data

val stream = FlumeUtils.createStream(ssc, "chavin.king", 9999, StorageLevel.MEMORY_ONLY_SER_2)

stream.count().map(cnt => "Received " + cnt + " flume events." ).print()

ssc.start() // Start the computation

ssc.awaitTermination() // Wait for the computation to terminate

5、 spark-shell local模式测试spark streaming集成flume

$ bin/spark-shell --master local[2] --jars \

/opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/spark-streaming-flume_2.10-1.3.0.jar,/opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/flume-avro-source-1.5.0-cdh5.3.6.jar,/opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/flume-ng-sdk-1.5.0-cdh5.3.6.jar

执行步骤4中程序：

scala> import org.apache.spark._

import org.apache.spark._

scala> import org.apache.spark.streaming._

import org.apache.spark.streaming._

scala> import org.apache.spark.streaming.StreamingContext._

import org.apache.spark.streaming.StreamingContext._

scala> import org.apache.spark.streaming.flume._

import org.apache.spark.streaming.flume._

scala> import org.apache.spark.storage.StorageLevel

import org.apache.spark.storage.StorageLevel

scala> val ssc = new StreamingContext(sc, Seconds(5))

ssc: org.apache.spark.streaming.StreamingContext = org.apache.spark.streaming.StreamingContext@412dea3c

scala> val stream = FlumeUtils.createStream(ssc, "chavin.king", 9999, StorageLevel.MEMORY_ONLY_SER_2)

stream: org.apache.spark.streaming.dstream.ReceiverInputDStream[org.apache.spark.streaming.flume.SparkFlumeEvent] = org.apache.spark.streaming.flume.FlumeInputDStream@2bf9884

scala> stream.count().map(cnt => "Received " + cnt + " flume events." ).print()

//输入以下命令启动spark streaming

scala> ssc.start()

scala> ssc.awaitTermination()

6、 启动flume

bin/flume-ng agent -c conf -n ss -f conf/flume-spark-push.conf -Dflume.root.logger=DEBUG,console

7、 测试spark streaming集成flume：

$ echo “hadoop oracle mysql” >>/opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6-bin/wctotal.log

执行上边命令，可以在spark streaming命令行界面下看到如下内容：

-------------------------------------------

Time: 1499976790000 ms

-------------------------------------------

Received 1 flume events.

8、参考文档：http://spark.apache.org/docs/1.3.0/streaming-flume-integration.html