Maven+Eclipse+SparkStreaming+Kafka整合

版本号：

maven3.5.0     scala IDE for Eclipse：版本（4.6.1）    spark-2.1.1-bin-hadoop2.7    kafka_2.11-0.8.2.1   JDK1.8

基础环境：

Maven3.5.0安装与配置+Eclipse应用

Maven下载项目依赖jar包和使用方法

maven中把依赖的JAR包一起打包

MAVEN Scope使用

一、指定JDK为1.8

在pom.xml配置文件中添加以下参数即可：

1. <properties>
2. <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
3. <encoding>UTF-8</encoding>
4. <java.version>1.8</java.version>
5. <maven.compiler.source>1.8</maven.compiler.source>
6. <maven.compiler.target>1.8</maven.compiler.target>
7. </properties>

1. <plugin>
2. <groupId>org.apache.maven.plugins</groupId>
3. <artifactId>maven-compiler-plugin</artifactId>
4. <configuration>
5. <source>1.8</source>
6. <target>1.8</target>
7. </configuration>
8. </plugin>

配置之后的pom.xml文件如下：

1. <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
2. xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3. <modelVersion>4.0.0</modelVersion>
4. <groupId>Test</groupId>
5. <artifactId>test</artifactId>
6. <version>0.0.1-SNAPSHOT</version>
7. <packaging>jar</packaging>
8. <name>test</name>
9. <url>http://maven.apache.org</url>
10. <properties>
11. <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
12. <encoding>UTF-8</encoding>
13. <!-- 配置JDK为1.8 -->
14. <java.version>1.8</java.version>
15. <maven.compiler.source>1.8</maven.compiler.source>
16. <maven.compiler.target>1.8</maven.compiler.target>
17. </properties>
18. <dependencies>
19. <dependency>
20. <groupId>junit</groupId>
21. <artifactId>junit</artifactId>
22. <version>3.8.1</version>
23. <scope>test</scope>
24. </dependency>
25. </dependencies>
26. <build>
27. <plugins>
28. <!-- 配置JDK为1.8 -->
29. <plugin>
30. <groupId>org.apache.maven.plugins</groupId>
31. <artifactId>maven-compiler-plugin</artifactId>
32. <configuration>
33. <source>1.8</source>
34. <target>1.8</target>
35. </configuration>
36. </plugin>
37. <!-- 配置打包依赖包maven-assembly-plugin -->
38. <plugin>
39. <artifactId> maven-assembly-plugin </artifactId>
40. <configuration>
41. <descriptorRefs>
42. <descriptorRef>jar-with-dependencies</descriptorRef>
43. </descriptorRefs>
44. <archive>
45. <manifest>
46. <mainClass></mainClass>
47. </manifest>
48. </archive>
49. </configuration>
50. <executions>
51. <execution>
52. <id>make-assembly</id>
53. <phase>package</phase>
54. <goals>
55. <goal>assembly</goal>
56. </goals>
57. </execution>
58. </executions>
59. </plugin>
60. </plugins>
61. </build>
62. </project>

二、配置Spark依赖包

查看spark-2.1.1-bin-hadoop2.7/jars目录下的jar包版本

到maven远程仓库http://mvnrepository.com中搜索对应jar包即可。

1、配置spark-core_2.11-2.1.1.jar

往pom.xml文件中添加以下配置：

1. <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 -->
2. <dependency>
3. <groupId>org.apache.spark</groupId>
4. <artifactId>spark-core_2.11</artifactId>
5. <version>2.1.1</version>
6. <scope>runtime</scope>
7. </dependency>

为了后面打包时把依赖包也一起打包，需要把<scope>provided</scope>配置成<scope>runtime</scope>。

2、配置spark-streaming_2.11-2.1.1.jar

往pom.xml文件中添加以下配置：

1. <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.11 -->
2. <dependency>
3. <groupId>org.apache.spark</groupId>
4. <artifactId>spark-streaming_2.11</artifactId>
5. <version>2.1.1</version>
6. <scope>runtime</scope>
7. </dependency>

为了后面打包时把依赖包也一起打包，需要把<scope>provided</scope>配置成<scope>runtime</scope>。

三、配置Spark+Kafka

1、配置spark-streaming-kafka-0-8_2.11-2.1.1.jar

往pom.xml文件中添加以下配置：

1. <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-8_2.11 -->
2. <dependency>
3. <groupId>org.apache.spark</groupId>
4. <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
5. <version>2.1.1</version>
6. </dependency>

四、pom.xml完整配置内容

1. <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
2. xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3. <modelVersion>4.0.0</modelVersion>
4. <groupId>Test</groupId>
5. <artifactId>test</artifactId>
6. <version>0.0.1-SNAPSHOT</version>
7. <packaging>jar</packaging>
8. <name>test</name>
9. <url>http://maven.apache.org</url>
10. <properties>
11. <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
12. <encoding>UTF-8</encoding>
13. <!-- 配置JDK为1.8 -->
14. <java.version>1.8</java.version>
15. <maven.compiler.source>1.8</maven.compiler.source>
16. <maven.compiler.target>1.8</maven.compiler.target>
17. </properties>
18. <dependencies>
19. <dependency>
20. <groupId>junit</groupId>
21. <artifactId>junit</artifactId>
22. <version>3.8.1</version>
23. <scope>test</scope>
24. </dependency>
25. <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 -->
26. <dependency>
27. <groupId>org.apache.spark</groupId>
28. <artifactId>spark-core_2.11</artifactId>
29. <version>2.1.1</version>
30. <scope>runtime</scope>
31. </dependency>
32. <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.11 -->
33. <dependency>
34. <groupId>org.apache.spark</groupId>
35. <artifactId>spark-streaming_2.11</artifactId>
36. <version>2.1.1</version>
37. <scope>runtime</scope>
38. </dependency>
39. <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-8_2.11 -->
40. <dependency>
41. <groupId>org.apache.spark</groupId>
42. <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
43. <version>2.1.1</version>
44. </dependency>
45. </dependencies>
46. <build>
47. <plugins>
48. <!-- 配置JDK为1.8 -->
49. <plugin>
50. <groupId>org.apache.maven.plugins</groupId>
51. <artifactId>maven-compiler-plugin</artifactId>
52. <configuration>
53. <source>1.8</source>
54. <target>1.8</target>
55. </configuration>
56. </plugin>
57. <!-- 配置打包依赖包maven-assembly-plugin -->
58. <plugin>
59. <artifactId> maven-assembly-plugin </artifactId>
60. <configuration>
61. <descriptorRefs>
62. <descriptorRef>jar-with-dependencies</descriptorRef>
63. </descriptorRefs>
64. <archive>
65. <manifest>
66. <mainClass></mainClass>
67. </manifest>
68. </archive>
69. </configuration>
70. <executions>
71. <execution>
72. <id>make-assembly</id>
73. <phase>package</phase>
74. <goals>
75. <goal>assembly</goal>
76. </goals>
77. </execution>
78. </executions>
79. </plugin>
80. </plugins>
81. </build>
82. </project>

五、本地开发spark代码上传spark集群服务并运行

JavaDirectKafkaCompare.java

1. package com.spark.main;
2. import java.util.HashMap;
3. import java.util.HashSet;
4. import java.util.Arrays;
5. import java.util.Iterator;
6. import java.util.Map;
7. import java.util.Set;
8. import java.util.regex.Pattern;
9. import scala.Tuple2;
10. import kafka.serializer.StringDecoder;
11. import org.apache.spark.SparkConf;
12. import org.apache.spark.api.java.function.*;
13. import org.apache.spark.streaming.api.java.*;
14. import org.apache.spark.streaming.kafka.KafkaUtils;
15. import org.apache.spark.streaming.Durations;
16. public class JavaDirectKafkaCompare {
17. public static void main(String[] args) throws Exception {
18. /**
19. * setMaster("local[2]")，至少要指定两个线程，一条用于用于接收消息，一条线程用于处理消息
20. *  Durations.seconds(2)每两秒读取一次kafka
21. */
22. SparkConf sparkConf = new SparkConf().setAppName("JavaDirectKafkaWordCount").setMaster("local[2]");
23. JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(2));
24. /**
25. * checkpoint("hdfs://192.168.168.200:9000/checkpoint")防止数据丢包
26. */
27. jssc.checkpoint("hdfs://192.168.168.200:9000/checkpoint");
28. /**
29. * 配置连接kafka的相关参数
30. */
31. Set<String> topicsSet = new HashSet<>(Arrays.asList("test"));
32. Map<String, String> kafkaParams = new HashMap<>();
33. kafkaParams.put("metadata.broker.list", "192.168.168.200:9092");
34. // Create direct kafka stream with brokers and topics
35. JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(
36. jssc,
37. String.class,
38. String.class,
39. StringDecoder.class,
40. StringDecoder.class,
41. kafkaParams,
42. topicsSet
43. );
44. // Get the lines, split them into words, count the words and print
45. /**
46. * _2()获取第二个对象的值
47. */
48. JavaDStream<String> lines = messages.map(new Function<Tuple2<String, String>, String>() {
49. @Override
50. public String call(Tuple2<String, String> tuple2) {
51. return tuple2._2();
52. }
53. });
54. String sfzh = "432922196105276721";
55. JavaDStream<String> wordCounts = lines.filter(new Function<String, Boolean>(){
56. @Override
57. public Boolean call(String s) throws Exception {
58. // TODO Auto-generated method stub
59. /**
60. * 通过身份证号筛选出相关数据
61. */
62. if(s.contains(sfzh)){
63. System.out.println("比对出来的结果：" + s);
64. return true;
65. }
66. return false;
67. }
68. });
69. wordCounts.print();
70. // Start the computation
71. jssc.start();
72. jssc.awaitTermination();
73. }
74. }

右键Run As ------>Maven install，运行成功之后，会在target目录生成一个test-0.0.1-SNAPSHOT-jar-with-dependencies.jar，把该jar包复制到LInux集群环境下的SPARK_HOME/myApp目录下：

执行命令：

1. cd /usr/local/spark/spark-2.1.1-bin-hadoop2.7;
2. bin/spark-submit --class "com.spark.main.JavaDirectKafkaCompare" --master local[4] myApp/test-0.0.1-SNAPSHOT-jar-with-dependencies.jar;

六、附上离线Maven仓库

下载地址：  链接：http://pan.baidu.com/s/1eS7Ywme 密码：y3qz