<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>com.jd.ads</groupId>

    <modelVersion>4.0.0</modelVersion>
 
    <groupId>com.jd.ads</groupId>
    <artifactId>fun2spark</artifactId>
    <version>1.0.0</version>
    <packaging>jar</packaging>
    <properties>
        <spark.version>2.0.1</spark.version>
        <proto.path>${project.basedir}/src/main/proto</proto.path>
        <proto.file>*.proto</proto.file>
        <proto.gene>${project.basedir}/src/main/java/gen-java</proto.gene>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.9</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang.modules</groupId>
            <artifactId>scala-xml_2.11</artifactId>
            <version>1.0.2</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.11.2</version>
        </dependency>
 
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
 
        <dependency>
            <groupId>com.github.scopt</groupId>
            <artifactId>scopt_2.10</artifactId>
            <version>3.2.0</version>
        </dependency>
    </dependencies>
 
    <build>
        <resources>
            <resource>
                <directory>src/main/java</directory>
            </resource>
            <resource>
                <directory>src/main/scala</directory>
            </resource>
            <resource>
                <directory>src/main/resources</directory>
            </resource>
        </resources>
        <plugins>
 
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-clean-plugin</artifactId>
                <version>2.4.1</version>
                <configuration>
                    <verbose>true</verbose>
                    <filesets>
                        <fileset>
                            <directory>${proto.gene}</directory>
                        </fileset>
                    </filesets>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-resources-plugin</artifactId>
                <version>2.4.3</version>
                <configuration>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>2.3.1</version>
            </plugin>
        </plugins>
    </build>
    <pluginRepositories>
        <pluginRepository>
            <id>scala</id>
            <name>Scala Tools</name>
            <url>http://scala-tools.org/repo-releases/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </pluginRepository>
    </pluginRepositories>
</project>

    <artifactId>fun2spark</artifactId>

    <version>1.0.0</version>

    <packaging>jar</packaging>

    <properties>

        <spark.version>2.0.1</spark.version>

        <proto.path>${project.basedir}/src/main/proto</proto.path>

        <proto.file>*.proto</proto.file>

        <proto.gene>${project.basedir}/src/main/java/gen-java</proto.gene>

    </properties>

    <dependencies>

        <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-client</artifactId>

            <version>2.2.0</version>

        </dependency>

        <dependency>

            <groupId>commons-codec</groupId>

            <artifactId>commons-codec</artifactId>

            <version>1.9</version>

        </dependency>

        <dependency>

            <groupId>org.scala-lang.modules</groupId>

            <artifactId>scala-xml_2.11</artifactId>

            <version>1.0.2</version>

        </dependency>

        <dependency>

            <groupId>org.scala-lang</groupId>

            <artifactId>scala-library</artifactId>

            <version>2.11.2</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-core_2.11</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-sql_2.11</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-hive_2.11</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-streaming_2.11</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>com.github.scopt</groupId>

            <artifactId>scopt_2.10</artifactId>

            <version>3.2.0</version>

        </dependency>

    </dependencies>

    <build>

        <resources>

            <resource>

                <directory>src/main/java</directory>

            </resource>

            <resource>

                <directory>src/main/scala</directory>

            </resource>

            <resource>

                <directory>src/main/resources</directory>

            </resource>

        </resources>

        <plugins>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-clean-plugin</artifactId>

                <version>2.4.1</version>

                <configuration>

                    <verbose>true</verbose>

                    <filesets>

                        <fileset>

                            <directory>${proto.gene}</directory>

                        </fileset>

                    </filesets>

                </configuration>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-compiler-plugin</artifactId>

                <version>2.3.2</version>

                <configuration>

                    <source>1.6</source>

                    <target>1.6</target>

                    <encoding>UTF-8</encoding>

                </configuration>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-resources-plugin</artifactId>

                <version>2.4.3</version>

                <configuration>

                    <encoding>UTF-8</encoding>

                </configuration>

            </plugin>

            <plugin>

                <groupId>org.scala-tools</groupId>

                <artifactId>maven-scala-plugin</artifactId>

                <version>2.15.2</version>

                <executions>

                    <execution>

                        <goals>

                            <goal>compile</goal>

                            <goal>testCompile</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-jar-plugin</artifactId>

                <version>2.3.1</version>

            </plugin>

        </plugins>

    </build>

    <pluginRepositories>

        <pluginRepository>

            <id>scala</id>

            <name>Scala Tools</name>

            <url>http://scala-tools.org/repo-releases/</url>

            <releases>

                <enabled>true</enabled>

            </releases>

            <snapshots>

                <enabled>false</enabled>

            </snapshots>

        </pluginRepository>

    </pluginRepositories>

</project>

spark2.0配置的更多相关文章

  1. Spark2.0集成Hive操作的相关配置与注意事项

    前言 已完成安装Apache Hive,具体安装步骤请参照,Linux基于Hadoop2.8.0集群安装配置Hive2.1.1及基础操作 补充说明 Hive中metastore(元数据存储)的三种方式 ...

  2. spark2.0.1 安装配置

    1. 官网下载 wget http://d3kbcqa49mib13.cloudfront.net/spark-2.0.1-bin-hadoop2.7.tgz 2. 解压 tar -zxvf spar ...

  3. Ubuntu14.04或16.04下安装JDK1.8+Scala+Hadoop2.7.3+Spark2.0.2

    为了将Hadoop和Spark的安装简单化,今日写下此帖. 首先,要看手头有多少机器,要安装伪分布式的Hadoop+Spark还是完全分布式的,这里分别记录. 1. 伪分布式安装 伪分布式的Hadoo ...

  4. Spark2.0编译

    Spark2.0编译 1 前言 Spark2.0正式版于今天正式发布,本文基于CDH5.0.2的Spark编译. 2 编译步骤 #2.1 下载源码 wget https://github.com/ap ...

  5. CentOS下SparkR安装部署：hadoop2.7.3+spark2.0.0+scale2.11.8+hive2.1.0

    注:之前本人写了一篇SparkR的安装部署文章:SparkR安装部署及数据分析实例,当时SparkR项目还没正式入主Spark,需要自己下载SparkR安装包,但现在spark已经支持R接口,so更新 ...

  6. windows 本地构建hadoop-spark运行环境（hadoop-2.6, spark2.0）

    下载hadoop http://hadoop.apache.org/releases.html --> http://mirrors.tuna.tsinghua.edu.cn/apache/ha ...

  7. 初识Spark2.0之Spark SQL

    内存计算平台spark在今年6月份的时候正式发布了spark2.0,相比上一版本的spark1.6版本,在内存优化,数据组织,流计算等方面都做出了较大的改变,同时更加注重基于DataFrame数据组织 ...

  8. 图文解析Spark2.0核心技术(转载)

    导语 Spark2.0于2016-07-27正式发布,伴随着更简单.更快速.更智慧的新特性,spark 已经逐步替代 hadoop 在大数据中的地位,成为大数据处理的主流标准.本文主要以代码和绘图的方 ...

  9. 在centos7上安装部署hadoop2.7.3和spark2.0.0

    一.安装装备 下载安装包: vmware workstations pro 12 三台centos7.1 mini 虚拟机 网络配置NAT网络如下: 二.创建hadoop用户和hadoop用户组 1. ...

随机推荐

  1. Hibernate中evict方法和clear方法说明

    Hibernate中evict方法和clear方法说明 先创建一个对象,然后调用session.save方法,然后调用evict方法把该对象清除出缓存,最后提交事务.结果报错: Exception i ...

  2. js正则表达式(常用)

    正则表达式(常用) 写法 js写法 var re = new RegExp("a","i"); perl写法 var re = /a/ ; 量词 {n} 正好出 ...

  3. 对​O​p​e​n​C​V​直​方​图​的​数​据​结​构​C​v​H​i​s​t​o​g​r​a​m​的​理​解

    前几天被OpenCV的直方图的数据结构CvHistogram弄得很纠结.上网一搜,也没什么相关的资料.现在有点头绪了,就写点东西,让后面的人好走一些吧. 先来看看CvHistogram的定义: typ ...

  4. Makefile 使用总结

    参考: [1]. Makefile 使用总结. http://www.cnblogs.com/wang_yb/p/3990952.html

  5. Thinkphp源码分析系列（九）–视图view类

    视图类view主要用于页面内容的输出,模板调用等,用在控制器类中,可以使得控制器类把表现和数据结合起来.下面我们来看一下执行流程. 首先,在控制器类中保持着一个view类的对象实例,只要继承自控制器父 ...

  6. python celery + redis

    redis http://debugo.com/python-redis celery http://docs.jinkan.org/docs/celery/getting-started/intro ...

  7. 前端mvc框架backbone.js入门[转]

    原文地址:http://www.cnblogs.com/zhjh256/p/6083618.html 关于backbone.js的优缺点,这里就不详谈了,网上关于这方面的讨论很多了,而且各种框架之所以 ...

  8. MacBook下如何安装mysql-python

    解决方法: 先把之前装的卸载干净:pip uninstall mysql-pythonbrew uninstall mysql-connector-c 现在设置下mysql_config路径:首先修改 ...

  9. High Memory in the Linux Kernel

    This is enabled via the PAE (Physical Address Extension) extension of the PentiumPro processors. PAE ...

  10. Delphi JCL JEDI使用 jclDebug

    开源 https://github.com/project-jedi/jcl jclDebug 下载jcl,还要下载https://github.com/project-jedi/jedi里的2个in ...