Spark编译的三种方式

有三种编译方式，此文采用make-distribution.sh编译

其余两种为maven 和SBT编译

1.配置jdk 配置maven 配置scala

2.修改spark下make-distribution.sh中的配置  减少下载时间

查看下载地址：http://archive.cloudera.com/cdh5/cdh/5/
hadoop-2.6.0-cdh5.4.0-src.tar.gz

配置nameserver  1.sudo vi /etc/resolv.conf  添加
nameserver 8.8.8.8
nameserver 8.8.4.4

maven 镜像源
    <mirrors>  
        <mirror>  
            <id>alimaven</id>  
            <mirrorOf>central</mirrorOf>  
            <name>aliyun maven</name>  
            <url>http://maven.aliyun.com/nexus/content/groups/public/</url>  
        </mirror>  
      
        <mirror>  
            <id>ui</id>  
            <mirrorOf>central</mirrorOf>  
            <name>Human Readable Name for this Mirror.</name>  
            <url>http://uk.maven.org/maven2/</url>  
        </mirror>  
      
        <mirror>  
            <id>jboss-public-repository-group</id>  
            <mirrorOf>central</mirrorOf>  
            <name>JBoss Public Repository Group</name>  
            <url>http://repository.jboss.org/nexus/content/groups/public</url>  
        </mirror>  
    </mirrors>

编译命令
./make-distribution.sh  --tgz -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive-0.13.1 -Phive-thriftserver