windows 本地构建hadoop-spark运行环境（hadoop-2.6, spark2.0）

1. 下载hadoop
1. http://hadoop.apache.org/releases.html --> http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.6.5/hadoop-2.6.5.tar.gz
2. 安装hadoop，配置HADOOP_HOME， 把${HADOOP_HOME}/bin放到path
2. 下载spark
1. http://spark.apache.org/downloads.html --> https://d3kbcqa49mib13.cloudfront.net/spark-2.0.2-bin-hadoop2.6.tgz 注意与hadoop版本匹配
2. 安装，配置SPARK_HOME，把${SPARK_HOME}/bin放到path
3. 在运行spark程序时，会报找不到 winutils.exe
1. 下载 https://github.com/srccodes/hadoop-common-2.2.0-bin.git 放到${HADOOP_HOME}/bin下
4. 运行时设置本地运行即可
5. spark样例：

 

LocalSparkContext.scala

 

1. import org.apache.spark.{SparkConf, SparkContext}
2. import org.scalatest._
3. trait LocalSparkContext extends BeforeAndAfterAll {
4. self: Suite =>
5. @transient var sc: SparkContext = _
6. override def beforeAll() {
7. val conf = new SparkConf()
8. .setMaster("local[2]")
9. .setAppName("test")
10. sc = new SparkContext(conf)
11. }
12. override def afterAll() {
13. if (sc != null) {
14. sc.stop()
15. }
16. }
17. }

 

SparkWCSuit.scala

1. import org.apache.spark.rdd.RDD
2. import org.apache.spark.sql.{Row, SQLContext}
3. import org.apache.spark.util.LongAccumulator
4. import org.scalatest.FunSuite
5. import tool.LocalSparkContext
6. import algos.{MergedPCtr, PCtrUtils}
7. class SparkWCSuit extends FunSuite with LocalSparkContext {
8. //rdd wordCount
9. test("test rdd wc") {
10. sc.setLogLevel("ERROR")
11. val rdd = sc.makeRDD(Seq("a", "b", "b"))
12. val res = rdd.map((_, 1)).reduceByKey(_ + _).collect().sorted
13. assert(res === Array(("a", 1), ("b", 2)))
14. }
15. }

 

build.sbt

1. name := "doc_rank"
2. version := "1.0"
3. scalaVersion := "2.10.5"
4. libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.0.2"
5. libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "2.0.2"
6. libraryDependencies += "commons-cli" % "commons-cli" % "1.2"
7. libraryDependencies ++= Seq(
8. "org.scalanlp" %% "breeze" % "0.11.2",
9. "org.scalanlp" %% "breeze-natives" % "0.11.2",
10. "org.scalanlp" %% "breeze-viz" % "0.11.2"
11. )
12. libraryDependencies ++= Seq(
13. "org.apache.hadoop" % "hadoop-core" % "2.6.0-mr1-cdh5.4.4",
14. "org.apache.hbase" % "hbase-client" % "1.0.0-cdh5.4.4",
15. "org.apache.hbase" % "hbase-common" % "1.0.0-cdh5.4.4",
16. "org.apache.hbase" % "hbase-server" % "1.0.0-cdh5.4.4",
17. "org.apache.hbase" % "hbase-protocol" % "1.0.0-cdh5.4.4"
18. )
19. resolvers += "Akka Repository" at "http://repo.akka.io/releases/";
20. resolvers += "cloudera-repo-releases" at "https://repository.cloudera.com/artifactory/repo/";
21. resolvers ++= Seq(
22. "Sonatype Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots/";,
23. "Sonatype Releases" at "https://oss.sonatype.org/content/repositories/releases/";
24. )

 

 

 

1. hadoop样例

        

目录结构：

src/

├── main

│   ├── java

│   │   ├── io

│   │   │   └── longwind

│   │   │       └── mapreduce

│   │   │           ├── main

│   │   │           │   └── Main.java

│   │   │           ├── mapreduce

│   │   │           │   └── InfoidUniquer.java

│   │   │           └── utils

│   │   │               ├── Constant.java

│   │   │               └── HadoopUtils.java

│   │   └── org

│   │       └── apache

│   │           └── hadoop

│   │               ├── io

│   │               │   └── nativeio

│   │               │       └── NativeIO.java

│   │               └── mapred

│   │                   ├── ClientCache.java

│   │                   ├── ClientServiceDelegate.java

│   │                   ├── NotRunningJob.java

│   │                   ├── ResourceMgrDelegate.java

│   │                   ├── YarnClientProtocolProvider.java

│   │                   └── YARNRunner.java

│   └── resources

│       └── log4j.properties

└── test

    ├── java

    │   └── test

    └── resources

        └── log4j.properties

 

pom.xml中关键依赖

<dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-common</artifactId>

<version>2.6.0-cdh5.4.4</version>

</dependency>

 

<dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-mapreduce-client-core</artifactId>

<version>2.6.0-cdh5.4.4</version>

</dependency>

 

<dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-mapreduce-client-common</artifactId>

<version>2.6.0-cdh5.4.4</version>

</dependency>

 

代码方面：

上面目录结构显示的org.apache.hadoop.* 那些是从hadoop源码包里拷出来的，注意是2.6.0-cdh5.4.4版本的

程序运行起来报错access0，如果是NativeIO.java 那应该是权限问题，需要手动修改NativeIO.java 中的

 

public static boolean access(String path, AccessRight desiredAccess)throws IOException {

    return true;//修改后

    //return access0(path, desiredAccess.accessRight());//修改前

}

---

这样，就能在windows本地，轻松进行hadoop, spark开发调试了，顺便吐槽一下mrunit不是很给力，问题一般是版本，包冲突，权限。

 

参考：

1. 平野大荒 http://www.cnblogs.com/tq03/p/5101916.html --windows上的mapreduce运行环境
2. 在前进的路上 http://blog.csdn.net/congcong68/article/details/42043093 -- access0 问题解决
3. xuweimdm http://blog.csdn.net/u011513853/article/details/52865076 -- spark在windows上