在Win7虚拟机下搭建Hadoop2.6.0+Spark1.4.0单机环境

Hadoop的安装和配置可以参考我之前的文章：在Win7虚拟机下搭建Hadoop2.6.0伪分布式环境。

本篇介绍如何在Hadoop2.6.0基础上搭建spark1.4.0单机环境。

1. 软件准备

scala-2.11.7.tgz

spark-1.4.0-bin-hadoop2.6.tgz

都可以从官网下载。

2. scala安装和配置

scala-2.11.7.tgz解压缩即可。我解压缩到目录/home/vm/tools/scala，之后配置~/.bash_profile环境变量。

#scala

export SCALA_HOME=/home/vm/tools/scala

export PATH=$SCALA_HOME/bin:$PATH

使用source ~/.bash_profile生效。

验证scala安装是否成功：

交互式使用scala：

3. spark安装和配置

解压缩spark-1.4.0-bin-hadoop2.6.tgz到/home/vm/tools/spark目录，之后配置~/.bash_profile环境变量。

#spark

export SPARK_HOME=/home/vm/tools/spark

export PATH=$SPARK_HOME/bin:$PATH

修改$SPARK_HOME/conf/spark-env.sh

export SPARK_HOME=/home/vm/tools/spark

export SCALA_HOME=/home/vm/tools/scala

export JAVA_HOME=/home/vm/tools/jdk

export SPARK_MASTER_IP=192.168.62.129

export SPARK_WORKER_MEMORY=512m

修改$SPARK_HOME/conf/spark-defaults.conf

spark.master spark://192.168.62.129:7077

spark.serializer org.apache.spark.serializer.KryoSerializer

修改$SPARK_HOME/conf/spark-defaults.conf

192.168.62.129 这是我测试机器的IP地址

启动spark

cd /home/vm/tools/spark/sbin

sh start-all.sh

测试Spark是否安装成功

cd $SPARK_HOME/bin/

./run-example SparkPi

SparkPi的执行日志：

 vm@ubuntu:~/tools/spark/bin$ ./run-example SparkPi

 Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

 // :: INFO SparkContext: Running Spark version 1.4.

 // :: WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

 // :: INFO SecurityManager: Changing view acls to: vm

 // :: INFO SecurityManager: Changing modify acls to: vm

 // :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(vm); users with modify permissions: Set(vm)

 // :: INFO Slf4jLogger: Slf4jLogger started

 // :: INFO Remoting: Starting remoting

 // :: INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.62.129:34337]

 // :: INFO Utils: Successfully started service 'sparkDriver' on port .

 // :: INFO SparkEnv: Registering MapOutputTracker

 // :: INFO SparkEnv: Registering BlockManagerMaster

 // :: INFO DiskBlockManager: Created local directory at /tmp/spark--e4c4-4dcc-8c16-f46fce5e657d/blockmgr-be03da6d-31fe-43dd-959c-6cfa4307b269

 // :: INFO MemoryStore: MemoryStore started with capacity 267.3 MB

 // :: INFO HttpFileServer: HTTP File server directory is /tmp/spark--e4c4-4dcc-8c16-f46fce5e657d/httpd-fdc26a4d-c0b6-4fc9-9dee-fb085191ee5a

 // :: INFO HttpServer: Starting HTTP Server

 // :: INFO Utils: Successfully started service 'HTTP file server' on port .

 // :: INFO SparkEnv: Registering OutputCommitCoordinator

 // :: INFO Utils: Successfully started service 'SparkUI' on port .

 // :: INFO SparkUI: Started SparkUI at http://192.168.62.129:4040

 // :: INFO SparkContext: Added JAR file:/home/vm/tools/spark/lib/spark-examples-1.4.-hadoop2.6.0.jar at http://192.168.62.129:56880/jars/spark-examples-1.4.0-hadoop2.6.0.jar with timestamp 1438099360726

 // :: INFO Executor: Starting executor ID driver on host localhost

 // :: INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port .

 // :: INFO NettyBlockTransferService: Server created on 

 // :: INFO BlockManagerMaster: Trying to register BlockManager

 // :: INFO BlockManagerMasterEndpoint: Registering block manager localhost: with 267.3 MB RAM, BlockManagerId(driver, localhost, )

 // :: INFO BlockManagerMaster: Registered BlockManager

 // :: INFO SparkContext: Starting job: reduce at SparkPi.scala:

 // :: INFO DAGScheduler: Got job  (reduce at SparkPi.scala:) with  output partitions (allowLocal=false)

 // :: INFO DAGScheduler: Final stage: ResultStage (reduce at SparkPi.scala:)

 // :: INFO DAGScheduler: Parents of final stage: List()

 // :: INFO DAGScheduler: Missing parents: List()

 // :: INFO DAGScheduler: Submitting ResultStage  (MapPartitionsRDD[] at map at SparkPi.scala:), which has no missing parents

 // :: INFO MemoryStore: ensureFreeSpace() called with curMem=, maxMem=

 // :: INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1888.0 B, free 267.3 MB)

 // :: INFO MemoryStore: ensureFreeSpace() called with curMem=, maxMem=

 // :: INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1186.0 B, free 267.3 MB)

 // :: INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost: (size: 1186.0 B, free: 267.3 MB)

 // :: INFO SparkContext: Created broadcast  from broadcast at DAGScheduler.scala:

 // :: INFO DAGScheduler: Submitting  missing tasks from ResultStage  (MapPartitionsRDD[] at map at SparkPi.scala:)

 // :: INFO TaskSchedulerImpl: Adding task set 0.0 with  tasks

 // :: INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID , localhost, PROCESS_LOCAL,  bytes)

 // :: INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID , localhost, PROCESS_LOCAL,  bytes)

 // :: INFO Executor: Running task 1.0 in stage 0.0 (TID )

 // :: INFO Executor: Running task 0.0 in stage 0.0 (TID )

 // :: INFO Executor: Fetching http://192.168.62.129:56880/jars/spark-examples-1.4.0-hadoop2.6.0.jar with timestamp 1438099360726

 // :: INFO Utils: Fetching http://192.168.62.129:56880/jars/spark-examples-1.4.0-hadoop2.6.0.jar to /tmp/spark-78277899-e4c4-4dcc-8c16-f46fce5e657d/userFiles-27c8dd76-e417-4d13-9bfd-a978cbbaacd1/fetchFileTemp5302506499464337647.tmp

 // :: INFO Executor: Adding file:/tmp/spark--e4c4-4dcc-8c16-f46fce5e657d/userFiles-27c8dd76-e417-4d13-9bfd-a978cbbaacd1/spark-examples-1.4.-hadoop2.6.0.jar to class loader

 // :: INFO Executor: Finished task 1.0 in stage 0.0 (TID ).  bytes result sent to driver

 // :: INFO Executor: Finished task 0.0 in stage 0.0 (TID ).  bytes result sent to driver

 // :: INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID ) in  ms on localhost (/)

 // :: INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID ) in  ms on localhost (/)

 // :: INFO DAGScheduler: ResultStage  (reduce at SparkPi.scala:) finished in 2.817 s

 // :: INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool

 // :: INFO DAGScheduler: Job  finished: reduce at SparkPi.scala:, took 4.244145 s

 Pi is roughly 3.14622

 // :: INFO SparkUI: Stopped Spark web UI at http://192.168.62.129:4040

 // :: INFO DAGScheduler: Stopping DAGScheduler

 // :: INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

 // :: INFO Utils: path = /tmp/spark--e4c4-4dcc-8c16-f46fce5e657d/blockmgr-be03da6d-31fe-43dd-959c-6cfa4307b269, already present as root for deletion.

 // :: INFO MemoryStore: MemoryStore cleared

 // :: INFO BlockManager: BlockManager stopped

 // :: INFO BlockManagerMaster: BlockManagerMaster stopped

 // :: INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

 // :: INFO SparkContext: Successfully stopped SparkContext

 // :: INFO Utils: Shutdown hook called

 // :: INFO Utils: Deleting directory /tmp/spark--e4c4-4dcc-8c16-f46fce5e657d

在浏览器中打开地址 http://192.168.62.129:8080 可以查看spark集群和任务基本情况：

4. spark-shell工具

在/home/vm/tools/spark/bin下执行./spark-shell，即可进入spark-shell交互界面。通过spark-shell可以进行一些调试工作。

 vm@ubuntu:~/tools/spark/bin$ ./spark-shell

 log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).

 log4j:WARN Please initialize the log4j system properly.

 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

 // :: INFO SecurityManager: Changing view acls to: vm

 // :: INFO SecurityManager: Changing modify acls to: vm

 // :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(vm); users with modify permissions: Set(vm)

 // :: INFO HttpServer: Starting HTTP Server

 // :: INFO Utils: Successfully started service 'HTTP class server' on port .

 Welcome to

 ____ __

 / __/__ ___ _____/ /__

 _\ \/ _ \/ _ `/ __/ '_/

 /___/ .__/\_,_/_/ /_/\_\ version 1.4.

 /_/

 Using Scala version 2.10. (Java HotSpot(TM) -Bit Server VM, Java 1.7.0_80)

 Type in expressions to have them evaluated.

 Type :help for more information.

 // :: INFO SparkContext: Running Spark version 1.4.

 // :: INFO SecurityManager: Changing view acls to: vm

 // :: INFO SecurityManager: Changing modify acls to: vm

 // :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(vm); users with modify permissions: Set(vm)

 // :: INFO Slf4jLogger: Slf4jLogger started

 // :: INFO Remoting: Starting remoting

 // :: INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.62.129:59312]

 // :: INFO Utils: Successfully started service 'sparkDriver' on port .

 // :: INFO SparkEnv: Registering MapOutputTracker

 // :: INFO SparkEnv: Registering BlockManagerMaster

 // :: INFO DiskBlockManager: Created local directory at /tmp/spark-621ebed4-8bd8-4e87-9ea5-08b5c7f05e98/blockmgr-a12211dd-e0ba--999c-6249b9c44d8a

 // :: INFO MemoryStore: MemoryStore started with capacity 267.3 MB

 // :: INFO HttpFileServer: HTTP File server directory is /tmp/spark-621ebed4-8bd8-4e87-9ea5-08b5c7f05e98/httpd-8512d909-5a81--8fbd-2b2ed741ae26

 // :: INFO HttpServer: Starting HTTP Server

 // :: INFO Utils: Successfully started service 'HTTP file server' on port .

 // :: INFO SparkEnv: Registering OutputCommitCoordinator

 // :: INFO Utils: Successfully started service 'SparkUI' on port .

 // :: INFO SparkUI: Started SparkUI at http://192.168.62.129:4040

 // :: INFO Executor: Starting executor ID driver on host localhost

 // :: INFO Executor: Using REPL class URI: http://192.168.62.129:56464

 // :: INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port .

 // :: INFO NettyBlockTransferService: Server created on 

 // :: INFO BlockManagerMaster: Trying to register BlockManager

 // :: INFO BlockManagerMasterEndpoint: Registering block manager localhost: with 267.3 MB RAM, BlockManagerId(driver, localhost, )

 // :: INFO BlockManagerMaster: Registered BlockManager

 // :: INFO SparkILoop: Created spark context..

 Spark context available as sc.

 // :: INFO HiveContext: Initializing execution hive, version 0.13.

 // :: INFO HiveMetaStore: : Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

 // :: INFO ObjectStore: ObjectStore, initialize called

 // :: INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored

 // :: INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored

 // :: WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

 // :: WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

 // :: INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"

 // :: INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line , column . Encountered: "@" (), after : "".

 // :: INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

 // :: INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

 // :: INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

 // :: INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

 // :: INFO ObjectStore: Initialized ObjectStore

 // :: WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa

 // :: INFO HiveMetaStore: Added admin role in metastore

 // :: INFO HiveMetaStore: Added public role in metastore

 // :: INFO HiveMetaStore: No user is added in admin role, since config is empty

 // :: INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.

 // :: INFO SparkILoop: Created sql context (with Hive support)..

 SQL context available as sqlContext.

 scala>

下一篇将介绍分别用eclipse和IDEA搭建spark开发环境。