Spark1.0 安装

1、下载Scala

wget  http://www.scala-lang.org/files/archive/scala-2.10.3.tgz

tar xvzf scala-2.10.3.tgz -C /usr/local

2、下载Spark

wget http://www.apache.org/dist/spark/spark-1.0.0/spark-1.0.0-bin-hadoop2.tgz

tar -zxvf spark-1.0.0-bin-hadoop2.tgz

3、设置spark的配置文件

mv spark-env.sh.template spark-env.sh

cat >> spark-env.sh

export JAVA_HOME=/usr/java/jdk1.6.0_26

export SCALA_HOME=/usr/local/scala-2.10.3

export HADOOP_HOME=/root/hadoop-2.2.0

SPARK_LOCAL_DIR="/data/spark/tmp"

4、环境变量设置

cat >> /etc/profile

export SCALA_HOME=/usr/local/scala-2.10.3

export SPARK_HOME=/root/spark

export PATH=$SCALA_HOME/bin:$PATH

source /etc/profile

5、启动Hadoop相关进程

start-all.sh

6、启动Spark Worker

sbin/start-all.sh

此时通过jps可知启动了例如以下进程：

21624 Worker

10664 MainGenericRunner

20515 SecondaryNameNode

21057 Master

20311 NameNode

20689 ResourceManager

7、測试

root@master:~/spark/bin# ./run-example  org.apache.spark.examples.SparkPi 1 local

输出：

...

[main] INFO org.apache.spark.SparkContext - Job finished: reduce at SparkPi.scala:39, took 5.0273654 s

Pi is roughly 3.13986

....

附：用Eclipse配置Spark源代码阅读环境http://www.cnblogs.com/lam99v/p/3716207.html