Spark编译安装和运行

一、环境说明

Mac OSX
Java  1.7.0_71
Spark 

二、编译安装

tar -zxvf spark-.tgz
cd spark-
./sbt/sbt assembly

ps：如果之前执行过编译，需要执行 ./sbt/sbt clean  清理后才能重新编译。

三、运行

adeMacBook-Pro:spark- apple$ ./bin/spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
// :: INFO SecurityManager: Changing view acls to: apple
// :: INFO SecurityManager: Changing modify acls to: apple
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(apple); users with modify permissions: Set(apple)
// :: INFO HttpServer: Starting HTTP Server
// :: INFO Server: jetty-.y.z-SNAPSHOT
// :: INFO AbstractConnector: Started SocketConnector@
// :: INFO Utils: Successfully started service .
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version
      /_/

Using Scala version  (Java HotSpot(TM) -Bit Server VM, Java 1.7.0_71)
Type in expressions to have them evaluated.
Type :help for more information.
// :: INFO SparkContext: Running Spark version
// :: INFO SecurityManager: Changing view acls to: apple
// :: INFO SecurityManager: Changing modify acls to: apple
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(apple); users with modify permissions: Set(apple)
// :: INFO Slf4jLogger: Slf4jLogger started
// :: INFO Remoting: Starting remoting
// :: INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.106:61567]
// :: INFO Utils: Successfully started service .
// :: INFO SparkEnv: Registering MapOutputTracker
// :: INFO SparkEnv: Registering BlockManagerMaster
// :: INFO DiskBlockManager: Created local directory at /-4d54-89f3-8d97bf15205f/blockmgr-b8410cda-aa29---d6155512cd53
// :: INFO MemoryStore: MemoryStore started with capacity 265.4 MB
// :: INFO HttpFileServer: HTTP File server directory -4d54-89f3-8d97bf15205f/httpd-a1838f08-2ccd-42d2--6e91cb6fdfad
// :: INFO HttpServer: Starting HTTP Server
// :: INFO Server: jetty-.y.z-SNAPSHOT
// :: INFO AbstractConnector: Started SocketConnector@
// :: INFO Utils: Successfully started service .
// :: INFO SparkEnv: Registering OutputCommitCoordinator
// :: INFO Server: jetty-.y.z-SNAPSHOT
// :: INFO AbstractConnector: Started SelectChannelConnector@
// :: INFO Utils: Successfully started service .
// :: INFO SparkUI: Started SparkUI at http://192.168.1.106:4040
// :: INFO Executor: Starting executor ID driver on host localhost
// :: INFO Executor: Using REPL class URI: http://192.168.1.106:61566
// :: INFO Utils: Successfully started service .
// :: INFO NettyBlockTransferService: Server created on
// :: INFO BlockManagerMaster: Trying to register BlockManager
// :: INFO BlockManagerMasterEndpoint: Registering block manager localhost: with )
// :: INFO BlockManagerMaster: Registered BlockManager
// :: INFO SparkILoop: Created spark context..
Spark context available as sc.
// :: INFO SparkILoop: Created sql context..
SQL context available as sqlContext.

scala> 

参考：

https://spark.apache.org/docs/latest/

三、使用spark交互模式

. 运行./spark-shell.sh

. scala> val data = Array(, , , , ) //产生data

data: Array[Int] = Array(, , , , )

. scala> val distData = sc.parallelize(data) //将data处理成RDD

distData: spark.RDD[Int] = spark.ParallelCollection@7a0ec850 （显示出的类型为RDD）

. scala> distData.reduce(_+_) //在RDD上进行运算，对data里面元素进行加和

// :: INFO spark.SparkContext: Starting job...

. 最后运行得到

// :: INFO spark.SparkContext: Job finished in 0.076729174 s

res2: Int =