【Spark-core学习之三】 Spark集群搭建 & spark-shell & Master HA

环境
　　虚拟机：VMware 10
　　Linux版本：CentOS-6.5-x86_64
　　客户端：Xshell4
　　FTP：Xftp4
　　jdk1.8
　　scala-2.10.4(依赖jdk1.8)
　　spark-1.6

一、搭建集群
组建方案：
master:PCS101，slave:PCS102、PCS103

搭建方式一：Standalone

步骤一：解压文件 改名

[root@PCS101 src]# tar -zxvf spark-1.6.-bin-hadoop2..tgz -C /usr/local
[root@PCS101 local]# mv spark-1.6.-bin-hadoop2. spark-1.6.

步骤二：修改配置文件
1、slaves.template 设置从节点

[root@PCS101 conf]# cd /usr/local/spark-1.6./conf && mv slaves.template slaves && vi slaves

PCS102
PCS103

2、spark-config.sh 设置java_home

export JAVA_HOME=/usr/local/jdk1..0_65

3、spark-env.sh

[root@PCS101 conf]# mv spark-env.sh.template spark-env.sh && vi spark-env.sh

#SPARK_MASTER_IP:master的ip
export SPARK_MASTER_IP=PCS101

#SPARK_MASTER_PORT:提交任务的端口，默认是7077
export SPARK_MASTER_PORT=

#SPARK_WORKER_CORES：每个worker从节点能够支配的core的个数
export SPARK_WORKER_CORES=

#SPARK_WORKER_MEMORY:每个worker从节点能够支配的内存数
export SPARK_WORKER_MEMORY=3g

#SPARK_MASTER_WEBUI_PORT：sparkwebUI端口 默认8080 或者修改spark-master.sh
export SPARK_MASTER_WEBUI_PORT=

步骤三、分发spark到另外两个节点

[root@PCS101 local]# scp -r /usr/local/spark-1.6. root@PCS102:`pwd`
[root@PCS101 local]# scp -r /usr/local/spark-1.6. root@PCS103:`pwd`

步骤四：启动集群

[root@PCS101 sbin]# /usr/local/spark-1.6./sbin/start-all.sh

步骤五：关闭集群

[root@PCS101 sbin]# /usr/local/spark-1.6./sbin/stop-all.sh

master界面：端口默认8080

job界面，端口是4040：

步骤六：搭建客户端
将spark安装包原封不动的拷贝到一个新的节点上，然后，在新的节点上提交任务即可

提交任务测试：

[root@PCS101 bin]# ./spark-submit
Usage: spark-submit [options] <app jar | python file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]

Options:
  --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
  --deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
                              on one of the worker machines inside the cluster ("cluster")
                              (Default: client).
  --class CLASS_NAME          Your application's main class (for Java / Scala apps).
  --name NAME                 A name of your application.
  --jars JARS                 Comma-separated list of local jars to include on the driver
                              and executor classpaths.
  --packages                  Comma-separated list of maven coordinates of jars to include
                              on the driver and executor classpaths. Will search the local
                              maven repo, then maven central and any additional remote
                              repositories given by --repositories. The format for the
                              coordinates should be groupId:artifactId:version.
  --exclude-packages          Comma-separated list of groupId:artifactId, to exclude while
                              resolving the dependencies provided in --packages to avoid
                              dependency conflicts.
  --repositories              Comma-separated list of additional remote repositories to
                              search for the maven coordinates given with --packages.
  --py-files PY_FILES         Comma-separated list of .zip, .egg, or .py files to place
                              on the PYTHONPATH for Python apps.
  --files FILES               Comma-separated list of files to be placed in the working
                              directory of each executor.

  --conf PROP=VALUE           Arbitrary Spark configuration property.
  --properties-file FILE      Path to a file from which to load extra properties. If not
                              specified, this will look for conf/spark-defaults.conf.

  --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 1024M).
  --driver-java-options       Extra Java options to pass to the driver.
  --driver-library-path       Extra library path entries to pass to the driver.
  --driver-class-path         Extra class path entries to pass to the driver. Note that
                              jars added with --jars are automatically included in the
                              classpath.

  --executor-memory MEM       Memory per executor (e.g. 1000M, 2G) (Default: 1G).

  --proxy-user NAME           User to impersonate when submitting the application.

  --help, -h                  Show this help message and exit
  --verbose, -v               Print additional debug output
  --version,                  Print the version of current Spark

 Spark standalone with cluster deploy mode only:
  --driver-cores NUM          Cores for driver (Default: ).

 Spark standalone or Mesos with cluster deploy mode only:
  --supervise                 If given, restarts the driver on failure.
  --kill SUBMISSION_ID        If given, kills the driver specified.
  --status SUBMISSION_ID      If given, requests the status of the driver specified.

 Spark standalone and Mesos only:
  --total-executor-cores NUM  Total cores for all executors.

 Spark standalone and YARN only:
  --executor-cores NUM        Number of cores per executor. (Default:  in YARN mode,
                              or all available cores on the worker in standalone mode)

 YARN-only:
  --driver-cores NUM          Number of cores used by the driver, only in cluster mode
                              (Default: ).
  --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
  --num-executors NUM         Number of executors to launch (Default: ).
  --archives ARCHIVES         Comma separated list of archives to be extracted into the
                              working directory of each executor.
  --principal PRINCIPAL       Principal to be used to login to KDC, while running on
                              secure HDFS.
  --keytab KEYTAB             The full path to the file that contains the keytab for the
                              principal specified above. This keytab will be copied to
                              the node running the Application Master via the Secure
                              Distributed Cache, for renewing the login tickets and the
                              delegation tokens periodically.
      

(1)三台节点选择任意一个节点提交：
#--master指定master 四种方式：
　　standlone方式：spark://host:port；
　　mesos方式：mesos://host:port；
　　yarn方式：yarn
　　local方式:local
#--class指定任务类名以及所在的jar包路径
#--deploy-mode 提交任务模式：client客户端模式 cluster 集群模式

示例：SparkPi：后面的10000是SparkPi的参数

[root@PCS101 bin]# /usr/local/spark-1.6./bin/spark-submit --master spark://PCS101:7077 --class org.apache.spark.examples.SparkPi ../lib/spark-examples-1.6.0-hadoop2.6.0.jar 10000

(2)使用单独节点作为客户端提交任务：
将PCS101上spark-1.6.0安装目录整个拷贝到新节点PCS104上，然后在PCS104上提交任务，效果是一样的；
PCS104与spark集群无关的，仅仅是上面有spark的提交任务脚本，可以将PCS104当作spark客户端。

搭建方式二：Yarn

注意：如果数据来源于HDFS或者需要使用YARN提交任务，那么spark需要依赖HDFS，否则两者没有联系
(1)步骤一、二、三、六同standalone
(2)在客户端中配置:

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

提交任务：

[root@PCS101 bin]# /usr/local/spark-1.6./bin/spark-submit --master yarn --class org.apache.spark.examples.SparkPi ../lib/spark-examples-1.6.-hadoop2.6.0.jar 

二、spark-shell

SparkShell是Spark自带的一个快速原型开发工具，也可以说是Spark的scala REPL(Read-Eval-Print-Loop),即交互式shell。支持使用scala语言来进行Spark的交互式编程。

1、运行
步骤一：启动standalone集群和HDFS集群（PCS102上HDFS伪分布），之后启动spark-shell

[root@PCS101 bin]# /usr/local/spark-1.6./bin/spark-shell --master spark://PCS101:7077

步骤二：运行wordcount

scala>sc.textFile("hdfs://PCS102:9820/spark/test/wc.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).foreach(println)

2、配置historyServer

2、配置historyServer
(2.1)临时配置，对本次提交的应用程序起作用

[root@PCS101 bin]# /usr/local/spark-1.6./bin/spark-shell --master spark://PCS101:7077 --name myapp1 --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs://PCS102:9820/spark/test

停止程序，在Web UI中Completed Applications对应的ApplicationID中能查看history。

(2.2)spark-default.conf配置文件中配置HistoryServer，对所有提交的Application都起作用
如果想看历史事件日志，可以新搭建一个HistoryServer专门用来看历史应用日志，跟当前的集群没有关系，

这里我们新启动客户端PCS104节点，进入../spark-1.6.0/conf/spark-defaults.conf最后加入:

//开启记录事件日志的功能
spark.eventLog.enabled true
//设置事件日志存储的目录
spark.eventLog.dir hdfs://PCS102:9820/spark/test
//设置HistoryServer加载事件日志的位置  恢复查看
spark.history.fs.logDirectory hdfs://PCS102:9820/spark/test
//日志优化选项,压缩日志
spark.eventLog.compress true

启动HistoryServer：

[root@PCS104 sbin]# /usr/local/spark-1.6./sbin/start-history-server.sh

访问HistoryServer：PCS104:18080,之后所有提交的应用程序运行状况都会被记录。这里的HistoryServer和当前spark集群无关的

三、Master HA

1、Master的高可用可以使用fileSystem(文件系统)和zookeeper（分布式协调服务），一般使用zookeeper方式

2、Master高可用搭建
搭建方案：
Spark集群：PCS101、PCS102、PCS103
主master：PCS101
备master：PCS102

1)在Spark Master节点上配置主Master，配置/usr/local/spark-1.6.0/conf/spark-env.sh

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=PCS101:2181,PCS102:2181,PCS103:2181  -Dspark.deploy.zookeeper.dir=/sparkmaster0409"

2)发送到其他worker节点上

[root@PCS101 conf]# scp spark-env.sh root@PCS102:`pwd`
[root@PCS101 conf]# scp spark-env.sh root@PCS103:`pwd`

3)PCS102配置备用 Master,修改spark-env.sh配置节点上的MasterIP

export SPARK_MASTER_IP=PCS102

4)启动集群之前启动zookeeper集群：
../zkServer.sh start

5)启动spark Standalone集群，启动备用Master

[root@PCS101 sbin]# /usr/local/spark-1.6./sbin/start-all.sh
[root@PCS102 sbin]# /usr/local/spark-1.6./sbin/start-master.sh

6) 打开主Master和备用Master WebUI页面，观察状态。

PCS101:Status:ALIVE
PCS102:Status:STANDBY
kill掉PCS101的master,PCS102的Status:STANDBY变为ALIVE

3. 注意点
　　(1)主备切换过程中不能提交Application。
　　(2)主备切换过程中不影响已经在集群中运行的Application。因为Spark是粗粒度资源调度。

4. 测试验证
提交SparkPi程序，kill主Master观察现象。

./spark-submit
--master spark://PCS101:7077,PCS102:7077
--class org.apache.spark.examples.SparkPi
../lib/spark-examples-1.6.-hadoop2.6.0.jar

参考：
Spark2.3 HA集群的分布式安装

Spark