spark-2.4.0-hadoop2.7-高可用(HA)安装部署

1. 主机规划

主机名称	IP地址	操作系统	部署软件	运行进程	备注
mini01	172.16.1.11【内网】 10.0.0.11  【外网】	CentOS 7.5	Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、kafka_2.11-2.0.0、spark-2.4.0-hadoop2.7【主】	QuorumPeerMain、
mini02	172.16.1.12【内网】 10.0.0.12  【外网】	CentOS 7.5	Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、kafka_2.11-2.0.0、spark-2.4.0-hadoop2.7【主】	QuorumPeerMain、
mini03	172.16.1.13【内网】 10.0.0.13  【外网】	CentOS 7.5	Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、kafka_2.11-2.0.0、spark-2.4.0-hadoop2.7	QuorumPeerMain、
mini04	172.16.1.14【内网】 10.0.0.14  【外网】	CentOS 7.5	Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、spark-2.4.0-hadoop2.7	QuorumPeerMain、
mini05	172.16.1.15【内网】 10.0.0.15  【外网】	CentOS 7.5	Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、spark-2.4.0-hadoop2.7	QuorumPeerMain、

说明

借助zookeeper，并且启动至少两个Master节点来实现高可靠。

2. 免密码登录

实现mini01、mini02到mini01、mini02、mini03、mini04、mini05通过秘钥免密码登录。

参见文章：Hadoop2.7.6_01_部署

3. Jdk【java8】

参见文章：Hadoop2.7.6_01_部署

4. Zookeeper部署

参见文章：zookeeper-02 部署

并启动zookeeper服务

5. Spark部署步骤

5.1. Spark安装

 [yun@mini01 software]$ pwd
 /app/software
 [yun@mini01 software]$ ll
 total
 -rw-r--r--   yun yun  Nov  : spark-2.4.-bin-hadoop2..tgz
 [yun@mini01 software]$ tar xf spark-2.4.-bin-hadoop2..tgz
 [yun@mini01 software]$ mv spark-2.4.-bin-hadoop2. /app/
 [yun@mini01 software]$ cd /app/
 [yun@mini01 ~]$ ln -s spark-2.4.-bin-hadoop2./ spark
 [yun@mini01 ~]$ ll -d spark-*
 drwxr-xr-x  yun yun  Oct  : spark-2.4.-bin-hadoop2.
 lrwxrwxrwx   yun yun   Nov  : spark -> spark-2.4.-bin-hadoop2./

5.2. 环境变量修改

　　根据规划，该环境变量的修改包括mini01、mini02、mini03、mini04、mini05。

 # 需要root权限去添加环境变量
 [root@mini01 ~]# tail /etc/profile
 ………………
 # spark环境变量
 export SPARK_HOME="/app/spark"
 export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

 [root@mini01 ~]# logout
 [yun@mini01 conf]$ source /etc/profile  # 重新加载该环境变量

5.3. 配置修改

 [yun@mini01 conf]$ pwd
 /app/spark/conf
 [yun@mini01 conf]$ cp -a spark-env.sh.template spark-env.sh
 [yun@mini01 conf]$ tail spark-env.sh  # 修改环境变量配置
 # Options for native BLAS, like Intel MKL, OpenBLAS, and so on.
 # You might get better performance to enable these options if using native BLAS (see SPARK-).
 # - MKL_NUM_THREADS=        Disable multi-threading of Intel MKL
 # - OPENBLAS_NUM_THREADS=   Disable multi-threading of OpenBLAS

 # 添加配置如下
 # 配置JAVA_HOME
 export JAVA_HOME=/app/jdk
 # -Dspark.deploy.recoverMode=ZOOKEEPER #代表发生故障使用zookeeper服务
 # -Dspark.depoly.zookeeper.url=mini01:,mini02:,mini03:,mini04:,mini05: #zookeeper的连接信息
 # -Dspark.deploy.zookeeper.dir=/app/zookeeper/spark #spark要在zookeeper上写数据时的保存目录
 export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=mini01:2181,mini02:2181,mini03:2181,mini04:2181,mini05:2181 -Dspark.deploy.zookeeper.dir=/spark"
 # 每一个Worker最多可以使用的内存，我的虚拟机就2g
 # 真实服务器如果有128G，你可以设置为100G
 # 所以这里设置为1024m或1g
 export SPARK_WORKER_MEMORY=1024m
 # 每一个Worker最多可以使用的cpu core的个数，我虚拟机就一个...
 # 真实服务器如果有32个，你可以设置为32个
 export SPARK_WORKER_CORES=
 # 提交Application的端口，默认就是这个，万一要改呢，改这里
 export SPARK_MASTER_PORT=

 [yun@mini01 conf]$ pwd
 /app/spark /conf
 [yun@mini01 conf]$ cp -a slaves.template slaves
 [yun@mini01 conf]$ tail slaves  # 修改slaves 配置
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #

 # A Spark Worker will be started on each of the machines listed below.
 mini03
 mini04
 mini05

配置说明

# -Dspark.deploy.zookeeper.dir=/app/zookeeper/spark   # spark要在zookeeper上写数据时的保存目录

 [yun@mini05 ~]$ zkCli.sh  # 进入zookeeper命令行  【在spark启动后查看】
 [zk: localhost:(CONNECTED) ] ls /   # 其中的 /spark 就是 我们在spark-env.sh中的配置
 [cluster, brokers, zookeeper, yarn-leader-election, hadoop-ha, admin, isr_change_notification, log_dir_event_notification, controller_epoch, spark, consumers, latest_producer_id_block, config, hbase]
 [zk: localhost:(CONNECTED) ] ls /spark
 [leader_election, master_status]
 [zk: localhost:(CONNECTED) ] ls /spark/master_status
 [worker_worker--172.16.1.13-, worker_worker--172.16.1.14-, worker_worker--172.16.1.15-]
 [zk: localhost:(CONNECTED) ] ls /spark/leader_election
 [_c_6c6d0c36---a05c-9414a78d79e2-latch-, _c_04ceffff-b763-454a-b3f1-7fb56f56fa84-latch-]

5.4. 分发到其他机器

　　分发到mini02、mini03、mini04和mini05

　　其中mini01和mini02作为master

 [yun@mini01 ~]$ scp -pr spark-2.4.-bin-hadoop2./ yun@mini02:/app  # 拷贝到mini02
 [yun@mini01 ~]$ scp -pr spark-2.4.-bin-hadoop2./ yun@mini03:/app  # 拷贝到mini03
 [yun@mini01 ~]$ scp -pr spark-2.4.-bin-hadoop2./ yun@mini04:/app  # 拷贝到mini04
 [yun@mini01 ~]$ scp -pr spark-2.4.-bin-hadoop2./ yun@mini05:/app  # 拷贝到mini05

在mini02、mini03、mini04和mini05上操作

 [yun@mini04 ~]$ pwd
 /app
 [yun@mini04 ~]$ ll -d spark-2.4.-bin-hadoop2.
 drwxr-xr-x  yun yun  Oct  : spark-2.4.-bin-hadoop2.
 [yun@mini04 ~]$ ln -s spark-2.4.-bin-hadoop2./ spark
 [yun@mini04 ~]$ ll -d spark-*
 drwxr-xr-x  yun yun  Oct  : spark-2.4.-bin-hadoop2.
 lrwxrwxrwx   yun yun   Nov  : spark -> spark-2.4.-bin-hadoop2./

5.5. 启动spark

5.5.1. 在mini01上操作

 [yun@mini01 sbin]$ pwd
 /app/spark/sbin
 [yun@mini01 sbin]$ ./start-all.sh  # 关闭使用 stop-all.sh 脚本
 [yun@mini01 sbin]$ ./start-all.sh
 starting org.apache.spark.deploy.master.Master, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.master.Master--mini01.out
 mini03: starting org.apache.spark.deploy.worker.Worker, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.worker.Worker--mini03.out
 mini04: starting org.apache.spark.deploy.worker.Worker, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.worker.Worker--mini04.out
 mini05: starting org.apache.spark.deploy.worker.Worker, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.worker.Worker--mini05.out
 [yun@mini01 ~]$
 [yun@mini01 ~]$ jps  # 查看进程状态
  QuorumPeerMain
  Jps
  Master

5.5.2. 在mini02上操作

 [yun@mini02 sbin]$ pwd
 /app/spark/sbin
 [yun@mini02 sbin]$ ./start-master.sh
 starting org.apache.spark.deploy.master.Master, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.master.Master--mini02.out
 [yun@mini02 sbin]$ jps  # 查看进程状态
  Master
  Jps
  QuorumPeerMain

5.5.3. mini03进程查看

 [yun@mini03 ~]$ jps
  Jps
  QuorumPeerMain
  Worker

5.5.4. mini04进程查看

 [yun@mini04 ~]$ jps
  Jps
  Worker
  QuorumPeerMain

5.5.5. mini05进程查看

 [yun@mini05 ~]$ jps
  Jps
  Worker
  QuorumPeerMain

5.6. 浏览器访问

 http://mini01:8080/    

 http://mini02:8080/    

说明

如果我们停了mini01的spark master，稍等一会儿可见mini02的master状态从standby变为了alive。

此时再启动mini01的master，可见mini01的master状态是standby。