大数据高可用集群环境安装与配置（09）—

1. 获取spark下载链接

登录官网：http://spark.apache.org/downloads.html 选择要下载的版本

2. 执行命令下载并安装

cd /usr/local/src/

wget http://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz

tar -zxvf spark-2.4.-bin-hadoop2..tgz

mv spark-2.4.-bin-hadoop2. /usr/local/spark

cd /usr/local/spark/conf

mv spark-env.sh.template spark-env.sh

3. 修改spark-env.sh配置

vi spark-env.sh

在尾部添加下面配置，绑定hadoop的配置文件路径

export JAVA_HOME=/usr/local/java/jdk

export HADOOP_CONF_DIR=/usr/local/hadoop/etc/Hadoop

export SPARK_HOME=/usr/local/spark

export SPARK_MASTER_PORT=

# 非高可用集群配置

# export SPARK_MASTER_IP=master

# 高可用集群配置

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=master:2181,master-backup:2181 -Dspark.deploy.zookeeper.dir=/spark"

4. 添加log4j.properties配置

vi log4j.properties

添加下面配置

# Set everything to be logged to the console

log4j.rootCategory=WARN, console

log4j.appender.console=org.apache.log4j.ConsoleAppender

log4j.appender.console.target=System.err

log4j.appender.console.layout=org.apache.log4j.PatternLayout

log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the

# log level for this class is used to overwrite the root logger's log level, so that

# the user can have different defaults for the shell and regular Spark apps.

log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose

log4j.logger.org.spark_project.jetty=WARN

log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR

log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO

log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

log4j.logger.org.apache.parquet=ERROR

log4j.logger.parquet=ERROR

# SPARK-: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support

log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL

log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

5. 修改slaves配置

mv slaves.template slaves

vi slaves

删除里面的localhost，添加下面配置

node1

node2

node3

6. 指定spark的主节点

mv spark-defaults.conf.template spark-defaults.conf

vi spark-defaults.conf

添加下面配置

spark.master spark://master:7077,master-backup:7077

7. 修改服务器系统环境变量

所有服务器都需要按要求修改配置

vi /etc/profile

在尾部添加下面配置

export SPARK_HOME=/usr/local/spark/

export PATH=$PATH:$SPARK_HOME/bin

# 这里根据具体需要进行修改，如果你运行的是python2版本的程序，则不需要修改，python3的话后面需要安装相关环境

export PYSPARK_PYTHON=/usr/local/bin/python3

保存退出后，运行命令，让配置马上生效

source /etc/profile

8. 安装插件，配置pyspark访问hbase

拷贝spark访问hbase所需要的jar到spark/jar引用文件夹

cp /usr/local/hbase/lib/hbase-*.jar /usr/local/spark/jars/

配置spark访问Phoenix

# 复制phoenix客户端插件到spark的jars目录下

cp phoenix-5.0.-HBase-2.0-client.jar /usr/local/spark/jars/

9. 将spark同步到其他服务器上

rsync -avz /usr/local/spark/ master-backup:/usr/local/spark/

rsync -avz /usr/local/spark/ node1:/usr/local/spark/

rsync -avz /usr/local/spark/ node2:/usr/local/spark/

rsync -avz /usr/local/spark/ node3:/usr/local/spark/

10. 启动spark

重启hbase服务

/usr/local/hbase/bin/stop-hbase.sh

/usr/local/hbase/bin/start-hbase.sh

在master服务器上启动spark服务

/usr/local/spark/sbin/start-all.sh

在master-backup服务器上，启动第二个master

/usr/local/spark/sbin/start-master.sh

在master与master-backup服务器输入jps，都可以查看到Master

 Master

在其他服务器输入jps

 Worker

启动后就可以看到spark的web控制台地址了，在浏览器中输入地址访问，就可以查看到master节点的spark，Status为ALIVE，master-backup节点的spark，Status为STANDBY

http://192.168.10.90:8080/

http://192.168.10.91:8080/

11. 测试master切换

首先打开http://192.168.10.90:8080/ 与 http://192.168.10.91:8080/ 页面

在master服务器上输入jps，查看到Master服务的PID

 Master

然后输入命令，杀掉Master进程

kill -

运行scala（不运行的话，刷新页面看不到切换效果）

spark-shell --master spark://master:7077,master-backup:7077

接着在浏览器中刷新打开的两个页面，查看Workers是否已切换到另一台服务器上了

作者博客：http://www.cnblogs.com/EmptyFS/