docker使用Dockerfile搭建spark集群

1、创建Dockerfile文件，内容如下

# 基础镜像，包括jdk
FROM openjdk:8u131-jre-alpine

#作者
LABEL maintainer "tony@163.com"

#用户
USER root

#编码

ENV LANG=C.UTF-8 \
    TZ=Asia/Shanghai

#下载到时候安装spark需要的工具

RUN apk add --no-cache --update-cache bash curl tzdata wget tar \
    && cp /usr/share/zoneinfo/$TZ /etc/localtime \
    && echo $TZ > /etc/timezone

#设置工作目录

WORKDIR /usr/local

#拷贝 当下载过慢时先下载到本地在拷贝
# COPY spark-2.4.0-bin-hadoop2.7.tgz /usr/local 

#下载spark

RUN wget "http://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz" \
    && tar -vxf spark-* \
    && mv spark-2.4.0-bin-hadoop2.7 spark \
    && rm -rf spark-2.4.0-bin-hadoop2.7.tgz

#设定spark的home

ENV SPARK_HOME=/usr/local/spark \
    JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk \
    PATH=${PATH}:${JAVA_HOME}/bin:${SPARK_HOME}/bin

#暴露端口，8080 6066 7077 4044 18080

EXPOSE 6066 8080 7077 4044 18080

#工作目录

WORKDIR $SPARK_HOME

CMD ["/bin/bash"]

2、在Dockerfile所在目录下构建镜像

docker build -t spark:v2.4.0 .

3、启动主节点

docker run -itd --name spark-master -p : -p : -p : -h spark-master spark:v2.4.0 ./bin/spark-class org.apache.spark.deploy.master.Master

4、启动从节点

docker run -itd --name spark-worker -P -h spark-worker --link spark-master spark:v2.4.0 ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077

5、浏览器访问

http://192.168.75.142:8081/

6、启动spark-shell客户端

docker exec -it 99306051a9d4 ./bin/spark-shell

其中 99306051a9d4 是 spark-master 的容器ID

7、spark-shell测试

scala> val file = sc.textFile("/usr/local/spark/bin/beeline")
file: org.apache.spark.rdd.RDD[String] = /usr/local/spark/bin/beeline MapPartitionsRDD[] at textFile at <console>:

scala> val words = file.flatMap(line => line.split(" ")).map(word => (word,)).reduceByKey((a,b) => a+b)
words: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[] at reduceByKey at <console>:

scala> words.collect
res0: Array[(String, Int)] = Array(($CLASS,), (Software,), (Unless,), (this,), (starting,), (under,), (The,), (limitations,),
("$0")"/find-spark-home,1), (Figure,1), (express,1), (contributor,1), (#,20), (WITHOUT,1), ("AS,), (#!/usr/bin/env,), (See,), (License.,), 
(for,), (fi,), (software,), (IS",1), (obtain,1), (ANY,1), (SPARK_HOME,1), (out,1), (required,1), (2.0,1), (OR,1), (file,3), (the,9), (-o,1), 
(licenses,1), (not,1), (either,1), (if,2), (posix,2), (source,1), (Apache,2), (then,1), ("License");,1), (language,1), (License,3), (Enter,1),
(permissions,1), (WARRANTIES,1), (license,1), (by,1), (];,1), ("$(dirname,), (an,), ([,), (agreed,), (Version,), (implied.,), (KIND,,), 
(is,), ((the,), (exec,), ("${SPARK_HOME}",), (agreements.,), (on,), (You,), (one,)...
scala>