Spark(1) - Getting Started with Apache Spark

Introduction

Apache Spark is a general-purpose cluster computing system to process big data workloads. What sets Spark apart from its predecessors, such as MapReduce, is its speed, ease-of-use, and sophisticated analytics.

Apache Spark was originally developed at AMPLab, UC Berkeley, in 2009. It was made open source in 2010 under the BSD license and switched to the Apache 2.0 license in 2013. Toward the later part of 2013, the creators of Spark founded Databricks to focus on Spark's development and future releases.

Talking about speed, Spark can achieve sub-second latency on big data workloads. To achieve such low latency, Spark makes use of the memory for storage. In MapReduce, memory is primarily used for actual computation. Spark uses memory both to compute and store objects.

Spark also provides a unified runtime connecting to various big data storage sources, such as HDFS, Cassandra, HBase, and S3. It also provides a rich set of higher-level libraries for different big data compute tasks, such as machine learning, SQL processing, graph processing, and real-time streaming. These libraries make development faster and can be combined in an arbitrary fashion.

Though Spark is written in Scala, and this book only focuses only recipes in Scala, Spark also supports Java and Python.

Spark is an open source community project, and everyone uses the pure open source Apache distributions for deployments, unlike Hadoop, which has multiple distributinos available with vendor enhancements.

The Spark runtime runs on top of a variety of cluster managers, including YARN(Hadoop's compute framework), Mesos, and Spark's own cluster manager called standalone mode. Tachyon is a memory-centric distributed file system that enables reliable file sharing at memory speed across cluster frameworks. In short, it is an off-heap storage layer in memory, which helps share data across jobs and users. Mesos is a cluster manager, which is evolving into a data center operating system. YARN is Hadoop's compute framework that has a robust resource management reature that Spark can seamlessly use.

Installing Spark from binaries

http://spark.apache.org/downloads.html

1. download binaries

wget http://d3kbcqa49mib13.cloudfront.net/spark-1.4.0-bin-hadoop2.4.tgz

2. unpack binaries

tar -zxf spark-1.4.0-bin-hadoop2.4.tgz

3. rename the folder

sudo mv spark-1.4.0-bin-hadoop2.4 spark

4. move the configuration folder to the /etc folder

sudo mv spark/conf/* /etc/spark

5. create installation directory under /opt

sudo mkdir -p /opt/infoobjects

6. move the spark directory to /opt/infoobjects

sudo mv spark /opt/infoobjects/

7. change ownership of the spark home to root

sudo chown -R root:root /opt/infoobjects/spark

8. change permission for the spark home

sudo chmod -R 755 /opt/infoobjects/spark

9. move to the spark home

cd /opt/infoobjects/spark

10. create the symbolic link

sudo ln -s /etc/spark conf

11. append to PATH in .bashrc

echo "export PATH=$PATH:/opt/infoobjects/spark/bin" >> /home/hduser/.bashrc

12. open a new terminal

13. create a log directory in /var

sudo mkdir -p /var/log/spark

14. make hduser the owner of the spark log

sudo chown -R hduser:hduser /var/log/spark

15. create the spark tmp directory

mkdir /tmp/spark

16. configure spark

cd /etc/spark

echo "export HADOOP_CONF_DIR=/opt/infoobjects/hadoop/etc/hadoop" >> spark-env.sh
echo "export YARN_CONF_DIR=/opt/infoobjects/hadoop/etc/Hadoop" >> spark-env.sh
echo "export SPARK_LOG_DIR=/var/log/spark" >> spark-env.sh
echo "export SPARK_WORKER_DIR=/tmp/spark" >> spark-env.sh

Building the Spark source code with Maven

Java 1.6 & Maven 3.x

1. increase MaxPermSize for heap

echo "export _JAVA_OPTIONS=\"-XX:MaxPermSize=1G\"" >> /home/hduser/.bashrc

2. open a new terminal and download the spark source code from GitHub

wget https://github.com/apache/spark/archive/branch-1.4.zip

3. unpack the archive

gunzip branch-1.4.zip

4. move to the spark directory

cd spark

5. compile the sources

mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -DskipTests clean package

6. move the conf folder to the etc folder

sudo mv spark/conf /etc/

7. move the spark directory to /opt

sudo mv spark /opt/infoobjects/spark

8. change ownership of the spark home to root

sudo chown -R root:root /opt/infoobjects/spark

9. change permission for the spark home

sudo chmod -R 755 /opt/infoobjects/spark

10. move to the spark home

cd /opt/infoobjects/spark

11. create the symbolic link

sudo ln -s /etc/spark conf

12. append to PATH in .bashrc

echo "export PATH=$PATH:/opt/infoobjects/spark/bin" >> /home/hduser/.bashrc

13. open a new terminal

14. create a log directory in /var

sudo mkdir -p /var/log/spark

15. make hduser the owner of the spark log

sudo chown -R hduser:hduser /var/log/spark

16. create the spark tmp directory

mkdir /tmp/spark

17. configure spark

cd /etc/spark

echo "export HADOOP_CONF_DIR=/opt/infoobjects/hadoop/etc/hadoop" >> spark-env.sh
echo "export YARN_CONF_DIR=/opt/infoobjects/hadoop/etc/Hadoop" >> spark-env.sh
echo "export SPARK_LOG_DIR=/var/log/spark" >> spark-env.sh
echo "export SPARK_WORKER_DIR=/tmp/spark" >> spark-env.sh

Launching Spark on Amazon EC2

Getting ready

1. login to the Amazon AWS account(http://aws.amazon.com)
2. click on Security Credentials under your account name in the top-right corner
3. click on Access Keys and Create New Access Key
4. get access key id and secret access key
5. go to Services | EC2
6. click on Key Pairs in left-hand menu under NETWORK & SECURITY
7. click on Create Key Pair and enter kp-spark as key-pair name
8. download the private key file and copy it in the /home/hduser/keypairs folder
9. set permissions on key file to 600
10. set environment variables to reflect access key ID and secret access key

echo "export AWS_ACCESS_KEY_ID=\"{ACCESS_KEY_ID}\"" >> /home/hduser/.bashrc
echo "export AWS_SECRET_ACESS_KEY=\"{AWS_SECRET_ACESS_KEY}\"" >> /home/hduser/.bashrc
echo "export PATH=$PATH:/opt/infoobject/spark/ec2" >> /home/hduser/.bashrc

1. launch the cluster

cd /home/hduser
spark-ec2 -k <key-pair> -i <key-file> -s <num-slaves> launch <cluster-name>

2. launch the cluser with example value

spark-ec2 -k kp-spark -i /home/hduser/keypairs/kp-spark.pem --hadoop-major-version 2 -s 3 launch spark-cluster

3. specify zone if default availability zones not available

spark-ec2 -k kp-spark -i /home/hduser/keypairs/kp-spark.pem -z us-east-1b --hadoop-major-version 2 -s 3 launch spark-cluster

4. attach EBS volume if needs to retain data after the instance shuts down

spark-ec2 -k kp-spark -i /home/hduser/keypairs/kp-spark.pem --hadoop-major-version 2 -ebs-vol-size 10 -s 3 launch spark-cluster

5. use Amazon spot instances

spark-ec2 -k kp-spark -i /home/hduser/keypairs/kp-spark.pem -spot-price=0.15 --hadoop-major-version 2 -s 3 launch spark-cluster

6. check the status of the cluster

the url will be printed at the end

7. connect to the master node

spark-ec2 -k kp-spark -i /home/hduser/kp/kp-spark.pem login spark-cluster

8. check the HDFS version in an ephemeral instance

ephemeral-hdfs/bin/hadoop version

9. check the HDFS version in persistent instance

persistent-hdfs/bin/hadoop version

Deploying on a cluster in standalone mode

Compute resources in a distributed environment need to be managed so that resource utilization is efficient and every job gets a fair chance to run. Spark comes along with its own cluster manager conveniently called standalone mode. Spark also supports working with YARN and Mesos cluster managers.

The cluster manager that should be chosen is mostly driven by both legacy concerns and whether other frameworks, such as MapReduce, are sharing the same compute resource pool. If your cluster has legacy MapReduce jobs running, and all of them cannot be converted to Spark jobs, it is a good idea to use YARN as the cluster manager. Mesos is emerging as a data center operating system to conveniently manage jobs across frameworks, and is very compatible with Spark.

If the Spark framework is the only framework in your cluster, then standalone mode is good enough. As Spark evolves as technology, you will see more and more use cases of Spark being used as the standalone framework serving all big data compute needs. For example, some jobs may be using Apache Mahout at present because MLlib does not have a specific machine-learning library, which the job needs. As soon as MLlib gets this library, this particular job can be moved to Spark.

one master and five slaves

Master
m1.zettabytes.com

Slaves
s1.zettabytes.com
s2.zettabytes.com
s3.zettabytes.com
s4.zettabytes.com
s5.zettabytes.com

1. install spark binaries on both master and slave machines, put /opt/infoobjects/spark/sbin in path on every node

echo "export PATH=$PATH:/opt/infoobjects/spark/sbin" >> /home/hduser/.bashrc

2. ssh to master and start the standalone master server

start-master.sh

3. ssh to slave and start slaves

spark-class org.apache.spark.deploy.worker.Worker spark://m1.zettabytes.com:7077

4. create conf/slaves file on a master node and add one line per slave hostname

echo "s1.zettabytes.com" >> conf/slaves
echo "s2.zettabytes.com" >> conf/slaves
echo "s3.zettabytes.com" >> conf/slaves
echo "s4.zettabytes.com" >> conf/slaves
echo "s5.zettabytes.com" >> conf/slaves

start-master.sh
start-slaves.sh
start-all.sh
stop-master.sh
stop-slaves.sh
stop-all.sh

5. connect an application to the cluster through Scala code

val sparkContext = new SparkContext(new SparkConf().setMaster("spark://m1.zettabytes.com:7077"))

6. connect to the cluster through spark shell

spark-shell --master spark://master:7077

Deploying on a cluster with Mesos

Mesos is slowly emerging as a data center operating system to manage all compute resources across a data center. Mesos runs on any computer running the Linux operating system. Mesos is built using the same principles as Linux kernel.

1. Execute Mesos on Ubuntu OS with the trusty version

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') CODENAME=$(lsb_release -cs)

sudo vi /etc/apt/sources.list.d/mesosphere.list

deb http://repos.mesosphere.io/Ubuntu trusty main

2. install mesos

sudo apt-get -y update
sudo apt-get -y install mesos

3. make spark binaries available to mesos and configure the spark driver to connect to mesos

4. upload spark binaries to HDFS

hdfs dfs -put spark-1.4.0-bin-hadoop2.4.tgz spark-1.4.0-bin-hadoop2.4.tgz

5. the master url for single master mesos is mesos://host:5050, and for the ZooKeeper managed mesos cluster, it is mesos://zk://host:2181

6. set variables in spark-env.sh

sudo vi spark-env.sh

export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
export SPARK_EXECUTOR_URI= hdfs://localhost:9000/user/hduser/spark-1.4.0-bin-hadoop2.4.tgz

7. run from Scala program

val conf = new SparkConf().setMaster("mesos://host:5050")
val sparkContext = new SparkContext(conf)

8. run from the Spark shell

spark-shell --master mesos://host:5050

Mesos has two run modes:
Fine-grained: In fine-grained (default) mode, every Spark task runs as a separate Mesos task
Coarse-grained: This mode will launch only one long-running Spark task on each Mesos machine

9. set to run in the coarse-grained mode

conf.set("spark.mesos.coarse", "true")

Deploying on a cluster with YARN

Yet another resource negotiator (YARN) is Hadoop's compute framework that runs on top of HDFS, which is Hadoop's storage layer.

YARN follows the master slave architecture. The master daemon is called ResourceManager and the slave daemon is called NodeManager. Besides this application, life cycle management is done by ApplicationMaster, which can be spawned on any slave node and is alive for the lifetime of an application.

When Spark is run on YARN, ResourceManager performs the role of Spark master and NodeManagers work as executor nodes.

While running Spark with YARN, each Spark executor is run as YARN container.

1. set the configuration

HADOOP_CONF_DIR: to write to HDFS
YARN_CONF_DIR: to connect to YARN ResourceManager

cd /opt/infoobjects/spark/conf (or /etc/spark)

sudo vi spark-env.sh
export HADOOP_CONF_DIR=/opt/infoobjects/hadoop/etc/Hadoop
export YARN_CONF_DIR=/opt/infoobjects/hadoop/etc/hadoop

2. launch YARN Spark in the yarn-client mode

spark-submit --class path.to.your.Class --master yarn-client [options] <app jar> [app options]

spark-submit --class com.infoobjects.TwitterFireHose --master yarn-client --num-executors 3 --driver-memory 4g --executor-memory 2g --executor-cores 1 target/sparkio.jar 10

3. launch spark shell in the yarn-client mode

spark-shell --master yarn-client

4. launch in the yarn-cluster mode

spark-submit --class path.to.your.Class --master yarn-cluster [options] <app jar> [app options]

spark-submit --class com.infoobjects.TwitterFireHose --master yarn-cluster --num-executors 3 --driver-memory 4g --executor-memory 2g --executor-cores 1 target/sparkio.jar 10