Spark 大数据平台

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write.

BDAS, the Berkeley Data Analytics Stack, is an open source software stack that integrates software components being built by the AMPLab to make sense of Big Data.





Spark Components	VS.	Hadoop Components
Spark Core	<------>	Apache Hadoop MR
Spark Streaming	<------>	Apache Storm
Spark SQL	<------>	Apache Hive
Spark GraphX	<------>	MPI(taobao)
Spark MLlib	<------>	Apache Mahout

BlinkDB is a massively parallel, approximate query engine for running interactive SQL queries on large volumes of data. It allows users to +, enabling interactive queries over massive data by running queries on data samples and presenting results annotated with meaningful error bars.
Two key ideas:

• An adaptive optimization framework that builds and maintains a set of multi-dimensional samples from original data over time
• A dynamic sample selection strategy that selects an appropriately sized sample based on a query’s accuracy and/or response time requirements.

Why spark is fast:

• in-memory computing
• Directed Acyclic Graph (DAG) engine, compiler can see the whole computing graph in advance so that it can optimize it. Delay Scheduling

Resilient Distributed Dataset

• A list of partitions
• A function for computing each split
• A list of dependencies on other RDDs
• Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)
• Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)

Storage Strategy

class StorageLevel private(
    private var useDisk_ : Boolean,
    private var useMemory_ : Boolean,
    private var deserialized_ : Boolean,
    private var replication_ : Int = 1)

val MEMORY_ONLY_ = new StorageLevel(false, true, true)

RDD, transformation & action

lazy evaluation