Spark高级
Spark源码分析:
https://yq.aliyun.com/articles/28400?utm_campaign=wenzhang&utm_medium=article&utm_source=QQ-qun&utm_content=m_11999
Spark shuffle:
http://blog.csdn.net/johnny_lee/article/details/22619585
Spark java.lang.OutOfMemoryError: Java heap space
My cluster: 1 master, 11 slaves, each node has 6 GB memory.
My settings:
spark.executor.memory=4g, Dspark.akka.frameSize=512
Here is the problem:
First, I read some data (2.19 GB) from HDFS to RDD:
val imageBundleRDD = sc.newAPIHadoopFile(...)
Second, do something on this RDD:
val res = imageBundleRDD.map(data => {
val desPoints = threeDReconstruction(data._2, bg)
(data._1, desPoints)
})
Last, output to HDFS:
res.saveAsNewAPIHadoopFile(...)
When I run my program it shows:
.....
14/01/15 21:42:27 INFO cluster.ClusterTaskSetManager: Starting task 1.0:24 as TID 33 on executor 9: Salve7.Hadoop (NODE_LOCAL)
14/01/15 21:42:27 INFO cluster.ClusterTaskSetManager: Serialized task 1.0:24 as 30618515 bytes in 210 ms
14/01/15 21:42:27 INFO cluster.ClusterTaskSetManager: Starting task 1.0:36 as TID 34 on executor 2: Salve11.Hadoop (NODE_LOCAL)
14/01/15 21:42:28 INFO cluster.ClusterTaskSetManager: Serialized task 1.0:36 as 30618515 bytes in 449 ms
14/01/15 21:42:28 INFO cluster.ClusterTaskSetManager: Starting task 1.0:32 as TID 35 on executor 7: Salve4.Hadoop (NODE_LOCAL)
Uncaught error from thread [spark-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark]
I have a few suggestions:
- If your nodes are configured to have 6g maximum for Spark (and are leaving a little for other processes), then use 6g rather than 4g,
spark.executor.memory=6g
. Make sure you're using as much memory as possible by checking the UI (it will say how much mem you're using) - Try using more partitions, you should have 2 - 4 per CPU. IME increasing the number of partitions is often the easiest way to make a program more stable (and often faster). For huge amounts of data you may need way more than 4 per CPU, I've had to use 8000 partitions in some cases!
- Decrease the fraction of memory reserved for caching, using
spark.storage.memoryFraction
. If you don't usecache()
orpersist
in your code, this might as well be 0. It's default is 0.6, which means you only get 0.4 * 4g memory for your heap. IME reducing the mem frac often makes OOMs go away. UPDATE: From spark 1.6 apparently we will no longer need to play with these values, spark will determine them automatically. - Similar to above but shuffle memory fraction. If your job doesn't need much shuffle memory then set it to a lower value (this might cause your shuffles to spill to disk which can have catastrophic impact on speed). Sometimes when it's a shuffle operation that's OOMing you need to do the opposite i.e. set it to something large, like 0.8, or make sure you allow your shuffles to spill to disk (it's the default since 1.0.0).
- Watch out for memory leaks, these are often caused by accidentally closing over objects you don't need in your lambdas. The way to diagnose is to look out for the "task serialized as XXX bytes" in the logs, if XXX is larger than a few k or more than an MB, you may have a memory leak. See http://stackoverflow.com/a/25270600/1586965
- Related to above; use broadcast variables if you really do need large objects.
- If you are caching large RDDs and can sacrifice some access time consider serialising the RDDhttp://spark.apache.org/docs/latest/tuning.html#serialized-rdd-storage. Or even caching them on disk (which sometimes isn't that bad if using SSDs).
- (Advanced) Related to above, avoid
String
and heavily nested structures (likeMap
and nested case classes). If possible try to only use primitive types and index all non-primitives especially if you expect a lot of duplicates. ChooseWrappedArray
over nested structures whenever possible. Or even roll out your own serialisation - YOU will have the most information regarding how to efficiently back your data into bytes, USE IT! - (bit hacky) Again when caching, consider using a
Dataset
to cache your structure as it will use more efficient serialisation. This should be regarded as a hack when compared to the previous bullet point. Building your domain knowledge into your algo/serialisation can minimise memory/cache-space by 100x or 1000x, whereas all aDataset
will likely give is 2x - 5x in memory and 10x compressed (parquet) on disk.
http://spark.apache.org/docs/1.2.1/configuration.html
EDIT: (So I can google myself easier) The following is also indicative of this problem:
java.lang.OutOfMemoryError : GC overhead limit exceeded
Answer2:
Have a look at the start up scripts a Java heap size is set there, it looks like you're not setting this before running Spark worker.
# Set SPARK_MEM if it isn't already set since we also use it for this process
SPARK_MEM=${SPARK_MEM:-512m}
export SPARK_MEM
# Set JAVA_OPTS to be able to load native libraries and to set heap size
JAVA_OPTS="$OUR_JAVA_OPTS"
JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$SPARK_LIBRARY_PATH"
JAVA_OPTS="$JAVA_OPTS -Xms$SPARK_MEM -Xmx$SPARK_MEM"
You can find the documentation to deploy scripts here.
Spark高级的更多相关文章
- Spark高级数据分析——纽约出租车轨迹的空间和时间数据分析
Spark高级数据分析--纽约出租车轨迹的空间和时间数据分析 一.地理空间分析: 二.pom.xml 原文地址:https://www.jianshu.com/p/eb6f3e0c09b5 作者:II ...
- Learning Spark中文版--第六章--Spark高级编程(2)
Working on a Per-Partition Basis(基于分区的操作) 以每个分区为基础处理数据使我们可以避免为每个数据项重做配置工作.如打开数据库连接或者创建随机数生成器这样的操作,我们 ...
- Learning Spark中文版--第六章--Spark高级编程(1)
Introduction(介绍) 本章介绍了之前章节没有涵盖的高级Spark编程特性.我们介绍两种类型的共享变量:用来聚合信息的累加器和能有效分配较大值的广播变量.基于对RDD现有的transform ...
- spark高级排序彻底解秘
排序,真的非常重要! RDD.scala(源码) 在其,没有罗列排序,不是说它不重要! 1.基础排序算法实战 2.二次排序算法实战 3.更高级别排序算法 4.排序算法内幕解密 1.基础排序算法实战 启 ...
- spark高级编程
启动spark-shell 如果你有一个Hadoop 集群, 并且Hadoop 版本支持YARN, 通过为Spark master 设定yarn-client 参数值,就可以在集群上启动Spark 作 ...
- Spark高级数据分析· 3推荐引擎
推荐算法流程 推荐算法 预备 wget http://www.iro.umontreal.ca/~lisa/datasets/profiledata_06-May-2005.tar.gz cd /Us ...
- Spark高级数据分析-第2章 用Scala和Spark进行数据分析
2.4 小试牛刀:Spark shell和SparkContext 本章使用的资料来自加州大学欧文分校机器学习资料库(UC Irvine Machine Learning Repository),这个 ...
- Spark高级函数应用【combineByKey、transform】
一.combineByKey算子简介 功能:实现分组自定义求和及计数. 特点:用于处理(key,value)类型的数据. 实现步骤: 1.对要处理的数据进行初始化,以及一些转化操作 2.检测key是否 ...
- 10、spark高级编程
一.基于排序机制的wordcount程序 1.要求 1.对文本文件内的每个单词都统计出其出现的次数. 2.按照每个单词出现次数的数量,降序排序. 2.代码实现 ------java实现------- ...
随机推荐
- django 和 mongdb 写一个简陋的网址,以及用django内置的分页功能
https://github.com/factsbenchmarks/simple_websit_about_58 一 设置 数据库的设置 在settings文件中加入这样一段代码: from mon ...
- 安卓解析JSON文件
安卓解析JSON文件 根据JOSN文件的格式,文件只有两种数据,一是对象数据,以 {}为分隔,二是数组,以[]分隔 以下介绍安卓如何解析一个JSON文件,该文件存放在assets目录下,即:asset ...
- msp430项目编程55
msp430综合项目---扩展项目五55 1.电路工作原理 2.代码(显示部分) 3.代码(功能实现) 4.项目总结
- AC日记——[网络流24题]方格取数问题 cogs 734
734. [网络流24题] 方格取数问题 ★★☆ 输入文件:grid.in 输出文件:grid.out 简单对比时间限制:1 s 内存限制:128 MB «问题描述: 在一个有m*n ...
- 实现TTCP (检测TCP吞吐量)
实现TTCP (检测TCP吞吐量) 应用层协议 为了解决TCP粘包问题以及客户端阻塞问题 设计的应用层协议如下: //告知要发送的数据包个数和长度 struct SessionMessage { in ...
- CentOS 7.5 初始网络配置
最近刚装完 CentOS 7.5 系统,由于网络不通,导致无法用 yum 命令下载软件,经过了各种折腾,终于搞定了,这里讲解一下 如何设置初始网络. 本案例环境 VmWare 11.0 , 操作系统 ...
- Adding an Exception Breakpoint - Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSArrayM objectAtIndex:]: index 25 bey
用如下的方法可以非常方便停留到具体crash的某行代码 Adding an Exception Breakpoint Add an exception breakpoint to your proje ...
- 【面试 AOP】【第八篇】AOP的问题
1.AOP的原理以及应用场景 面向切面编程,不修改原有代码逻辑的情况下进行逻辑增强. 使用场景:短信业务,restful返回统一响应体等等. ============================= ...
- remove_if的问题
#include<iostream> #include<list> #include<algorithm> #include"PRINT_ELEMENTS ...
- codeforces 553 A Kyoya and Colored Balls
这个题.比赛的时候一直在往dp的方向想,可是总有一个组合数学的部分没办法求, 纯粹组合数学撸,也想不到办法-- 事实上,非常显然.. 从后往前推,把第k种颜色放在最后一个,剩下的k球.还有C(剩余的位 ...