[Hadoop] - Hadoop Mapreduce Error: GC overhead limit exceeded
在运行mapreduce的时候,出现Error: GC overhead limit exceeded,查看log日志,发现异常信息为
2015-12-11 11:48:44,716 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.io.DataInputStream.readUTF(DataInputStream.java:661)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at xxxx.readFields(DateDimension.java:186)
at xxxx.readFields(StatsUserDimension.java:67)
at xxxx.readFields(StatsBrowserDimension.java:68)
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:158)
at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:158)
at org.apache.hadoop.mapreduce.task.ReduceContextImpl$ValueIterator.next(ReduceContextImpl.java:239)
at xxx.reduce(BrowserReducer.java:37)
at xxx.reduce(BrowserReducer.java:16)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
从异常中我们可以看到,在reduce读取一下个数据的时候,出现内存不够的问题,从代码中我发现再reduce端使用了读个map集合,这样会导致内存不够的问题。在hadoop2.x中默认Container的yarn child jvm堆大小为200M,通过参数mapred.child.java.opts指定,可以在job提交的时候给定,是一个客户端生效的参数,配置在mapred-site.xml文件中,通过将该参数修改为-Xms200m -Xmx1000m来更改jvm堆大小,异常解决。
参数名称 | 默认值 | 描述 |
mapred.child.java.opts | -Xmx200m | 定义mapreduce执行的container容器的执行jvm参数 |
mapred.map.child.java.opts | 单独指定map阶段的执行jvm参数 | |
mapred.reduce.child.java.opts | 单独指定reduce阶段的执行jvm参数 | |
mapreduce.admin.map.child.java.opts |
-Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN
|
管理员指定map阶段执行的jvm参数 |
mapreduce.admin.reduce.child.java.opts |
-Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN
|
管理员指定reduce阶段的执行jvm参数 |
上述五个参数生效的分别执行顺序为:
map阶段:mapreduce.admin.map.child.java.opts < mapred.child.java.opts < mapred.map.child.java.opts, 也就是说最终会采用mapred.map.child.java.opts定义的jvm参数,如果有冲突的话。
reduce阶段:mapreduce.admin.reduce.child.java.opts < mapred.child.java.opts < mapred.reduce.child.java.opts
hadoop源码参考:org.apache.hadoop.mapred.MapReduceChildJVM.getChildJavaOpts方法。
private static String getChildJavaOpts(JobConf jobConf, boolean isMapTask) {
String userClasspath = "";
String adminClasspath = "";
if (isMapTask) {
userClasspath = jobConf.get(JobConf.MAPRED_MAP_TASK_JAVA_OPTS,
jobConf.get(JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS));
adminClasspath = jobConf.get(
MRJobConfig.MAPRED_MAP_ADMIN_JAVA_OPTS,
MRJobConfig.DEFAULT_MAPRED_ADMIN_JAVA_OPTS);
} else {
userClasspath = jobConf.get(JobConf.MAPRED_REDUCE_TASK_JAVA_OPTS,
jobConf.get(JobConf.MAPRED_TASK_JAVA_OPTS,
JobConf.DEFAULT_MAPRED_TASK_JAVA_OPTS));
adminClasspath = jobConf.get(
MRJobConfig.MAPRED_REDUCE_ADMIN_JAVA_OPTS,
MRJobConfig.DEFAULT_MAPRED_ADMIN_JAVA_OPTS);
} // Add admin classpath first so it can be overridden by user.
return adminClasspath + " " + userClasspath;
}
[Hadoop] - Hadoop Mapreduce Error: GC overhead limit exceeded的更多相关文章
- Out of memory error : GC overhead limit exceeded
GC overhead limit exceeded 是指垃圾回收器通过分析当前内存使用趋势,提前抛出异常而不是真正等到内存耗尽再抛出异常.如果真正等到内存耗尽再抛出异常,可能的后果是:我们连保存重要 ...
- troubleshooting-sqoop mysql导入hive 报:GC overhead limit exceeded
Halting due to Out Of Memory Error...18/09/13 21:42:17 INFO mapreduce.Job: Task Id : attempt_1536756 ...
- sqoop import mysql to hive table:GC overhead limit exceeded
1. Scenario description when I use sqoop to import mysql table into hive, I got the following error: ...
- android studio Error:java.lang.OutOfMemoryError: GC overhead limit exceeded
android studio Error:java.lang.OutOfMemoryError: GC overhead limit exceeded 在app下的build.gradle中找到and ...
- eclipse一直报An internal error occurred during: "Building workspace". GC overhead limit exceeded
最近导入到eclipse里的工程挺大的,每次eclipse启动之后都回update workspace,然后就一直报: An internal error occurred during: " ...
- eclipse:An internal error occurred during: "Build Project". GC overhead limit exceeded
在使用Eclipse的Build Project功能时,提示以下错误: An internal error occurred during: "Build Project". GC ...
- An internal error occurred during: "Retrieving archetypes:". GC overhead limit exceeded
An internal error occurred during: "Retrieving archetypes:".GC overhead limit exceeded 异常, ...
- oozie: GC overhead limit exceeded 解决方法
1.异常表现形式 1) 提示信息 Error java.lang.OutOfMemoryError: GC overhead limit exceeded 2)提示出错 Erro ...
- java.lang.OutOfMemoryError:GC overhead limit exceeded填坑心得
我遇到这样的问题,本地部署时抛出异常java.lang.OutOfMemoryError:GC overhead limit exceeded导致服务起不来,查看日志发现加载了太多资源到内存,本地的性 ...
随机推荐
- ucos任务控制块详解
Ucos实现多任务的基础包括几个方面:任务控制块,任务堆栈,中断,任务优先级,一一说起 首先,任务控制块的结构如下 //系统在运行一个任务的时候,按照任务的优先级获取任务控制块,再在任务堆栈中获得任务 ...
- Matlab笔记
提取RGB分量,并显示出来 rgb=imread('mao.jpg'); rgb_r=rgb(:,:,); rgb_g=rgb(:,:,); rgb_b=rgb(:,:,); zero=zeros(, ...
- 【NOIP2015】反思+题解
D1T1> 神奇的幻方 模拟即可. #include <cstdio> #include <cstring> #include <algorithm> #de ...
- iOS开发中视图控制器ViewControllers之间的数据传递
iOS开发中视图控制器ViewControllers之间的数据传递 这里我们用一个demo来说明ios是如何在视图控制器之间传递重要的参数的.本文先从手写UI来讨论,在下一篇文章中讨论在storybo ...
- Android滚动选择控件
现在觉得github特别方便,我一般直接使用github中的内容, https://github.com/wangjiegulu/WheelView 这里面都有详细的介绍
- Sublime text追踪函数插件
Sublime Text2/3怎样在Ubuntu中配置CTags插件 | 浏览:1278 | 更新:2014-03-05 10:34 1 2 3 4 5 6 7 分步阅读 本文详解在Ubuntu Li ...
- 要重定向 IO 流,Process 对象必须将 UseShellExecute 属性设置为 False。
Process p1= new Process(); p1.StartInfo.UseShellExecute = false;
- HibernateSessionFactory类中Session对象的创建步骤
HibernateSessionFactory类中Session对象的创建步骤: 1.初始化Hibernate配置管理类Configuration 2.通过Configuration类实例创建Sess ...
- ViewFlipper的功能与用法
ViewFlipper组件继承了ViewAnimator,它可调用addView(View v)添加多个组件,一旦向ViewFlipper中添加了多个组件之后,ViewFlipper可使用动画控制多个 ...
- ViewSwitcher的功能与用法
ViewSwtcher代表了视图切换组件,它本身继承了FrameLayout,因此可以将多个View层叠在一起,每次只显示一个组件.当程序控制从一个View切换到另一个View时,ViewSwitch ...