解决sqoop报错:java.lang.OutOfMemoryError: Java heap space
报错栈:
-- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "CTJX60","CTJX61","CTJX62","CTJX63","CTJX64","CTJX65","CTJX66","CTJX67","CTJX68","CTJX69","CTJX70","CTJX71","CTJX72","CTJX73","CTJX74","CTJX75","CTJX76","CTJX77","CTJX78","CTJX79","CTJX80","CTJX81","CTJX82","CTJX83","CTJX84","CTJX85","CTJX86","CTJX87","CTJX88","CTJX89","CTJX90","CTJX91","CTJX92","CTJX93","CTJX94","CTJX95","CTJX96","CTJX97","CTJX98","CTJX99","CTJX55","IID","IEXAM_IID","CTSBGLX","CTJX01","CTJX02","CTJX03","CTJX04","CTJX05","CTJX06","CTJX07","CTJX08","CTJX09","CTJX10","CTJX11","CTJX12","CTJX13","CTJX14","CTJX15","CTJX16","CTJX17","CTJX18","CTJX19","CTJX20","CTJX21","CTJX22","CTJX23","CTJX24","CTJX25","CTJX26","CTJX27","CTJX28","CTJX29","CTJX30","CTJX31","CTJX32","CTJX33","CTJX34","CTJX35","CTJX36","CTJX37","CTJX38","CTJX39","CTJX40","CTJX41","CTJX42","CTJX43","CTJX44","CTJX45","CTJX46","CTJX47","CTJX48","CTJX49","CTJX50","CTJX51","CTJX52","CTJX53","CTJX54","CTJX56","CTJX57","CTJX58","CTJX59" from RIS."ICNRIS_UIS_EXAM_TJX" tbl where ( = ) AND ( = )
-- ::, INFO [Thread-] org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
-- ::, FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:)
at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:)
at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)
at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
ps: 调大mapper参数不管用,曾设置过参数:-D mapreduce.map.memory.mb=8192 -D yarn.app.mapreduce.am.resource.mb=6144。问题是在从db fetch数据的时候出现的,原表的列不算多,但每个列的长度很大:

解决sqoop报错:java.lang.OutOfMemoryError: Java heap space的更多相关文章
- 排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space
报错栈: -- ::, INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: = AND = -- ::, INFO [mai ...
- 【Xamarin报错】 COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java heap space
Xamarin Android 编译报错: COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java ...
- Tomcat报java.lang.OutOfMemoryError: Java heap space错误停止运行如何解决
最近开发的一个商业项目,部署完成后,经常出现Tomcat挂掉的现象,报的异常是:java.lang.OutOfMemoryError: Java heap space,上网google了一下,了解了一 ...
- Tomcat 启动项目报错 java.lang.OutOfMemoryError: Java heap space
近日使用myeclipse 部署web项目,启动tomcat时报错: SEVERE: Error waiting for multi-thread deployment of directories ...
- Linux运行Java出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错
在运行如下程序时出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错: java ...
- eclipse内存溢出报错:java.lang.OutOfMemoryError:Java heap space
今天执行了一个比較大的程序,处理的数据达到126MB数据,将数据导入数据库中,用eclipse 来訪问时候,总是出现java.lang.OutOfMemoryError:Java heap space ...
- mycat 报错 java.lang.OutOfMemoryError: Java heap space
今天排查mysql的错误日志发现 wrapper.log 中有如下错误日志 INFO | jvm 1 | 2019/10/20 12:52:31 | java.lang.OutOfMem ...
- 解决eclipse maven install 造成JVM 内存溢出(java.lang.OutOfMemoryError: Java heap space)
maven install 报错信息: The system is out of resources.Consult the following stack trace for details.jav ...
- JMeter内存溢出:java.lang.OutOfMemoryError: Java heap space解决方法
一.问题原因 用JMeter压测,有时候当模拟并发请求较大或者脚本运行时间较长时,JMeter会停止,报OOM(内存溢出)错误. 原因是JMeter是一个纯Java开发的工具,内存由java虚拟机JV ...
随机推荐
- 通过okhttp3下载文件实现APP版本更新
原文:https://blog.csdn.net/qq_34261214/article/details/77124729 概况 思路是这样的,首先在服务器上把已经签名打包的apk放上去,还有一份TX ...
- CCF CSP 201709-2 公共钥匙盒
CCF计算机职业资格认证考试题解系列文章为meelo原创,请务必以链接形式注明本文地址 CCF CSP 201709-2 公共钥匙盒 问题描述 有一个学校的老师共用N个教室,按照规定,所有的钥匙都必须 ...
- CentOS7.5安装cairo-dock,比mac托盘还美
1.下载安装nux-desktop 到http://li.nux.ro/download/nux/dextop/el7/x86_64/找到nux-dextop-release-xxxx.nux.noa ...
- bzoj 1202: [HNOI2005]狡猾的商人
我居然用暴力跑过去了... 思路:两个区间合成一个新的区间才会产生冲突, 我们用并查集维护前缀和, 0 - n 个节点分别表示sum[ 0 ] - sum[ n ], d[ i ] 表示 前缀i 和它 ...
- ubuntu12.04上的mongodb卸载
如果您需要卸载 mongodb,然后有几种方法来完成这取决于你想实现. 一.卸载只是 mongodb 这将删除只是 mongodb 包本身. 1 sudo apt-get remove mongodb ...
- OneDrive 开机启动设置失效如何处理?
问题现象: 『设置里勾选了开机启动onedrive,但是重启电脑勾选就没了,重新勾选没用,一重启就没了』 win10的onedrive无法开机启动 - Microsoft Community 处理方式 ...
- Python并发编程系列之多线程
1 引言 上一篇博文详细总结了Python进程的用法,这一篇博文来所以说Python中线程的用法.实际上,程序的运行都是以线程为基本单位的,每一个进程中都至少有一个线程(主线程),线程又可以创建子线程 ...
- JAVAEE——宜立方商城05:前台系统搭建、首页展示、Cms系统的实现
1. 学习计划 1.前台系统搭建 2.商城首页展示 3.Cms系统的实现 a) 内容分类管理 b) 内容管理 4.前台内容动态展示 2. 商城首页展示 系统架构: 页面位置: 2.1. 工程搭建 可以 ...
- 用 Python 实现一个大数据搜索引擎
搜索是大数据领域里常见的需求.Splunk和ELK分别是该领域在非开源和开源领域里的领导者.本文利用很少的Python代码实现了一个基本的数据搜索功能,试图让大家理解大数据搜索的基本原理. 布隆过滤器 ...
- Django Q对象
使用Q 对象进行复杂的查询¶ filter() 等方法中的关键字参数查询都是一起进行“AND” 的. 如果你需要执行更复杂的查询(例如OR 语句),你可以使用Q 对象. Q 对象 (django.db ...