报错栈：

-- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "CTJX60","CTJX61","CTJX62","CTJX63","CTJX64","CTJX65","CTJX66","CTJX67","CTJX68","CTJX69","CTJX70","CTJX71","CTJX72","CTJX73","CTJX74","CTJX75","CTJX76","CTJX77","CTJX78","CTJX79","CTJX80","CTJX81","CTJX82","CTJX83","CTJX84","CTJX85","CTJX86","CTJX87","CTJX88","CTJX89","CTJX90","CTJX91","CTJX92","CTJX93","CTJX94","CTJX95","CTJX96","CTJX97","CTJX98","CTJX99","CTJX55","IID","IEXAM_IID","CTSBGLX","CTJX01","CTJX02","CTJX03","CTJX04","CTJX05","CTJX06","CTJX07","CTJX08","CTJX09","CTJX10","CTJX11","CTJX12","CTJX13","CTJX14","CTJX15","CTJX16","CTJX17","CTJX18","CTJX19","CTJX20","CTJX21","CTJX22","CTJX23","CTJX24","CTJX25","CTJX26","CTJX27","CTJX28","CTJX29","CTJX30","CTJX31","CTJX32","CTJX33","CTJX34","CTJX35","CTJX36","CTJX37","CTJX38","CTJX39","CTJX40","CTJX41","CTJX42","CTJX43","CTJX44","CTJX45","CTJX46","CTJX47","CTJX48","CTJX49","CTJX50","CTJX51","CTJX52","CTJX53","CTJX54","CTJX56","CTJX57","CTJX58","CTJX59" from RIS."ICNRIS_UIS_EXAM_TJX" tbl where ( = ) AND ( = )

-- ::, INFO [Thread-] org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false

-- ::, FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space

    at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:)

    at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:)

    at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:)

    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:)

    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:)

    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:)

    at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:)

    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:)

    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:)

    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:)

    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:)

    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:)

    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:)

    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:)

    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)

    at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
解决：调小sqoop参数：--fetch-size
解决过程，查看sqoop源码，看到fetchSize，想到调整这个参数：

ps: 调大mapper参数不管用，曾设置过参数：-D mapreduce.map.memory.mb=8192 -D yarn.app.mapreduce.am.resource.mb=6144。问题是在从db fetch数据的时候出现的，原表的列不算多，但每个列的长度很大：

 

解决sqoop报错：java.lang.OutOfMemoryError: Java heap space的更多相关文章

  1. 排查sqoop报错：Error running child : java.lang.OutOfMemoryError: Java heap space

    报错栈: -- ::, INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: = AND = -- ::, INFO [mai ...

  2. 【Xamarin报错】 COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java heap space

    Xamarin Android 编译报错: COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java ...

  3. Tomcat报java.lang.OutOfMemoryError: Java heap space错误停止运行如何解决

    最近开发的一个商业项目,部署完成后,经常出现Tomcat挂掉的现象,报的异常是:java.lang.OutOfMemoryError: Java heap space,上网google了一下,了解了一 ...

  4. Tomcat 启动项目报错 java.lang.OutOfMemoryError: Java heap space

    近日使用myeclipse 部署web项目,启动tomcat时报错: SEVERE: Error waiting for multi-thread deployment of directories ...

  5. Linux运行Java出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错

    在运行如下程序时出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错: java ...

  6. eclipse内存溢出报错：java.lang.OutOfMemoryError:Java heap space

    今天执行了一个比較大的程序,处理的数据达到126MB数据,将数据导入数据库中,用eclipse 来訪问时候,总是出现java.lang.OutOfMemoryError:Java heap space ...

  7. mycat 报错 java.lang.OutOfMemoryError: Java heap space

    今天排查mysql的错误日志发现  wrapper.log  中有如下错误日志 INFO   | jvm 1    | 2019/10/20 12:52:31 | java.lang.OutOfMem ...

  8. 解决eclipse maven install 造成JVM 内存溢出(java.lang.OutOfMemoryError: Java heap space)

    maven install 报错信息: The system is out of resources.Consult the following stack trace for details.jav ...

  9. JMeter内存溢出：java.lang.OutOfMemoryError: Java heap space解决方法

    一.问题原因 用JMeter压测,有时候当模拟并发请求较大或者脚本运行时间较长时,JMeter会停止,报OOM(内存溢出)错误. 原因是JMeter是一个纯Java开发的工具,内存由java虚拟机JV ...

随机推荐

  1. 【转】AndroidStudio升到最新版本（3.1.2）之后

    AndroidStudio升到最新版本(3.1.2)之后   暂时发现的需要大家注意的地方 1.androidstudio3无法导入moudle? 例如:我写了一个简单的项目,需要导入一个第三方的mo ...

  2. markdown 语法规则

    markdown 语法规则 参考Markdown--入门指南 也可以参考这篇 Markdown: Basics (快速入门) 一级标题 二级标题 三级标题 列表 无序 1 2 3 有序 1 2 3 需 ...

  3. 【LOJ】#2558. 「LNOI2014」LCA

    题解 当年LN还是有专门的省选题的,但是还不如没有 看到这道题,我就想到了一个清晰易懂,简单好写,代码优美的树链剖分线段树套主席树的\(O(q\log^{3}n)\)做法,且就5组数据出题人肯定是不会 ...

  4. 004 关于Java如何扫描指定package下所有的类

    q前言: 在工作中看到这个知识点,就顺便参考了百度的一些资料,整理一下,希望以后用的到. 一:理论部分 1.使用场景 写一个MVC框架,需要从包中扫描出组件并注册到容器中,而JDK没有提供现成的从方法 ...

  5. poj1182 食物链（带权并查集）

    题目链接 http://poj.org/problem?id=1182 思路 前面做的带权并查集的权值记录该结点与其父结点是否是同一类,只有两种取值情况(0,1),在这题中某结点a和其父结点b的取值共 ...

  6. ECshop 迁移到 PHP7版本时遇到的兼容性问题，ecshopphp7

    ECshop 迁移到 PHP7版本时遇到的兼容性问题,ecshopphp7 在 PHP7 上安装 ECShop V2.7.3时,报错! Deprecated: Methods with the sam ...

  7. customPage.class.php可添加js事件的分页类

    用于ajax动态加载数据的分页类,分页事件可以动态添加,去除了a链接中的href地址. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 ...

  8. 四、redis系列之主从复制与哨兵机制

    1. 绪言 在现实应用环境中,出于数据容量.容灾.性能等因素的考虑,往往不会只使用一台服务器,而是使用集群的方式.Redis 中也有类似的维持一主多从的方式提高 Redis 集群的高可用性的方案,而其 ...

  9. Category 特性在 iOS 组件化中的应用与管控

    背景 iOS Category功能简介 Category 是 Objective-C 2.0之后添加的语言特性. Category 就是对装饰模式的一种具体实现.它的主要作用是在不改变原有类的前提下, ...

  10. 【Codeforces 498B】 B. Name That Tune （概率DP）

    B. Name That Tune time limit per test 1 second memory limit per test 256 megabytes input standard in ...