报错栈:

-- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "CTJX60","CTJX61","CTJX62","CTJX63","CTJX64","CTJX65","CTJX66","CTJX67","CTJX68","CTJX69","CTJX70","CTJX71","CTJX72","CTJX73","CTJX74","CTJX75","CTJX76","CTJX77","CTJX78","CTJX79","CTJX80","CTJX81","CTJX82","CTJX83","CTJX84","CTJX85","CTJX86","CTJX87","CTJX88","CTJX89","CTJX90","CTJX91","CTJX92","CTJX93","CTJX94","CTJX95","CTJX96","CTJX97","CTJX98","CTJX99","CTJX55","IID","IEXAM_IID","CTSBGLX","CTJX01","CTJX02","CTJX03","CTJX04","CTJX05","CTJX06","CTJX07","CTJX08","CTJX09","CTJX10","CTJX11","CTJX12","CTJX13","CTJX14","CTJX15","CTJX16","CTJX17","CTJX18","CTJX19","CTJX20","CTJX21","CTJX22","CTJX23","CTJX24","CTJX25","CTJX26","CTJX27","CTJX28","CTJX29","CTJX30","CTJX31","CTJX32","CTJX33","CTJX34","CTJX35","CTJX36","CTJX37","CTJX38","CTJX39","CTJX40","CTJX41","CTJX42","CTJX43","CTJX44","CTJX45","CTJX46","CTJX47","CTJX48","CTJX49","CTJX50","CTJX51","CTJX52","CTJX53","CTJX54","CTJX56","CTJX57","CTJX58","CTJX59" from RIS."ICNRIS_UIS_EXAM_TJX" tbl where ( = ) AND ( = )
-- ::, INFO [Thread-] org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
-- ::, FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:)
at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:)
at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)
at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
解决:调小sqoop参数:--fetch-size
解决过程,查看sqoop源码,看到fetchSize,想到调整这个参数:

ps: 调大mapper参数不管用,曾设置过参数:-D mapreduce.map.memory.mb=8192 -D yarn.app.mapreduce.am.resource.mb=6144。问题是在从db fetch数据的时候出现的,原表的列不算多,但每个列的长度很大:

 

解决sqoop报错:java.lang.OutOfMemoryError: Java heap space的更多相关文章

  1. 排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space

    报错栈: -- ::, INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: = AND = -- ::, INFO [mai ...

  2. 【Xamarin报错】 COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java heap space

    Xamarin Android 编译报错: COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java ...

  3. Tomcat报java.lang.OutOfMemoryError: Java heap space错误停止运行如何解决

    最近开发的一个商业项目,部署完成后,经常出现Tomcat挂掉的现象,报的异常是:java.lang.OutOfMemoryError: Java heap space,上网google了一下,了解了一 ...

  4. Tomcat 启动项目报错 java.lang.OutOfMemoryError: Java heap space

    近日使用myeclipse 部署web项目,启动tomcat时报错: SEVERE: Error waiting for multi-thread deployment of directories ...

  5. Linux运行Java出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错

    在运行如下程序时出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错: java ...

  6. eclipse内存溢出报错:java.lang.OutOfMemoryError:Java heap space

    今天执行了一个比較大的程序,处理的数据达到126MB数据,将数据导入数据库中,用eclipse 来訪问时候,总是出现java.lang.OutOfMemoryError:Java heap space ...

  7. mycat 报错 java.lang.OutOfMemoryError: Java heap space

    今天排查mysql的错误日志发现  wrapper.log  中有如下错误日志 INFO   | jvm 1    | 2019/10/20 12:52:31 | java.lang.OutOfMem ...

  8. 解决eclipse maven install 造成JVM 内存溢出(java.lang.OutOfMemoryError: Java heap space)

    maven install 报错信息: The system is out of resources.Consult the following stack trace for details.jav ...

  9. JMeter内存溢出:java.lang.OutOfMemoryError: Java heap space解决方法

    一.问题原因 用JMeter压测,有时候当模拟并发请求较大或者脚本运行时间较长时,JMeter会停止,报OOM(内存溢出)错误. 原因是JMeter是一个纯Java开发的工具,内存由java虚拟机JV ...

随机推荐

  1. oracle创建job和删除job

    https://blog.csdn.net/u010001043/article/details/56479774

  2. springboot 1.5.X junit测试

    import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; impo ...

  3. codeforces Educational Codeforces Round 9 E - Thief in a Shop

    E - Thief in a Shop 题目大意:给你n ( n <= 1000)个物品每个物品的价值为ai (ai <= 1000),你只能恰好取k个物品,问你能组成哪些价值. 思路:我 ...

  4. vue 单向数据流 & 双向绑定

    在react中是单向数据绑定,而在vue中的特色是双向数据绑定.但是其实就我个人的理解是: 其实无论是vue还是react其实还是提倡单向数据流去管理状态,这一点在vuex和redux状态管理器上体现 ...

  5. WMRouter:美团外卖Android开源路由框架

    WMRouter是一款Android路由框架,基于组件化的设计思路,功能灵活,使用也比较简单. WMRouter最初用于解决美团外卖C端App在业务演进过程中的实际问题,之后逐步推广到了美团其他App ...

  6. [LOJ2541][PKUWC2018]猎人杀(容斥+分治+FFT)

    https://blog.csdn.net/Maxwei_wzj/article/details/80714129 n个二项式相乘可以用分治+FFT的方法,使用空间回收可以只开log个数组. #inc ...

  7. BZOJ.3489.A simple rmq problem(主席树 Heap)

    题目链接 当时没用markdown写,可能看起来比较难受...可以复制到别的地方看比如DevC++. \(Description\) 给定一个长为n的序列,多次询问[l,r]中最大的只出现一次的数.强 ...

  8. 本地文件包含漏洞(LFI漏洞)

    0x00 前言 本文的主要目的是分享在服务器遭受文件包含漏洞时,使用各种技术对Web服务器进行攻击的想法. 我们都知道LFI漏洞允许用户通过在URL中包括一个文件.在本文中,我使用了bWAPP和DVW ...

  9. Android源码目录结构详解

    Android 4.0|-- Makefile|-- bionic (bionic C库)|-- bootable (启动引导相关代码)|-- build (存放系统编译规则及generic等基础开发 ...

  10. 【2017多校训练08 1002】【HDOJ 6134】Battlestation Operational

    典型的数列反演题. 运用莫比乌斯反演的一个结论 $[n = 1] = \sum_{d | n} \mu(d)$,将表达式做如下转化: $$ ans = \sum_{i=1}^n \sum_{j=1}^ ...