排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space
报错栈:
-- ::, INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: = AND =
-- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Working on split: = AND =
-- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "EXTEND3","EXTEND2","EXTEND1","MEMO","OPER_DATE","OPER_CODE","FILE_CONTENT","FILE_NAME","INPATIENT_NO","ID" from HIS_SDZL."MDT_FILE" tbl where ( = ) AND ( = )
-- ::, INFO [Thread-] org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
-- ::, FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:)
at java.lang.StringBuffer.append(StringBuffer.java:)
at java.util.regex.Matcher.appendReplacement(Matcher.java:)
at java.util.regex.Matcher.replaceAll(Matcher.java:)
at java.lang.String.replaceAll(String.java:)
at QueryResult.readFields(QueryResult.java:)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)
at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
调小fetchsize参数也不能解决,那问题很可能是某行数据占用空间很大。根据Sqoop生成的导入表对应的实例化类QueryResult.java的244行可定位到报错列是FILE_CONTENT,是个二进制列, 然后查询原库,果然最大的列长达到180M:

ps: 怎么用标准的sql语句查询 blob字段的大小?
blob字段有好多种。如果是9i的简单的blob字段则应该是 length,或者lengthb也可。实在不行可以用 dbms_lob.getlength()
排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space的更多相关文章
- 解决sqoop报错:java.lang.OutOfMemoryError: Java heap space
报错栈: -- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: = ) AND ( = ) ...
- 【Xamarin报错】 COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java heap space
Xamarin Android 编译报错: COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java ...
- 执行Hive出现Error running child : java.lang.OutOfMemoryError: Java heap space错误
具体错误日志如下: 2018-05-11 15:16:49,429 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running chi ...
- 【IntellJ IDEA】idea启动测试类报错Error running 'Test1.test': Command line is too long. Shorten command line for Test1.test or also for JUnit default configuration.
idea启动测试类报错 Error running 'Test1.test': Command line is too long. Shorten command line for Test1.tes ...
- 运行springboot项目报错 Error running 'ResourceApplication': Command line is too long. Shorten comma
方法1 IDEA 运行报错:Error running '***': Command line is too long 技术标签: IDEA Error running 'Test': Com ...
- Tomcat 启动项目报错 java.lang.OutOfMemoryError: Java heap space
近日使用myeclipse 部署web项目,启动tomcat时报错: SEVERE: Error waiting for multi-thread deployment of directories ...
- Linux运行Java出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错
在运行如下程序时出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错: java ...
- mycat 报错 java.lang.OutOfMemoryError: Java heap space
今天排查mysql的错误日志发现 wrapper.log 中有如下错误日志 INFO | jvm 1 | 2019/10/20 12:52:31 | java.lang.OutOfMem ...
- eclipse内存溢出报错:java.lang.OutOfMemoryError:Java heap space
今天执行了一个比較大的程序,处理的数据达到126MB数据,将数据导入数据库中,用eclipse 来訪问时候,总是出现java.lang.OutOfMemoryError:Java heap space ...
随机推荐
- java IntelliJ IDEA 13 注册码 IDEA序列号 License Key
java IntelliJ IDEA 13 注册码 IDEA序列号 License Key Username: JavaDeveloper@sskaje.me License: 282971-M1NW ...
- 字节对齐&&sizeof
转:http://blog.chinaunix.net/uid-722885-id-124878.html 1. sizeof应用在结构上的情况 请看下面的结构: struct MyStruct { ...
- jQuery 最简化实现
window.jQuery = (selector) => { let nodes = {}; if (typeof selector === 'string') { //是字符串 let te ...
- vue表格导出
inportexcel: function() { //兼容ie10哦! require.ensure([], () => { const { export_json_to_excel } = ...
- mysql协议简析
前言 如果要在命令行中连接mysql,最常用的便是 mysql -u root -p 这样指定用户名和密码 当然还可以使用远程连接 mysql -h 127.0.0.1 -u root -p 还有一种 ...
- 【转】使用 python IDLE 自带的 debuger 进行调试
IDLE自带了调试器(虽然不怎么使吧). 关于如何在 IDLE 中进行调试网上的教程讲的很模糊,于是自己整理了一下,和大家分享~ 第一步:打开python shell,并在shell中打开debuge ...
- 图形管线之旅 Part3
原文:<A trip through the Graphics Pipeline 2011> 翻译:往昔之剑 转载请注明出处 此时,我们一路上通过多个驱动层和命令处理器将draw ...
- Python入门基础知识(1) :locals() 和globals()
Python有两个内置的函数,locals() 和globals(),它们提供了基于字典的访问局部和全局变量的方式. 首先,是关于名字空间的一个名词解释.是枯燥,但是很重要,所以要耐心些.Python ...
- js使用s:property标签接收json格式数据
js使用s:property接收json数据时,会出现字符被转译的错误. 错误如下: 引号会被转译成'"'字符,导致解析不了. 错误原因: html的s:property接收不会出错,而js ...
- SDOI 2017 Round1 解题报告
Day 1 T1 数字表格 题目大意 · 求\(\prod\limits_{i=1}^n\prod\limits_{j=1}^mFibonacci(\gcd(i,j))\),\(T\leq1000\) ...