排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space
报错栈:
-- ::, INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: = AND =
-- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Working on split: = AND =
-- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "EXTEND3","EXTEND2","EXTEND1","MEMO","OPER_DATE","OPER_CODE","FILE_CONTENT","FILE_NAME","INPATIENT_NO","ID" from HIS_SDZL."MDT_FILE" tbl where ( = ) AND ( = )
-- ::, INFO [Thread-] org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
-- ::, FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:)
at java.lang.StringBuffer.append(StringBuffer.java:)
at java.util.regex.Matcher.appendReplacement(Matcher.java:)
at java.util.regex.Matcher.replaceAll(Matcher.java:)
at java.lang.String.replaceAll(String.java:)
at QueryResult.readFields(QueryResult.java:)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)
at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
调小fetchsize参数也不能解决,那问题很可能是某行数据占用空间很大。根据Sqoop生成的导入表对应的实例化类QueryResult.java的244行可定位到报错列是FILE_CONTENT,是个二进制列, 然后查询原库,果然最大的列长达到180M:

ps: 怎么用标准的sql语句查询 blob字段的大小?
blob字段有好多种。如果是9i的简单的blob字段则应该是 length,或者lengthb也可。实在不行可以用 dbms_lob.getlength()
排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space的更多相关文章
- 解决sqoop报错:java.lang.OutOfMemoryError: Java heap space
报错栈: -- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: = ) AND ( = ) ...
- 【Xamarin报错】 COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java heap space
Xamarin Android 编译报错: COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java ...
- 执行Hive出现Error running child : java.lang.OutOfMemoryError: Java heap space错误
具体错误日志如下: 2018-05-11 15:16:49,429 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running chi ...
- 【IntellJ IDEA】idea启动测试类报错Error running 'Test1.test': Command line is too long. Shorten command line for Test1.test or also for JUnit default configuration.
idea启动测试类报错 Error running 'Test1.test': Command line is too long. Shorten command line for Test1.tes ...
- 运行springboot项目报错 Error running 'ResourceApplication': Command line is too long. Shorten comma
方法1 IDEA 运行报错:Error running '***': Command line is too long 技术标签: IDEA Error running 'Test': Com ...
- Tomcat 启动项目报错 java.lang.OutOfMemoryError: Java heap space
近日使用myeclipse 部署web项目,启动tomcat时报错: SEVERE: Error waiting for multi-thread deployment of directories ...
- Linux运行Java出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错
在运行如下程序时出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错: java ...
- mycat 报错 java.lang.OutOfMemoryError: Java heap space
今天排查mysql的错误日志发现 wrapper.log 中有如下错误日志 INFO | jvm 1 | 2019/10/20 12:52:31 | java.lang.OutOfMem ...
- eclipse内存溢出报错:java.lang.OutOfMemoryError:Java heap space
今天执行了一个比較大的程序,处理的数据达到126MB数据,将数据导入数据库中,用eclipse 来訪问时候,总是出现java.lang.OutOfMemoryError:Java heap space ...
随机推荐
- 使用Python快速查询所有指定匹配KEY的办法
import redis redis_ip = '10.10.14.224' redis_port = 18890 # 配置redis的连接办法 # http://blog.csdn.net/u010 ...
- ssh客户端乱码
export LC_ALL=zh_CN.GB2312;export LANG=zh_CN.GB2312
- Tornado入门资料整理
预备知识 没学过计网的苦逼找点现成一些的东西看吧…… <Restful Web Services>,<HTTP The Definitive Guide>,各种RFC WSGI ...
- 大理石在哪儿(UVa10474)
题目具体描述见:https://uva.onlinejudge.org/index.php?option=com_onlinejudge&Itemid=8&category=835&a ...
- Yii 简明学习教程
Yii是一个基于组件的高性能PHP框架,用于快速开发Web应用程序(下面内容基于Yii 1.1) 1. 典型的工作流 用户向入口脚本index.php发起请求 入口脚本加载应用配置config.php ...
- Java中的覆盖和隐藏以及final关键字
Java覆盖和隐藏 (1) 变量只能被隐藏(包括静态和非静态),不能被覆盖: (2) 可以用子类的静态变量隐藏父类的静态变量,也可以用子类的非静态变量隐藏父类的静态变量,也可以用非最终变量(final ...
- windows关闭aslr办法
关闭aslr方便调试分析. 转:https://www.52pojie.cn/thread-377450-1-1.html windows关闭aslr办法 如 @Hmily 前辈所说, Window ...
- Hibernate 对象的生命周期及CRUD操作
对象状态及生命周期 1.Transient(瞬时状态):new的对象为瞬时态 session中没有数据,数据库中没有数据. 2.Persistent(持久状态):被session管理的对象为持久状态, ...
- hdu 4819 二维线段树模板
/* HDU 4819 Mosaic 题意:查询某个矩形内的最大最小值, 修改矩形内某点的值为该矩形(Mi+MA)/2; 二维线段树模板: 区间最值,单点更新. */ #include<bits ...
- [GCJ2017R3]Cooclement
题目大意: 一种数列按照如下方式变化: 新数列第i位等于原数中数字i的出现次数. 变化过程中数列长度不变. 例如数列12的变化过程为12-11-20-01-10. 现在告诉你一个数列x,请求出x可能是 ...