报错栈:

-- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "CTJX60","CTJX61","CTJX62","CTJX63","CTJX64","CTJX65","CTJX66","CTJX67","CTJX68","CTJX69","CTJX70","CTJX71","CTJX72","CTJX73","CTJX74","CTJX75","CTJX76","CTJX77","CTJX78","CTJX79","CTJX80","CTJX81","CTJX82","CTJX83","CTJX84","CTJX85","CTJX86","CTJX87","CTJX88","CTJX89","CTJX90","CTJX91","CTJX92","CTJX93","CTJX94","CTJX95","CTJX96","CTJX97","CTJX98","CTJX99","CTJX55","IID","IEXAM_IID","CTSBGLX","CTJX01","CTJX02","CTJX03","CTJX04","CTJX05","CTJX06","CTJX07","CTJX08","CTJX09","CTJX10","CTJX11","CTJX12","CTJX13","CTJX14","CTJX15","CTJX16","CTJX17","CTJX18","CTJX19","CTJX20","CTJX21","CTJX22","CTJX23","CTJX24","CTJX25","CTJX26","CTJX27","CTJX28","CTJX29","CTJX30","CTJX31","CTJX32","CTJX33","CTJX34","CTJX35","CTJX36","CTJX37","CTJX38","CTJX39","CTJX40","CTJX41","CTJX42","CTJX43","CTJX44","CTJX45","CTJX46","CTJX47","CTJX48","CTJX49","CTJX50","CTJX51","CTJX52","CTJX53","CTJX54","CTJX56","CTJX57","CTJX58","CTJX59" from RIS."ICNRIS_UIS_EXAM_TJX" tbl where ( = ) AND ( = )
-- ::, INFO [Thread-] org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
-- ::, FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:)
at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:)
at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)
at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
解决:调小sqoop参数:--fetch-size
解决过程,查看sqoop源码,看到fetchSize,想到调整这个参数:

ps: 调大mapper参数不管用,曾设置过参数:-D mapreduce.map.memory.mb=8192 -D yarn.app.mapreduce.am.resource.mb=6144。问题是在从db fetch数据的时候出现的,原表的列不算多,但每个列的长度很大:

 

解决sqoop报错:java.lang.OutOfMemoryError: Java heap space的更多相关文章

  1. 排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space

    报错栈: -- ::, INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: = AND = -- ::, INFO [mai ...

  2. 【Xamarin报错】 COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java heap space

    Xamarin Android 编译报错: COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java ...

  3. Tomcat报java.lang.OutOfMemoryError: Java heap space错误停止运行如何解决

    最近开发的一个商业项目,部署完成后,经常出现Tomcat挂掉的现象,报的异常是:java.lang.OutOfMemoryError: Java heap space,上网google了一下,了解了一 ...

  4. Tomcat 启动项目报错 java.lang.OutOfMemoryError: Java heap space

    近日使用myeclipse 部署web项目,启动tomcat时报错: SEVERE: Error waiting for multi-thread deployment of directories ...

  5. Linux运行Java出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错

    在运行如下程序时出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错: java ...

  6. eclipse内存溢出报错:java.lang.OutOfMemoryError:Java heap space

    今天执行了一个比較大的程序,处理的数据达到126MB数据,将数据导入数据库中,用eclipse 来訪问时候,总是出现java.lang.OutOfMemoryError:Java heap space ...

  7. mycat 报错 java.lang.OutOfMemoryError: Java heap space

    今天排查mysql的错误日志发现  wrapper.log  中有如下错误日志 INFO   | jvm 1    | 2019/10/20 12:52:31 | java.lang.OutOfMem ...

  8. 解决eclipse maven install 造成JVM 内存溢出(java.lang.OutOfMemoryError: Java heap space)

    maven install 报错信息: The system is out of resources.Consult the following stack trace for details.jav ...

  9. JMeter内存溢出:java.lang.OutOfMemoryError: Java heap space解决方法

    一.问题原因 用JMeter压测,有时候当模拟并发请求较大或者脚本运行时间较长时,JMeter会停止,报OOM(内存溢出)错误. 原因是JMeter是一个纯Java开发的工具,内存由java虚拟机JV ...

随机推荐

  1. mongo blancer

    在 sharded cluster 体系结构中,Balancer 进程的作用是转移数据,当一个 shard 中的数据比其它 shard 的数据多并达到一定条件时,Balancer 进程触发. 为了减少 ...

  2. Python全栈开发之17、tornado和web基础知识

    一.web基础知识 学习web框架之前,先来看一下web基础知识,首先要明白其本质就是socket,用户对应一个socket客户端,但是如果从socket开始开发web应用程序那么效率太了,正确的做法 ...

  3. Django实战(22):处理登录和注销

    我们已经可以在view函数中判断用户是否已经登录以及获取用户信息: if request.user.is_authenticated(): #判断用户是否已登录 user = request.user ...

  4. Java 中的国际化

    国际化 ,英文叫 internationalization 单词太长 ,又被简称为 i18n(取头取尾中间有18个字母)不经大声呼喊 ,这都行 !接着看什么是国际化 , 国际化是指让产品或是程序在无需 ...

  5. windows下thrift的使用(python)

    1.下载thrift,下载地址:http://archive.apache.org/dist/thrift/0.9.3/ 2.在编写python的thrift代码时,需要先安装thrift modul ...

  6. Django项目启动之前执行流程剖析

    下面,我们只看看主要的步骤: 1.项目启动,遍历settings下面的INSTALLED_APPS,导入默认配置. INSTALLED_APPS = [ 'django.contrib.admin', ...

  7. Java 的类加载顺序

    Java 的类加载顺序 一.加载顺序:先父类后子类,先静态后普通 1.父类的静态成员变量初始化 2.父类的静态代码块 3.子类的静态成员变量初始化 4.子类的静态代码块 5.父类的普通成员变量初始化 ...

  8. luoguP4208 [JSOI2008]最小生成树计数 矩阵树定理

    题目大意: 求最小生成树的数量 曾今的我感觉这题十分的不可做 然而今天看了看,好像是个类模板的题.... 我们十分容易知道,记能出现在最小生成树中的边的集合为\(S\) 那么,只要是\(S\)中的边构 ...

  9. 【数论】【扩展欧几里得】Codeforces Round #484 (Div. 2) E. Billiard

    题意:给你一个台球桌面,一个台球的初始位置和初始速度方向(只可能平行坐标轴或者与坐标轴成45度角),问你能否滚进桌子四个角落的洞里,如果能,滚进的是哪个洞. 如果速度方向平行坐标轴,只需分类讨论,看它 ...

  10. python调用matlab

    官网链接: https://ww2.mathworks.cn/help/matlab/matlab_external/call-user-script-and-function-from-python ...