insert overwrite table canal_amt1......
2014-10-09 10:40:27,368 Stage-1 map = 100%,  reduce = 32%, Cumulative CPU 2772.48 sec
2014-10-09 10:40:28,426 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 2772.48 sec
2014-10-09 10:40:29,481 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 2774.12 sec
2014-10-09 10:40:30,885 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 2774.36 sec
2014-10-09 10:40:31,963 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2693.96 sec
2014-10-09 10:40:33,071 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2693.96 sec
2014-10-09 10:40:34,126 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2693.96 sec
2014-10-09 10:40:35,182 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2693.96 sec
MapReduce Total cumulative CPU time: 44 minutes 53 seconds 960 msec
Ended Job = job_1409124602974_0745 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1409124602974_0745_m_000003 (and more) from job job_1409124602974_0745
Examining task ID: task_1409124602974_0745_m_000002 (and more) from job job_1409124602974_0745
Examining task ID: task_1409124602974_0745_r_000000 (and more) from job job_1409124602974_0745
Examining task ID: task_1409124602974_0745_r_000006 (and more) from job job_1409124602974_0745 Task with the most failures(4):
-----
Task ID:
task_1409124602974_0745_r_000003 URL:
http://HADOOP2:8088/taskdetails.jsp?jobid=job_1409124602974_0745&tipid=task_1409124602974_0745_r_000003
-----
Diagnostic Messages for this Task:
Container [pid=22068,containerID=container_1409124602974_0745_01_000047] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1409124602974_0745_01_000047 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 22087 22068 22068 22068 (java) 2536 833 2730713088 265378 /usr/jdk64/jdk1.6.0_31/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx2048m -Djava.io.tmpdir=/hadoop/yarn/local/usercache/root/appcache/application_1409124602974_0745/container_1409124602974_0745_01_000047/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/hadoop/yarn/log/application_1409124602974_0745/container_1409124602974_0745_01_000047 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 54.0.88.58 41150 attempt_1409124602974_0745_r_000003_3 47
|- 22068 2381 22068 22068 (bash) 1 1 110755840 302 /bin/bash -c /usr/jdk64/jdk1.6.0_31/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx2048m -Djava.io.tmpdir=/hadoop/yarn/local/usercache/root/appcache/application_1409124602974_0745/container_1409124602974_0745_01_000047/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/hadoop/yarn/log/application_1409124602974_0745/container_1409124602974_0745_01_000047 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 54.0.88.58 41150 attempt_1409124602974_0745_r_000003_3 47 1>/hadoop/yarn/log/application_1409124602974_0745/container_1409124602974_0745_01_000047/stdout 2>/hadoop/yarn/log/application_1409124602974_0745/container_1409124602974_0745_01_000047/stderr Container killed on request. Exit code is 143 FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 23 Reduce: 7 Cumulative CPU: 2693.96 sec HDFS Read: 6278784712 HDFS Write: 590228229 FAIL
Total MapReduce CPU Time Spent: 44 minutes 53 seconds 960 msec

原因:空间不足

解决办法:

在执行hive语句前加上

set mapreduce.map.memory.mb=1025;//只要大于1024,hive默认分配的内存分大一倍,也就是2048M
set mapreduce.reduce.memory.mb=1025;

执行结果:

MapReduce Total cumulative CPU time: 0 days 1 hours 10 minutes 14 seconds 590 msec
Ended Job = job_1409124602974_0746
Loading data to table default.canal_amt1
Table default.canal_amt1 stats: [num_partitions: 0, num_files: 7, num_rows: 0, total_size: 4131948868, raw_data_size: 0]
MapReduce Jobs Launched:
Job 0: Map: 23 Reduce: 7 Cumulative CPU: 4214.59 sec HDFS Read: 6278784712 HDFS Write: 4131948868 SUCCESS
Total MapReduce CPU Time Spent: 0 days 1 hours 10 minutes 14 seconds 590 msec
OK
Time taken: 673.851 seconds

网上查询可能其他原因:

1.map阶段报空指针

原因:数据字段中插入了空值

2.Exception in thread "Thread-19" java.lang.IllegalArgumentException:
Does not contain a valid host:port authority: local

参考http://grokbase.com/p/cloudera/cdh-user/126wqvfwyt/hive-refuses-to-work-with-yarn

解决方法:

就是在hive-site.xml中添加设置

In the meantime I recommend doing the following if you need to run Hive on
MR2:
* Keep Hive happy by setting mapred.job.tracker to a bogus value.
* Disable task log retrieval by setting
hive.exec.show.job.failure.debug.info=false

3.protuf版本不一致。

hive: insert数据时Error during job, obtaining debugging information 以及beyond physical memory limits的更多相关文章

  1. 使用Sqoop从mysql向hdfs或者hive导入数据时出现的一些错误

    1.原表没有设置主键,出现错误提示: ERROR tool.ImportTool: Error during import: No primary key could be found for tab ...

  2. PHP使用prepare(),insert数据时要注意的一点!!!

    今天看了PHP防SQL注入,使用预处理prepare,但是我insert数据时,总是插不进去,但是select却可以,弄了很久终于知道原来问题在这里,先上代码 <?php header('con ...

  3. iOS解析数据时Error=3840

    1.解析JSon数据格式出错的问题 unescaped control character around character XXXX 和 The data couldn’t be read beca ...

  4. Kafka:ZK+Kafka+Spark Streaming集群环境搭建(十三)kafka+spark streaming打包好的程序提交时提示虚拟内存不足(Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 G)

    异常问题:Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical mem ...

  5. mysql中在表中insert数据时,有重复主键id时,变成update

    MySQL 自4.1版以后开始支持INSERT … ON DUPLICATE KEY UPDATE语法 例如:  id name sex age  1 kathy male 23  2 Javer f ...

  6. peewee insert 数据时报错:'buffer' object has no attribute 'translate'

    错误信息: "'buffer' object has no attribute 'translate'" 场景:使用peewee insert 数据时,BlobField 字段存储 ...

  7. hive insert 动态分区异常(Error encountered near token)与解决

    当insert数据到有分区的hive表里时若不明显指定分区会抛出异常 insert overwrite table persons_tmp select * from persons; FAILED: ...

  8. sqoop从hive导入数据到mysql时出现主键冲突

    今天在将一个hive数仓表导出到mysql数据库时出现进度条一直维持在95%一段时间后提示失败的情况,搞了好久才解决.使用的环境是HUE中的Oozie的workflow任何调用sqoop命令,该死的o ...

  9. Hive读取外表数据时跳过文件行首和行尾

    作者:Syn良子 出处:http://www.cnblogs.com/cssdongl 转载请注明出处 有时候用hive读取外表数据时,比如csv这种类型的,需要跳过行首或者行尾一些和数据无关的或者自 ...

随机推荐

  1. C# WinForm调用UnityWebPlayer Control控件 <学习笔记1>

    工具 1.三维场景 Unity 5.0.2f1 2.开发环境Microsoft Visual Studio 2010 3.需要使用的控件 UnityWebPlayer Control 出现的问题及解决 ...

  2. CentOS 通过yum来升级php到php5.6

    在文章中,我们将展示在centOS系统下如果将php升级到5.6,之前通过yum来安装lamp环境,直接升级的话,提示没有更新包,也就是说默认情况下php5.3.3是最新 1.查看已经安装的php版本 ...

  3. 【Java框架型项目从入门到装逼】第十五节 - jdbc模糊查询实现(附带详细调试过程)

    上一节,我们实现了用户列表查询,已经按条件精确查询: if(student.getUsername() != null && !"".equals(student. ...

  4. 常用的几个PHP加密函数

    1.前言 PHP加密方式分为单项散列加密,对称加密,非对称加密这几类. 像常用的MD5.hash.crypt.sha1这种就是单项散列加密,单项散列加密是不可逆的. 像URL编码.base64编码这种 ...

  5. bootstrap select2 使用简单介绍

    1. 基本属性配置: $("#select2-id").select2({ templateResult : formatState, // 列表带图片 templateSelec ...

  6. JUnit5 技术前瞻

    更多原创测试技术文章同步更新到微信公众号 :三国测,敬请扫码关注个人的微信号,感谢! 原文链接:http://www.cnblogs.com/zishi/p/6868495.html   JUnit ...

  7. c#实现自然排序效果,按1,2,11而不是1,11,12,区分字母文字和数字

    排序有时候要考虑后缀.这样看起来比较自然. 参考了codeproject上一篇文章:http://www.codeproject.com/Articles/22978/Implementing-the ...

  8. Lucene 5.X 版本索引文件格式

    原文链接:https://my.oschina.net/rickylau/blog/527602 名称 文件拓展名 描述 段文件 segments_N 保存了索引包含的多少段,每个段包含多少文档. 段 ...

  9. Java中的递归调用

    Java中不合理的使用递归调用,可能会导致栈内存溢出,这点是需要注意的. java将为每个线程维护一个栈,栈里将为每个方法保存一个栈帧,栈帧代表了一个方法的运行状态. 也就是我们常说的方法栈.最后一个 ...

  10. Python基础篇(二)

    Python最基本的数据结构是序列(sequence),序列中的每个元素被分以以0开头的唯一的一个id号. Python中有6种内建的序列:列表,元组,字符串,Unicode字符串,buffer对象和 ...