Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task  in stage 0.0 failed  times, most recent failure: Lost task 3.3 in stage 0.0 (TID , hadoop7, executor ): ExecutorLostFailure (executor  exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.2 GB of  GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.

Driver stacktrace:

at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:)

at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)

at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:)

at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:)

at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)

at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)

at scala.Option.foreach(Option.scala:)

at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:)

at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:)

at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)

at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)

at org.apache.spark.util.EventLoop$$anon$.run(EventLoop.scala:)

ERROR : FAILED: Execution Error, return code  from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed because of out of memory.

INFO : Completed executing command(queryId=hive_20190529100107_063ed2a4-e3b0-48a9-9bcc-49acd51925c1); Time taken: 1441.753 seconds

Error: Error while processing statement: FAILED: Execution Error, return code  from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed because of out of memory. (state=,code=)

Closing: : jdbc:hive2://hadoop1:10000/pdw_nameonce

Hive on spark时报错

解决
a.set spark.yarn.executor.memoryOverhead=512G 调大(权宜之计)，excutor-momery + memoryOverhead不能大于集群内存
b.该问题的原因是因为OS层面虚拟内存分配导致，物理内存没有占用多少，但检查虚拟内存的时候却发现OOM，因此可以通过关闭虚拟内存检查来解决该问题，yarn.nodemanager.vmem-check-enabled=false 将虚拟内存检测设置为false

Hive-Container killed by YARN for exceeding memory limits. 9.2 GB of 9 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.的更多相关文章

  1. Container killed by YARN for exceeding memory limits

    19/08/12 14:15:35 ERROR cluster.YarnScheduler: Lost executor 5 on worker01.hadoop.mobile.cn: Contain ...

  2. [转载]Memory Limits for Windows and Windows Server Releases

    Memory Limits for Windows and Windows Server Releases This topic describes the memory limits for sup ...

  3. hadoop的job执行在yarn中内存分配调节————Container [pid=108284,containerID=container_e19_1533108188813_12125_01_000002] is running beyond virtual memory limits. Current usage: 653.1 MB of 2 GB physical memory used

    实际遇到的真实问题,解决方法: 1.调整虚拟内存率yarn.nodemanager.vmem-pmem-ratio (这个hadoop默认是2.1) 2.调整map与reduce的在AM中的大小大于y ...

  4. hive： insert数据时Error during job, obtaining debugging information 以及beyond physical memory limits

    insert overwrite table canal_amt1...... 2014-10-09 10:40:27,368 Stage-1 map = 100%, reduce = 32%, Cu ...

  5. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十三）kafka+spark streaming打包好的程序提交时提示虚拟内存不足（Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 G）

    异常问题:Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical mem ...

  6. spark运行任务报错：Container [...] is running beyond physical memory limits. Current usage: 3.0 GB of 3 GB physical memory used; 5.0 GB of 6.3 GB virtual memory used. Killing container.

    spark版本:1.6.0 scala版本:2.10 报错日志: Application application_1562341921664_2123 failed 2 times due to AM ...

  7. [hadoop] - Container [xxxx] is running beyond physical/virtual memory limits.

    当运行mapreduce的时候,有时候会出现异常信息,提示物理内存或者虚拟内存超出限制,默认情况下:虚拟内存是物理内存的2.1倍.异常信息类似如下: Container [pid=13026,cont ...

  8. Container [pid=6263,containerID=container_1494900155967_0001_02_000001] is running beyond virtual memory limits

    以Spark-Client模式运行,Spark-Submit时出现了下面的错误: User: hadoop Name: Spark Pi Application Type: SPARK Applica ...

  9. Spark- Spark Yarn模式下跑yarn-client无法初始化SparkConext,Over usage of virtual memory

    在spark yarn模式下跑yarn-client时出现无法初始化SparkContext错误. // :: INFO mapreduce.Job: Task Id : attempt_142829 ...

随机推荐

  1. electron桌面通知,修改默认通知应用名electron.app.Electron为自己应用的名称

    在做electron桌面通知时,按照文档实现弹出通知,但是默认的应用名为electron.app.Electron 解决办法 就是在主进程中设置 app.setAppUserModelId('myAp ...

  2. 21.django中间件源码阅读

    回顾: 关于里面的源码流程大家可以全看视频,因为代码的跳动性很大,而且会多次调用通过一方法,所以关于中间源码的部分去找个视频看一看,我写的不是很清楚. # 1 cookie session # 2 f ...

  3. Python3学习笔记（十五）：常用时间模块time和datetime

    一.time模块 1.时间戳 time.time() :从1970-01-01到至今的秒数 import time print(time.time()) 1529238004.2784646 2.等待 ...

  4. windows 简单实用Elasticsearch

    官网下载地址:https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/zip/e ...

  5. TensorFlow基本计算单元——变量

    # -*- coding: utf-8 -*- import tensorflow as tf a = 3 # 创建变量 w = tf.Variable([[0.5, 1.0]]) #行向量 x = ...

  6. spark MLlib 概念 2：Stratified sampling 层次抽样

    定义: In statistical surveys, when subpopulations within an overall population vary, it is advantageou ...

  7. Uva 101 -- the block problem

    Uva 101 the block problem 题目大意: 输入n,得到编号为0~n-1的木块,分别摆放在顺序排列编号为0~n-1的位置.现对这些木块进行操作,操作分为四种. 1.move a o ...

  8. Nature/Science 论文阅读笔记

    Nature/Science 论文阅读笔记 Unsupervised word embeddings capture latent knowledge from materials science l ...

  9. 六、Java中一个key存多个value的MultiValueMap，一键多值

    一.MultiValueMap介绍,与map,HashMap相差不多 添加一个Key对应一个Value的:void add(K, V); 添加一个Key对应多个Value的:void add(K, L ...

  10. fstab中使用设备的uuid

    设备定位的方法有: 设备名称, 如:/dev/sda1, 随着linux内核加载模块顺序在每次启动的时候可能会不同, 在插拔U盘/移动硬盘的时候, 设备分配到的名称可能不同,这样fs映射就会失败 因此 ...