异常问题：Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

spark-submit提交脚本：

[spark@master work]$ more submit.sh

#! /bin/bash

jars=""

for file in `ls /home/spark/work/jars/*.jar`

do

        jars=$file,$jars

        #echo $jars

done

echo "------------------------------------"

echo $jars

echo "------------------------------------"

/opt/spark-2.2.1-bin-hadoop2.7/bin/spark-submit \

--jars $jars \

--master yarn \

--verbose \

--driver-java-options "-XX:+TraceClassPaths" \

--num-executors 2 \

--executor-memory 1G \

--executor-cores 1 \

--driver-memory 1G \

--class com.dx.streaming.producer.TestProducer \

/home/spark/work/kafka-streaming-test.jar

在master节点上提交后，抛出异常：

18/07/09 05:00:16 INFO yarn.Client: Application report for application_1531069155353_0001 (state: ACCEPTED)

18/07/09 05:00:17 INFO yarn.Client: Application report for application_1531069155353_0001 (state: FAILED)

18/07/09 05:00:17 INFO yarn.Client:

         client token: N/A

         diagnostics: Application application_1531069155353_0001 failed 2 times due to AM Container for appattempt_1531069155353_0001_000002 exited with  exitCode: -103

Failing this attempt.Diagnostics: [2018-07-09 05:00:16.132]Container [pid=5674,containerID=container_e22_1531069155353_0001_02_000001] is running beyond virtual memory limits. Current usage: 195.8 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_e22_1531069155353_0001_02_000001 :

        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

        |- 5674 5673 5674 5674 (bash) 0 0 115896320 350 /bin/bash -c /opt/jdk1.8.0_171/bin/java -server -Xmx512m -Djava.io.tmpdir=/opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/tmp -Dspark.yarn.app.container.log.dir=/opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg '192.168.0.120:40486' --properties-file /opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/__spark_conf__/__spark_conf__.properties 1> /opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/stdout 2> /opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/stderr

        |- 5687 5674 5674 5674 (java) 208 35 2323283968 49770 /opt/jdk1.8.0_171/bin/java -server -Xmx512m -Djava.io.tmpdir=/opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/tmp -Dspark.yarn.app.container.log.dir=/opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg 192.168.0.120:40486 --properties-file /opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/__spark_conf__/__spark_conf__.properties

[2018-07-09 05:00:16.133]

[2018-07-09 05:00:16.152]Container killed on request. Exit code is 143[2018-07-09 05:00:16.152]

[2018-07-09 05:00:16.156]Container exited with a non-zero exit code 143. [2018-07-09 05:00:16.156]

For more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1531069155353_0001 Then click on links to logs of each attempt.

. Failing the application.

         ApplicationMaster host: N/A

         ApplicationMaster RPC port: -1

         queue: default

         start time: 1531083596749

         final status: FAILED

         tracking URL: http://master:8088/cluster/app/application_1531069155353_0001

         user: spark

18/07/09 05:00:17 INFO yarn.Client: Deleted staging directory hdfs://HA/user/spark/.sparkStaging/application_1531069155353_0001

18/07/09 05:00:17 ERROR spark.SparkContext: Error initializing SparkContext.

org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)

        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)

▽       at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)

        at scala.Option.getOrElse(Option.scala:121)

        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)

        at com.dx.streaming.producer.TestProducer.main(TestProducer.java:39)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

18/07/09 05:00:17 INFO server.AbstractConnector: Stopped Spark@60d84f61{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

18/07/09 05:00:17 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.0.120:4040

18/07/09 05:00:17 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!

18/07/09 05:00:17 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors

18/07/09 05:00:17 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down

18/07/09 05:00:17 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices

(serviceOption=None,

 services=List(),

 started=false)

18/07/09 05:00:17 INFO cluster.YarnClientSchedulerBackend: Stopped

18/07/09 05:00:17 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

18/07/09 05:00:17 INFO memory.MemoryStore: MemoryStore cleared

18/07/09 05:00:17 INFO storage.BlockManager: BlockManager stopped

18/07/09 05:00:17 INFO storage.BlockManagerMaster: BlockManagerMaster stopped

18/07/09 05:00:17 WARN metrics.MetricsSystem: Stopping a MetricsSystem that is not running

18/07/09 05:00:17 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

18/07/09 05:00:17 INFO spark.SparkContext: Successfully stopped SparkContext

Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)

        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)

        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)

        at scala.Option.getOrElse(Option.scala:121)

        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)

        at com.dx.streaming.producer.TestProducer.main(TestProducer.java:39)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

18/07/09 05:00:17 INFO util.ShutdownHookManager: Shutdown hook called

18/07/09 05:00:17 INFO util.ShutdownHookManager: Deleting directory /opt/spark-2.2.1-bin-hadoop2.7/spark-255c65cf-19d8-481a-94bc-843e058eb669

错误原因：

Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

YARN的虚拟内存计算方式导致，上例中用户程序申请的内存为1Gb，YARN根据此值乘以一个比例（默认为2.1）得出申请的虚拟内存的值，当YARN计算的用户程序所需虚拟内存值大于计算出来的值时，就会报出以上错误。

解决方案：

上网一查，发现有两种方法能解决这个问题:

1.将yarn.nodemanager.vmem-check-enabled的值改为false，即不检查VM的值；（修改yarn-site.xml）参考《https://discuss.elastic.co/t/unable-to-start-elasticsearch-yarn-container-is-running-beyond-virtual-memory-limits/26102/6》

<property>

    <name>yarn.nodemanager.vmem-check-enabled</name>

    <value>false</value>

</property>

2.将yarn.scheduler.minimum-allocation-mb的值调高一些，默认是1024mb,或者修改yarn.nodemanager.vmem-pmem-ratio的值，默认为2.1，将该值改得更大。

于是，这里我采用了第一种方法，关闭vn检查，需要重新格式化namenode，参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十）安装hadoop2.9.0搭建HA》，再次提交，果然好了。终于解决了这个困扰一天的问题啦！

Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十三）kafka+spark streaming打包好的程序提交时提示虚拟内存不足（Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 G）的更多相关文章

  1. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二十一）NIFI1.7.1安装

    一.nifi基本配置 1. 修改各节点主机名,修改/etc/hosts文件内容. 192.168.0.120 master 192.168.0.121 slave1 192.168.0.122 sla ...

  2. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十二）VMW安装四台CentOS，并实现本机与它们能交互，虚拟机内部实现可以上网。

    Centos7出现异常:Failed to start LSB: Bring up/down networking. 按照<Kafka:ZK+Kafka+Spark Streaming集群环境搭 ...

  3. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十一）定制一个arvo格式文件发送到kafka的topic，通过Structured Streaming读取kafka的数据

    将arvo格式数据发送到kafka的topic 第一步:定制avro schema: { "type": "record", "name": ...

  4. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十）安装hadoop2.9.0搭建HA

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  5. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（九）安装kafka_2.11-1.1.0

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  6. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（八）安装zookeeper-3.4.12

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  7. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（三）安装spark2.2.1

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  8. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二）安装hadoop2.9.0

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  9. hadoop的job执行在yarn中内存分配调节————Container [pid=108284,containerID=container_e19_1533108188813_12125_01_000002] is running beyond virtual memory limits. Current usage: 653.1 MB of 2 GB physical memory used

    实际遇到的真实问题,解决方法: 1.调整虚拟内存率yarn.nodemanager.vmem-pmem-ratio (这个hadoop默认是2.1) 2.调整map与reduce的在AM中的大小大于y ...

随机推荐

  1. HTML5中的跨文档消息传递

    跨文档消息传送(cross-document messaging),有时候也简称为XDM,指的是来自不同域的页面间传递消息.例如,www.w3cmm.com域中的一个页面与一个位于内嵌框架中的p2p. ...

  2. CENTOS下搭建SVN服务器(转)

    1.安装svn yum install -y subversion 2.验证安装是否成功 svnserve --version 3.创建svn版本库 mkdir svn svnadmin create ...

  3. nginx优化(转)

    Puppet利用Nginx多端口实现负载均衡 对 Nginx SSL 的性能进行调整 一.nginx 配置文件中基本设置: 1.  worker_processes 8; 2.  worker_cpu ...

  4. HDU 4763 Theme Section （2013长春网络赛1005，KMP）

    Theme Section Time Limit: 2000/1000 MS (Java/Others)    Memory Limit: 32768/32768 K (Java/Others)Tot ...

  5. windows和linux 下将tomcat注册为服务

    参考文献: tomcat注册成windows服务 背景 当前项目需要运行两个Tomcat,每次启动系统以后都要手动进入到tomcat目录执行startup.bat,非常烦,所以想将这两个tomcat直 ...

  6. Python yield使用

    https://www.ibm.com/developerworks/cn/opensource/os-cn-python-yield/ 您可能听说过,带有 yield 的函数在 Python 中被称 ...

  7. 项目从.NET 4.5迁移到.NET 4.0遇到的问题

    当把项目从.NET 4.5迁移到.NET 4.0时,遇到的问题和解决如下: 在"属性--应用程序--目标框架"设置成.NET 4.0版本. 重新生成项目,报有关EF的错: 卸载掉项 ...

  8. panel内嵌程序窗体

    function RunAppInPanel(const AppFileName: string; ParentHandle: HWND; var WinHandle: HWND): Boolean; ...

  9. Arcgis for JavascriptAPI 常用接口

    转自原文arcgis for javascriptAPI常用接口 var map, navToolbar, editToolbar, tileLayer, toolbar; //var mapBase ...

  10. iOS 7.1 系统可以设置 button shapes,此功能可让按钮多一条下滑线

    IniOS 7, Apple completely revamped the user interface to give it a fresh and modern look. One of the ...