异常问题：Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

spark-submit提交脚本：

[spark@master work]$ more submit.sh

#! /bin/bash

jars=""

for file in `ls /home/spark/work/jars/*.jar`

do

        jars=$file,$jars

        #echo $jars

done

echo "------------------------------------"

echo $jars

echo "------------------------------------"

/opt/spark-2.2.1-bin-hadoop2.7/bin/spark-submit \

--jars $jars \

--master yarn \

--verbose \

--driver-java-options "-XX:+TraceClassPaths" \

--num-executors 2 \

--executor-memory 1G \

--executor-cores 1 \

--driver-memory 1G \

--class com.dx.streaming.producer.TestProducer \

/home/spark/work/kafka-streaming-test.jar

在master节点上提交后，抛出异常：

18/07/09 05:00:16 INFO yarn.Client: Application report for application_1531069155353_0001 (state: ACCEPTED)

18/07/09 05:00:17 INFO yarn.Client: Application report for application_1531069155353_0001 (state: FAILED)

18/07/09 05:00:17 INFO yarn.Client:

         client token: N/A

         diagnostics: Application application_1531069155353_0001 failed 2 times due to AM Container for appattempt_1531069155353_0001_000002 exited with  exitCode: -103

Failing this attempt.Diagnostics: [2018-07-09 05:00:16.132]Container [pid=5674,containerID=container_e22_1531069155353_0001_02_000001] is running beyond virtual memory limits. Current usage: 195.8 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_e22_1531069155353_0001_02_000001 :

        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

        |- 5674 5673 5674 5674 (bash) 0 0 115896320 350 /bin/bash -c /opt/jdk1.8.0_171/bin/java -server -Xmx512m -Djava.io.tmpdir=/opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/tmp -Dspark.yarn.app.container.log.dir=/opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg '192.168.0.120:40486' --properties-file /opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/__spark_conf__/__spark_conf__.properties 1> /opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/stdout 2> /opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/stderr

        |- 5687 5674 5674 5674 (java) 208 35 2323283968 49770 /opt/jdk1.8.0_171/bin/java -server -Xmx512m -Djava.io.tmpdir=/opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/tmp -Dspark.yarn.app.container.log.dir=/opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg 192.168.0.120:40486 --properties-file /opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/__spark_conf__/__spark_conf__.properties

[2018-07-09 05:00:16.133]

[2018-07-09 05:00:16.152]Container killed on request. Exit code is 143[2018-07-09 05:00:16.152]

[2018-07-09 05:00:16.156]Container exited with a non-zero exit code 143. [2018-07-09 05:00:16.156]

For more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1531069155353_0001 Then click on links to logs of each attempt.

. Failing the application.

         ApplicationMaster host: N/A

         ApplicationMaster RPC port: -1

         queue: default

         start time: 1531083596749

         final status: FAILED

         tracking URL: http://master:8088/cluster/app/application_1531069155353_0001

         user: spark

18/07/09 05:00:17 INFO yarn.Client: Deleted staging directory hdfs://HA/user/spark/.sparkStaging/application_1531069155353_0001

18/07/09 05:00:17 ERROR spark.SparkContext: Error initializing SparkContext.

org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)

        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)

▽       at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)

        at scala.Option.getOrElse(Option.scala:121)

        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)

        at com.dx.streaming.producer.TestProducer.main(TestProducer.java:39)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

18/07/09 05:00:17 INFO server.AbstractConnector: Stopped Spark@60d84f61{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

18/07/09 05:00:17 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.0.120:4040

18/07/09 05:00:17 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!

18/07/09 05:00:17 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors

18/07/09 05:00:17 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down

18/07/09 05:00:17 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices

(serviceOption=None,

 services=List(),

 started=false)

18/07/09 05:00:17 INFO cluster.YarnClientSchedulerBackend: Stopped

18/07/09 05:00:17 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

18/07/09 05:00:17 INFO memory.MemoryStore: MemoryStore cleared

18/07/09 05:00:17 INFO storage.BlockManager: BlockManager stopped

18/07/09 05:00:17 INFO storage.BlockManagerMaster: BlockManagerMaster stopped

18/07/09 05:00:17 WARN metrics.MetricsSystem: Stopping a MetricsSystem that is not running

18/07/09 05:00:17 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

18/07/09 05:00:17 INFO spark.SparkContext: Successfully stopped SparkContext

Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)

        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)

        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)

        at scala.Option.getOrElse(Option.scala:121)

        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)

        at com.dx.streaming.producer.TestProducer.main(TestProducer.java:39)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

18/07/09 05:00:17 INFO util.ShutdownHookManager: Shutdown hook called

18/07/09 05:00:17 INFO util.ShutdownHookManager: Deleting directory /opt/spark-2.2.1-bin-hadoop2.7/spark-255c65cf-19d8-481a-94bc-843e058eb669

错误原因：

Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

YARN的虚拟内存计算方式导致，上例中用户程序申请的内存为1Gb，YARN根据此值乘以一个比例（默认为2.1）得出申请的虚拟内存的值，当YARN计算的用户程序所需虚拟内存值大于计算出来的值时，就会报出以上错误。

解决方案：

上网一查，发现有两种方法能解决这个问题:

1.将yarn.nodemanager.vmem-check-enabled的值改为false，即不检查VM的值；（修改yarn-site.xml）参考《https://discuss.elastic.co/t/unable-to-start-elasticsearch-yarn-container-is-running-beyond-virtual-memory-limits/26102/6》

<property>

    <name>yarn.nodemanager.vmem-check-enabled</name>

    <value>false</value>

</property>

2.将yarn.scheduler.minimum-allocation-mb的值调高一些，默认是1024mb,或者修改yarn.nodemanager.vmem-pmem-ratio的值，默认为2.1，将该值改得更大。

于是，这里我采用了第一种方法，关闭vn检查，需要重新格式化namenode，参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十）安装hadoop2.9.0搭建HA》，再次提交，果然好了。终于解决了这个困扰一天的问题啦！

Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十三）kafka+spark streaming打包好的程序提交时提示虚拟内存不足（Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 G）的更多相关文章

  1. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二十一）NIFI1.7.1安装

    一.nifi基本配置 1. 修改各节点主机名,修改/etc/hosts文件内容. 192.168.0.120 master 192.168.0.121 slave1 192.168.0.122 sla ...

  2. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十二）VMW安装四台CentOS，并实现本机与它们能交互，虚拟机内部实现可以上网。

    Centos7出现异常:Failed to start LSB: Bring up/down networking. 按照<Kafka:ZK+Kafka+Spark Streaming集群环境搭 ...

  3. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十一）定制一个arvo格式文件发送到kafka的topic，通过Structured Streaming读取kafka的数据

    将arvo格式数据发送到kafka的topic 第一步:定制avro schema: { "type": "record", "name": ...

  4. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十）安装hadoop2.9.0搭建HA

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  5. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（九）安装kafka_2.11-1.1.0

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  6. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（八）安装zookeeper-3.4.12

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  7. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（三）安装spark2.2.1

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  8. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二）安装hadoop2.9.0

    如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

  9. hadoop的job执行在yarn中内存分配调节————Container [pid=108284,containerID=container_e19_1533108188813_12125_01_000002] is running beyond virtual memory limits. Current usage: 653.1 MB of 2 GB physical memory used

    实际遇到的真实问题,解决方法: 1.调整虚拟内存率yarn.nodemanager.vmem-pmem-ratio (这个hadoop默认是2.1) 2.调整map与reduce的在AM中的大小大于y ...

随机推荐

  1. @Transactional导致无法动态数据源切换

    公司目前数据源为主从模式:主库可读写,从库只负责读.使用spring-jdbc提供的AbstractRoutingDataSource结合ThreadLocal存储key,实现数据源动态切换. 最近项 ...

  2. C#——性能计数器

    简要Windows性能监视器: 打开Windows性能监视器的步骤如下: 开始→运行→perfmon→确定 在这里我们可以选择添加我们要监控的计数器,比如:cpu使用率.内存使用量等,作为asp.ne ...

  3. [Go] 如何正确地 抛出 错误 和 异常（error/panic/recover）？

    序言 错误 和 异常 是两个不同的概念,非常容易混淆.很多程序员习惯将一切非正常情况都看做错误,而不区分错误和异常,即使程序中可能有异常抛出,也将异常及时捕获并转换成错误.从表面上看,一切皆错误的思路 ...

  4. Delphi中,除了应用程序主窗口会显示在任务栏上,其它窗口默认都不会显示在任务栏.

    Delphi中,除了应用程序主窗口会显示在任务栏上,其它窗口默认都不会显示在任务栏. Delphi中,除了应用程序主窗口会显示在任务栏上,其它窗口默认都不会显示在任务栏.没有MS开发环境中的ShowI ...

  5. Android 关于“NetworkOnMainThreadException”出错提示的原因及解决办法

    几乎每天都在论坛里面看到有网友问这个问题,代码是无误的,在低版本的API上都可以运行的,但在3.0以上的版本就会出现NetworkOnMainThreadException 出现android.os. ...

  6. WordPress主题开发实例：显示最新发表文章

    实现效果 制作网站的时候,想在页面左栏显示最新发表的文章,如: 一.先开启侧边栏工具,在functions.php加上 $args = array( 'name' => __( '分类侧边栏') ...

  7. ASP.NET ：Virtual Application vs Virtual Directory

    原文地址:http://blogs.msdn.com/b/wenlong/archive/2006/11/22/virtual-application-vs-virtual-directory.asp ...

  8. Java中判断String不为空的问题性能比较

    function 1: 最多人使用的一个方法, 直观, 方便, 但效率很低. function 2: 比较字符串长度, 效率高, 是我知道的最好一个方法. function 3: Java SE 6. ...

  9. Java分布式系统高并发解决方案

    对于我们开发的网站,如果网站的访问量非常大的话,那么我们就需要考虑相关的并发访问问题了.而并发问题是绝大部分的程序员头疼的问题, 但话又说回来了,既然逃避不掉,那我们就坦然面对吧~今天就让我们一起来研 ...

  10. Android手机提示“未安装应用程序”

    用eclipse调试应用时,遇到了这个问题,网上给出的解决方案倒是挺多,但似乎一个都没奏效,而且我手机也重启了,还是有问题,郁闷ing-   然后看到一篇文章指出,可能不是签名和SD的卡问题,而是我们 ...