1 贴出完整日志信息

// :: INFO client.RMProxy: Connecting to ResourceManager at hdp1/192.168.56.201:
// :: INFO yarn.Client: Requesting a new application from cluster with NodeManagers
// :: INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster ( MB per container)
// :: INFO yarn.Client: Will allocate AM container, with MB memory including MB overhead
// :: INFO yarn.Client: Setting up container launch context for our AM
// :: INFO yarn.Client: Setting up the launch environment for our AM container
// :: INFO yarn.Client: Preparing resources for our AM container
// :: WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
// :: INFO yarn.Client: Uploading resource file:/mnt/software/spark-2.2.-bin-hadoop2./spark-3045ce36-b8d7--a9aa-e0509ef0ec89/__spark_libs__8280328345325459092.zip -> hdfs://hdp1:9000/user/root/.sparkStaging/application_1548663204656_0001/__spark_libs__8280328345325459092.zip
// :: INFO yarn.Client: Uploading resource file:/mnt/software/examples/jars/spark-examples*.jar -> hdfs://hdp1:9000/user/root/.sparkStaging/application_1548663204656_0001/spark-examples*.jar
// :: INFO yarn.Client: Deleted staging directory hdfs://hdp1:9000/user/root/.sparkStaging/application_1548663204656_0001

2 原因,原因是因为Spark提交任务到yarn集群,需要上传Hadoop相关yarn的jar包

3 解决办法,提前上传到HDFS集群,并且在Spark配置文件指定文件路径,就可以避免每次提交任务到Yarn都需要重复上传文件

4 配置步骤 Spark版本 spark-2.2.0-bin-hadoop2.6

## 打包jars
jar cv0f spark-libs.jar -C $SPARK_HOME/jars/ .
## 新建hdfs路径
hdfs dfs -mkdir -p /spark/jar
## 上传jars到HDFS
hdfs dfs -put spark-libs.jar /spark/jar
## 增加配置
vim spark-defaults.conf
spark.yarn.archive=hdfs:///spark/jar/spark-libs.jar

  

4 重新提交任务,验证结果,没有问题,warn信息消失

root@hdp1 /mnt]#spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 2g --executor-memory 1g --executor-cores  --queue thequeue /mnt/software/spark-2.2.-bin-hadoop2./examples/jars/spark-examples*.jar 

  

  

Spark On Yarn报警告信息 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.的更多相关文章

  1. Spark执行样例报警告:WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources

    搭建Spark环境后,调测Spark样例时,出现下面的错误:WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any res ...

  2. Spark操作HBase报:org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException异常解决方案

    一.异常信息 19/03/21 15:01:52 WARN scheduler.TaskSetManager: Lost task 4.0 in stage 21.0 (TID 14640, hnte ...

  3. spark:neither spark.yarn.jars not spark.yarn.archive is set

    1.Spark启动警告:neither spark.yarn.jars not spark.yarn.archive is set,falling back to uploading librarie ...

  4. Kibana6.x.x——执行yarn build出现的警告信息记录

    Running "_build:installDependencies" task Warning: Command failed: /home/kibana_git/kibana ...

  5. spark运行时出现Neither spark.yarn.jars nor spark.yarn.archive is set错误的解决办法(图文详解)

    不多说,直接上干货! 福利 => 每天都推送 欢迎大家,关注微信扫码并加入我的4个微信公众号:   大数据躺过的坑      Java从入门到架构师      人工智能躺过的坑          ...

  6. 解决log4j:WARN No appenders could be found for logger (org.springframework.web.context.ContextLoader)警告信息的问题

    spring项目经常在启动tomcat时报如下警告信息: log4j:WARN No appenders could be found for logger (org.springframework. ...

  7. spark编译报错信息简介

    spark编译需要环境 git java1.+ maven R 报错信息1: [INFO] ------------------------------------------------------ ...

  8. yarn报错error An unexpected error occurred:****ETIMEDOUT

    起因 在一个美好的下午,我敲击键盘输入quasar create Tangerine_Reception 我自豪的使用yarn作为包管理器,本以为万事无忧,不用担心出现npm的种种异常了...... ...

  9. vue运行时 eslint 报“import/first” WARN deprecated browserslist 问题解决

    vue运行时 eslint 报“import/first”  WARN deprecated browserslist 问题解决 这个信息的意思是导入文件顺序不对,绝对导入应该放在相对导入前面.将绝对 ...

随机推荐

  1. Fiddler 你需要了解的

    官网:http://www.telerik.com/fiddler Fiddler是一个http协议调试代理工具,它能够记录并检查所有你的电脑和互联网之间的HTTP通讯,设置断点,查看所有的“进出”F ...

  2. pycharm 出现 "PEP:8 expected 2 blank lines ,found 0"

    https://blog.csdn.net/modangtian/article/details/79687623 这句话的意思是“有两个空白行,但是没有发现.” 在声明函数的那一行的上方必须有两行的 ...

  3. centos7 mysql-server 安装过程

    官网下载安装mysql-server # wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm # rpm -i ...

  4. pycharm更新之后pip显示没有main

    更新pip之后,Pycharm安装package出现报错:module 'pip' has no attribute 'main' 找到安装目录下 helpers/packaging_tool.py文 ...

  5. commons工具类 FilenameUtils FileUtils

    首先要导入conmmon.jar包 FileUtils类 package cn.lijun.demo2; import java.io.File; import java.io.IOException ...

  6. python自动化开发-[第十八天]-django的ORM补充与ajax,分页器

    今日概要: 1.ORM一对多,多对多 2.正向查询,反向查询 3.聚合查询与分组查询 4.F查询和Q查询 5.ajax 6.分页器 一.ORM补充: django在终端打印sql语句设置: LOGGI ...

  7. 《玩转Django2.0》读书笔记-探究视图

    <玩转Django2.0>读书笔记-探究视图 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 视图(View)是Django的MTV架构模式的V部分,主要负责处理用户请求 ...

  8. 网络编程基础【day09】:socket实现文件发送(六)

    本节内容 1.概述 2.文件下载实现 3.MD5值校验 一.概述 我们如何利用socket去下载一个文件,整体思路是这样的: 读取文件名 检测文件是否存在 打开文件 检测文件大小 发送文件大小给客户端 ...

  9. vue表单校验提交报错TypeError: Cannot read property 'validate' of undefined

    TypeError: Cannot read property 'validate' of undefined at VueComponent.submitForm (plat_users.html: ...

  10. Mac下显示网页全屏快捷键

    control+command+F mac下谷歌浏览器全屏时隐藏头部:(隐藏标签页和地址栏) command+shift+B