Spark On Yarn报警告信息 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.

1 贴出完整日志信息

// :: INFO client.RMProxy: Connecting to ResourceManager at hdp1/192.168.56.201:
// :: INFO yarn.Client: Requesting a new application from cluster with  NodeManagers
// :: INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster ( MB per container)
// :: INFO yarn.Client: Will allocate AM container, with  MB memory including  MB overhead
// :: INFO yarn.Client: Setting up container launch context for our AM
// :: INFO yarn.Client: Setting up the launch environment for our AM container
// :: INFO yarn.Client: Preparing resources for our AM container
// :: WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
// :: INFO yarn.Client: Uploading resource file:/mnt/software/spark-2.2.-bin-hadoop2./spark-3045ce36-b8d7--a9aa-e0509ef0ec89/__spark_libs__8280328345325459092.zip -> hdfs://hdp1:9000/user/root/.sparkStaging/application_1548663204656_0001/__spark_libs__8280328345325459092.zip
// :: INFO yarn.Client: Uploading resource file:/mnt/software/examples/jars/spark-examples*.jar -> hdfs://hdp1:9000/user/root/.sparkStaging/application_1548663204656_0001/spark-examples*.jar
// :: INFO yarn.Client: Deleted staging directory hdfs://hdp1:9000/user/root/.sparkStaging/application_1548663204656_0001

2 原因，原因是因为Spark提交任务到yarn集群，需要上传Hadoop相关yarn的jar包

3 解决办法，提前上传到HDFS集群，并且在Spark配置文件指定文件路径，就可以避免每次提交任务到Yarn都需要重复上传文件

4 配置步骤 Spark版本 spark-2.2.0-bin-hadoop2.6

## 打包jars
jar cv0f spark-libs.jar -C $SPARK_HOME/jars/ .
## 新建hdfs路径
 hdfs dfs -mkdir -p /spark/jar
## 上传jars到HDFS
hdfs dfs -put spark-libs.jar /spark/jar
## 增加配置
vim spark-defaults.conf
spark.yarn.archive=hdfs:///spark/jar/spark-libs.jar

　　

4 重新提交任务，验证结果,没有问题，warn信息消失

root@hdp1 /mnt]#spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 2g --executor-memory 1g --executor-cores  --queue thequeue /mnt/software/spark-2.2.-bin-hadoop2./examples/jars/spark-examples*.jar