1.spark submit参数

$ ./bin/spark-submit --class path.to.your.Class --master yarn --deploy-mode cluster [options] <app jar> [app options]
Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default
Usage: spark-submit [options] <app jar | python file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...] Options:
--master MASTER_URL spark://host:port, mesos://host:port, yarn, or local.
--deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or
on one of the worker machines inside the cluster ("cluster")
(Default: client).
--class CLASS_NAME Your application's main class (for Java / Scala apps).
--name NAME A name of your application.
--jars JARS Comma-separated list of local jars to include on the driver
and executor classpaths.
--packages Comma-separated list of maven coordinates of jars to include
on the driver and executor classpaths. Will search the local
maven repo, then maven central and any additional remote
repositories given by --repositories. The format for the
coordinates should be groupId:artifactId:version.
--exclude-packages Comma-separated list of groupId:artifactId, to exclude while
resolving the dependencies provided in --packages to avoid
dependency conflicts.
--repositories Comma-separated list of additional remote repositories to
search for the maven coordinates given with --packages.
--py-files PY_FILES Comma-separated list of .zip, .egg, or .py files to place
on the PYTHONPATH for Python apps.
--files FILES Comma-separated list of files to be placed in the working
directory of each executor. --conf PROP=VALUE Arbitrary Spark configuration property.
--properties-file FILE Path to a file from which to load extra properties. If not
specified, this will look for conf/spark-defaults.conf. --driver-memory MEM Memory for driver (e.g. 1000M, 2G) (Default: 1024M).
--driver-java-options Extra Java options to pass to the driver.
--driver-library-path Extra library path entries to pass to the driver.
--driver-class-path Extra class path entries to pass to the driver. Note that
jars added with --jars are automatically included in the
classpath. --executor-memory MEM Memory per executor (e.g. 1000M, 2G) (Default: 1G). --proxy-user NAME User to impersonate when submitting the application.
This argument does not work with --principal / --keytab. --help, -h Show this help message and exit
--verbose, -v Print additional debug output
--version, Print the version of current Spark Spark standalone with cluster deploy mode only:
--driver-cores NUM Cores for driver (Default: ). Spark standalone or Mesos with cluster deploy mode only:
--supervise If given, restarts the driver on failure.
--kill SUBMISSION_ID If given, kills the driver specified.
--status SUBMISSION_ID If given, requests the status of the driver specified. Spark standalone and Mesos only:
--total-executor-cores NUM Total cores for all executors. Spark standalone and YARN only:
--executor-cores NUM Number of cores per executor. (Default: in YARN mode,
or all available cores on the worker in standalone mode) YARN-only:
--driver-cores NUM Number of cores used by the driver, only in cluster mode
(Default: ).
--queue QUEUE_NAME The YARN queue to submit to (Default: "default").
--num-executors NUM Number of executors to launch (Default: ).
--archives ARCHIVES Comma separated list of archives to be extracted into the
working directory of each executor.
--principal PRINCIPAL Principal to be used to login to KDC, while running on
secure HDFS.
--keytab KEYTAB The full path to the file that contains the keytab for the
principal specified above. This keytab will be copied to
the node running the Application Master via the Secure
Distributed Cache, for renewing the login tickets and the
delegation tokens periodically.

For example:

$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--driver-memory 4g \
--executor-memory 2g \
--executor-cores \
--queue thequeue \
lib/spark-examples*.jar \

In client mode:

$ ./bin/spark-shell --master yarn --deploy-mode client

最近一直测试spark程序,对spark-submit的参数进行了调节。

通过上面图片可以看到,总共有7个节点,总共的VCores为133,总共内存为1.49TB,有3个application在运行,2个分别为spark thrift server和spark2 thrift server,1个为我提交的任务,可以看出我提交的任务占用了81个cpu VCores(1个为application master)。

提交命令:

time spark-submit --master yarn-client --driver-memory 10g --executor-memory 10g --num-executors  --class com.test.test.ByHour ~/cs0308/quickstart-SNAPSHOT.jar /test/origin/20170306_02 /test/result/

其中

--num-executors 80           80个executor(默认是2个)
--executor-memory 10g        exector内存大小(默认1g)
--driver-memory 10g          Driver程序使用内存大小

--executor-cores             每个executor使用的内核数,默认为1

发现在设置过程中运行起来还有剩余的资源,但是若把参数直接设成80以上,程序一运行就直接报错,程序刚运行时申请资源会超过设置参数,然后就会下降80(设置参数)+1个cpu,1个为Application Master。这个是跑了1.2T大小的gz文件,跑完为4.6T的txt文件。
同样的程序在第二天跑的时候,发现老报sockettimeoutexception,重启集群(没有程序在运行)后,再一次运行同样的命令发现正常了(用的HDP的集群)。 具体spark-submit参数说明: https://my.oschina.net/u/140462/blog/519409

Spark on YARN资源申请的更多相关文章

  1. Apache Spark源码走读之8 -- Spark on Yarn

    欢迎转载,转载请注明出处,徽沪一郎. 概要 Hadoop2中的Yarn是一个分布式计算资源的管理平台,由于其有极好的模型抽象,非常有可能成为分布式计算资源管理的事实标准.其主要职责将是分布式计算集群的 ...

  2. Spark On Yarn的两种模式yarn-cluster和yarn-client深度剖析

    Spark On Yarn的优势 每个Spark executor作为一个YARN容器(container)运行.Spark可以使得多个Tasks在同一个容器(container)里面运行 1. Sp ...

  3. <YARN><MRv2><Spark on YARN>

    MRv1 VS MRv2 MRv1: - JobTracker: 资源管理 & 作业控制- 每个作业由一个JobInProgress控制,每个任务由一个TaskInProgress控制.由于每 ...

  4. 基础概念 之 Spark on Yarn

    先抛出问题:Spark on Yarn有cluster和client两种模式,它们有什么区别? 用Jupyter写Spark时,只能使用client模式,为什么? 写一篇文章,搞清楚 Spark on ...

  5. Spark on YARN的部署

    Spark on YARN的原理就是依靠yarn来调度Spark,比默认的Spark运行模式性能要好的多,前提是首先部署好hadoop HDFS并且运行在yarn上,然后就可以开始部署spark on ...

  6. 配置Spark on YARN集群内存

    参考原文:http://blog.javachen.com/2015/06/09/memory-in-spark-on-yarn.html?utm_source=tuicool 运行文件有几个G大,默 ...

  7. Spark on Yarn 学习(一)

    最近看到明风的关于数据挖掘平台下实用Spark和Yarn来做推荐的PPT,感觉很赞,现在基于大数据和快速计算方面技术的发展很快,随着Apache基金会上发布的一个个项目,感觉真的新技术将会不断出现在大 ...

  8. Spark on Yarn:任务提交参数配置

    当在YARN上运行Spark作业,每个Spark executor作为一个YARN容器运行.Spark可以使得多个Tasks在同一个容器里面运行. 以下参数配置为例子: spark-submit -- ...

  9. 运行 Spark on YARN

    运行 Spark on YARN Spark 0.6.0 以上的版本添加了在yarn上执行spark application的功能支持,并在之后的版本中持续的 改进.关于本文的内容是翻译官网的内容,大 ...

随机推荐

  1. How to setup Assigned Access in Windows 10 (Kiosk Mode) 设置分配的访问权限(Kiosk模式)

    Let’s say you’re building some sort of ingenious mechanical contraption to be displayed in public th ...

  2. 在Visual Studio2015中使用单元测试

    所谓的单元测试(Unit Test),就是对软件的一些模块进行测试以检查其正确性和可靠性,这些模块可以是一个类或者是一个方法等.在Visual studio中,这十分容易实现. 打开Visual st ...

  3. 笨重的mfc还在基于系统控件,熟练的mfc工程师还比不过学习Qt一个月的学生开发效率高(比较精彩,韦易笑)

    作者:韦易笑链接:https://www.zhihu.com/question/29636221/answer/45102191来源:知乎著作权归作者所有,转载请联系作者获得授权. 更新:擦,本来只有 ...

  4. QTableWidget简单应用之文件浏览器

     实现下面这个简单的文件浏览功能,常用的QTableWidget设置都用到了.  基本设置 ui->tableWidget->setColumnCount(5); //设置表格列数 ui- ...

  5. qt获得本地IP的方法,qt中域名解析的方法

    本文博客链接:http://blog.csdn.net/jdh99,作者:jdh,转载请注明. 环境:Ubuntu10.04 + Qt4.7.0 Linux获得本地IP的方法,我尝试了两种 1.用QH ...

  6. C# 简单TCP协议

    namespace TCPServer { class Program { static void Main(string[] args) { ;//端口 TcpClient tcpClient;// ...

  7. 《大型网站系统与Java中间件》读书笔记 (中)

    前言 只有光头才能变强. 文本已收录至我的GitHub仓库,欢迎Star:https://github.com/ZhongFuCheng3y/3y 回顾上一篇: <大型网站系统与Java中间件& ...

  8. H5 离线缓存的用法

    H5离线缓存基础系列   1.什么是离线缓存 离线缓存:离线缓存可以将站点的一些文件缓存到本地,它是浏览器自己的一种机制,将需要的文件缓存下来,以便后期即使没有连接网络,被缓存的页面也可以展示. 2. ...

  9. mount -o remount,rw /命令什么意思

    1.关于如何改变linux中,对只有Read-only filesystem的文件,如何改为为可写.可读权限? I.可以执行 mount -o remount,rw /II.执行完毕后,会出现以下字样 ...

  10. 02-三种Bean装配机制(一)

    Spring要创建哪些bean并且如何将其装配在一起,现有的,主要是三种装配机制: 自动化装配 通过java代码装配 通过XML装配 这三种装配机制可以互相搭配使用,即是可以共存的.接下来就分别介绍啦