yarn默认配置
name | value | description |
yarn.ipc.client.factory.class | Factory to create client IPC classes. | |
yarn.ipc.serializer.type | protocolbuffers | Type of serialization to use. |
yarn.ipc.server.factory.class | Factory to create server IPC classes. | |
yarn.ipc.exception.factory.class | Factory to create IPC exceptions. | |
yarn.ipc.record.factory.class | Factory to create serializeable records. | |
yarn.ipc.rpc.class | org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC | RPC class implementation |
yarn.resourcemanager.hostname | 0.0.0.0 | The hostname of the RM. |
yarn.resourcemanager.address | ${yarn.resourcemanager.hostname}:8032 | The address of the applications manager interface in the RM. |
yarn.resourcemanager.client.thread-count | 50 | The number of threads used to handle applications manager requests. |
yarn.am.liveness-monitor.expiry-interval-ms | 600000 | The expiry interval for application master reporting. |
yarn.resourcemanager.principal | The Kerberos principal for the resource manager. | |
yarn.resourcemanager.scheduler.address | ${yarn.resourcemanager.hostname}:8030 | The address of the scheduler interface. |
yarn.resourcemanager.scheduler.client.thread-count | 50 | Number of threads to handle scheduler interface. |
yarn.http.policy | HTTP_ONLY | This configures the HTTP endpoint for Yarn Daemons.The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https |
yarn.resourcemanager.webapp.address | ${yarn.resourcemanager.hostname}:8088 | The http address of the RM web application. |
yarn.resourcemanager.webapp.https.address | ${yarn.resourcemanager.hostname}:8090 | The https adddress of the RM web application. |
yarn.resourcemanager.resource-tracker.address | ${yarn.resourcemanager.hostname}:8031 | |
yarn.acl.enable | true | Are acls enabled. |
yarn.admin.acl | * | ACL of who can be admin of the YARN cluster. |
yarn.resourcemanager.admin.address | ${yarn.resourcemanager.hostname}:8033 | The address of the RM admin interface. |
yarn.resourcemanager.admin.client.thread-count | 1 | Number of threads used to handle RM admin interface. |
yarn.resourcemanager.amliveliness-monitor.interval-ms | 1000 | How often should the RM check that the AM is still alive. |
yarn.resourcemanager.connect.max-wait.ms | 900000 | Maximum time to wait to establish connection to ResourceManager. |
yarn.resourcemanager.connect.retry-interval.ms | 30000 | How often to try connecting to the ResourceManager. |
yarn.resourcemanager.am.max-attempts | 2 | The maximum number of application attempts. It's a global setting for all application masters. Each application master can specify its individual maximum number of application attempts via the API, but the individual number cannot be more than the global upper bound. If it is, the resourcemanager will override it. The default number is set to 2, to allow at least one retry for AM. |
yarn.resourcemanager.container.liveness-monitor.interval-ms | 600000 | How often to check that containers are still alive. |
yarn.resourcemanager.keytab | /etc/krb5.keytab | The keytab for the resource manager. |
yarn.nm.liveness-monitor.expiry-interval-ms | 600000 | How long to wait until a node manager is considered dead. |
yarn.resourcemanager.nm.liveness-monitor.interval-ms | 1000 | How often to check that node managers are still alive. |
yarn.resourcemanager.nodes.include-path | Path to file with nodes to include. | |
yarn.resourcemanager.nodes.exclude-path | Path to file with nodes to exclude. | |
yarn.resourcemanager.resource-tracker.client.thread-count | 50 | Number of threads to handle resource tracker calls. |
yarn.resourcemanager.scheduler.class | org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler | The class to use as the resource scheduler. |
yarn.scheduler.minimum-allocation-mb | 1024 | The minimum allocation for every container request at the RM, in MBs. Memory requests lower than this won't take effect, and the specified value will get allocated at minimum. |
yarn.scheduler.maximum-allocation-mb | 8192 | The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this won't take effect, and will get capped to this value. |
yarn.scheduler.minimum-allocation-vcores | 1 | The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won't take effect, and the specified value will get allocated the minimum. |
yarn.scheduler.maximum-allocation-vcores | 32 | The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value. |
yarn.resourcemanager.recovery.enabled | false | Enable RM to recover state after starting. If true, then yarn.resourcemanager.store.class must be specified |
yarn.resourcemanager.store.class | org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore | The class to use as the persistent store. |
yarn.resourcemanager.fs.state-store.uri | ${hadoop.tmp.dir}/yarn/system/rmstore | URI pointing to the location of the FileSystem path where RM state will be stored. This must be supplied when using org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore as the value for yarn.resourcemanager.store.class |
yarn.resourcemanager.max-completed-applications | 10000 | The maximum number of completed applications RM keeps. |
yarn.resourcemanager.delayed.delegation-token.removal-interval-ms | 30000 | Interval at which the delayed token removal thread runs |
yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs | 86400 | Interval for the roll over for the master key used to generate application tokens |
yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs | 86400 | Interval for the roll over for the master key used to generate container tokens. It is expected to be much greater than yarn.nm.liveness-monitor.expiry-interval-ms and yarn.rm.container-allocation.expiry-interval-ms. Otherwise the behavior is undefined. |
yarn.resourcemanager.nodemanagers.heartbeat-interval-ms | 1000 | The heart-beat interval in milliseconds for every NodeManager in the cluster. |
yarn.resourcemanager.scheduler.monitor.enable | false | Enable a set of periodic monitors (specified in yarn.resourcemanager.scheduler.monitor.policies) that affect the scheduler. |
yarn.resourcemanager.scheduler.monitor.policies | org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy | The list of SchedulingEditPolicy classes that interact with the scheduler. A particular module may be incompatible with the scheduler, other policies, or a configuration of either. |
yarn.nodemanager.hostname | 0.0.0.0 | The hostname of the NM. |
yarn.nodemanager.address | ${yarn.nodemanager.hostname}:0 | The address of the container manager in the NM. |
yarn.nodemanager.admin-env | MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX | Environment variables that should be forwarded from the NodeManager's environment to the container's. |
yarn.nodemanager.env-whitelist | JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME | Environment variables that containers may override rather than use NodeManager's default. |
yarn.nodemanager.container-executor.class | org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor | who will execute(launch) the containers. |
yarn.nodemanager.container-manager.thread-count | 20 | Number of threads container manager uses. |
yarn.nodemanager.delete.thread-count | 4 | Number of threads used in cleanup. |
yarn.nodemanager.delete.debug-delay-sec | 0 | Number of seconds after an application finishes before the nodemanager's DeletionService will delete the application's localized file directory and log directory. To diagnose Yarn application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property's value, you must restart the nodemanager in order for it to have an effect. The roots of Yarn applications' work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots of the Yarn applications' log directories is configurable with the yarn.nodemanager.log-dirs property (see also below). |
yarn.nodemanager.keytab | /etc/krb5.keytab | Keytab for NM. |
yarn.nodemanager.local-dirs | ${hadoop.tmp.dir}/nm-local-dir | List of directories to store localized files in. An application's localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this. |
yarn.nodemanager.local-cache.max-files-per-directory | 8192 | It limits the maximum number of files which will be localized in a single local directory. If the limit is reached then sub-directories will be created and new files will be localized in them. If it is set to a value less than or equal to 36 [which are sub-directories (0-9 and then a-z)] then NodeManager will fail to start. For example; [for public cache] if this is configured with a value of 40 ( 4 files + 36 sub-directories) and the local-dir is "/tmp/local-dir1" then it will allow 4 files to be created directly inside "/tmp/local-dir1/filecache". For files that are localized further it will create a sub-directory "0" inside "/tmp/local-dir1/filecache" and will localize files inside it until it becomes full. If a file is removed from a sub-directory that is marked full, then that sub-directory will be used back again to localize files. |
yarn.nodemanager.localizer.address | ${yarn.nodemanager.hostname}:8040 | Address where the localizer IPC is. |
yarn.nodemanager.localizer.cache.cleanup.interval-ms | 600000 | Interval in between cache cleanups. |
yarn.nodemanager.localizer.cache.target-size-mb | 10240 | Target size of localizer cache in MB, per local directory. |
yarn.nodemanager.localizer.client.thread-count | 5 | Number of threads to handle localization requests. |
yarn.nodemanager.localizer.fetch.thread-count | 4 | Number of threads to use for localization fetching. |
yarn.nodemanager.log-dirs | ${yarn.log.dir}/userlogs | Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container. |
yarn.log-aggregation-enable | false | Whether to enable log aggregation |
yarn.log-aggregation.retain-seconds | -1 | How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node. |
yarn.log-aggregation.retain-check-interval-seconds | -1 | How long to wait between aggregated log retention checks. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful set this too small and you will spam the name node. |
yarn.nodemanager.log.retain-seconds | 10800 | Time in seconds to retain user logs. Only applicable if log aggregation is disabled |
yarn.nodemanager.remote-app-log-dir | /tmp/logs | Where to aggregate logs to. |
yarn.nodemanager.remote-app-log-dir-suffix | logs | The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam} |
yarn.nodemanager.resource.memory-mb | 8192 | Amount of physical memory, in MB, that can be allocated for containers. |
yarn.nodemanager.pmem-check-enabled | true | Whether physical memory limits will be enforced for containers. |
yarn.nodemanager.vmem-check-enabled | true | Whether virtual memory limits will be enforced for containers. |
yarn.nodemanager.vmem-pmem-ratio | 2.1 | Ratio between virtual memory to physical memory when setting memory limits for containers. Container allocations are expressed in terms of physical memory, and virtual memory usage is allowed to exceed this allocation by this ratio. |
yarn.nodemanager.resource.cpu-vcores | 8 | Number of CPU cores that can be allocated for containers. |
yarn.nodemanager.webapp.address | ${yarn.nodemanager.hostname}:8042 | NM Webapp address. |
yarn.nodemanager.container-monitor.interval-ms | 3000 | How often to monitor containers. |
yarn.nodemanager.container-monitor.resource-calculator.class | Class that calculates containers current resource utilization. | |
yarn.nodemanager.health-checker.interval-ms | 600000 | Frequency of running node health script. |
yarn.nodemanager.health-checker.script.timeout-ms | 1200000 | Script time out period. |
yarn.nodemanager.health-checker.script.path | The health check script to run. | |
yarn.nodemanager.health-checker.script.opts | The arguments to pass to the health check script. | |
yarn.nodemanager.disk-health-checker.interval-ms | 120000 | Frequency of running disk health checker code. |
yarn.nodemanager.disk-health-checker.min-healthy-disks | 0.25 | The minimum fraction of number of disks to be healthy for the nodemanager to launch new containers. This correspond to both yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there are less number of healthy local-dirs (or log-dirs) available, then new containers will not be launched on this node. |
yarn.nodemanager.linux-container-executor.path | The path to the Linux container executor. | |
yarn.nodemanager.linux-container-executor.resources-handler.class | org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler | The class which should help the LCE handle resources. |
yarn.nodemanager.linux-container-executor.cgroups.hierarchy | /hadoop-yarn | The cgroups hierarchy under which to place YARN proccesses (cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount is false (that is, if cgroups have been pre-configured), then this cgroups hierarchy must already exist and be writable by the NodeManager user, otherwise the NodeManager may fail. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler. |
yarn.nodemanager.linux-container-executor.cgroups.mount | false | Whether the LCE should attempt to mount cgroups if not found. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler. |
yarn.nodemanager.linux-container-executor.cgroups.mount-path | Where the LCE should attempt to mount cgroups if not found. Common locations include /sys/fs/cgroup and /cgroup; the default location can vary depending on the Linux distribution in use. This path must exist before the NodeManager is launched. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler, and yarn.nodemanager.linux-container-executor.cgroups.mount is true. | |
yarn.nodemanager.log-aggregation.compression-type | none | T-file compression types used to compress aggregated logs. |
yarn.nodemanager.principal | The kerberos principal for the node manager. | |
yarn.nodemanager.aux-services | the valid service name should only contain a-zA-Z0-9_ and can not start with numbers | |
yarn.nodemanager.sleep-delay-before-sigkill.ms | 250 | No. of ms to wait between sending a SIGTERM and SIGKILL to a container |
yarn.nodemanager.process-kill-wait.ms | 2000 | Max time to wait for a process to come up when trying to cleanup a container |
yarn.nodemanager.resourcemanager.connect.wait.secs | 900 | Max time, in seconds, to wait to establish a connection to RM when NM starts. The NM will shutdown if it cannot connect to RM within the specified max time period. If the value is set as -1, then NM will retry forever. |
yarn.nodemanager.resourcemanager.connect.retry_interval.secs | 30 | Time interval, in seconds, between each NM attempt to connect to RM. |
yarn.client.nodemanager-client-async.thread-pool-max-size | 500 | Max number of threads in NMClientAsync to process container management events |
yarn.client.max-nodemanagers-proxies | 500 | Maximum number of proxy connections for node manager. It should always be more than 1. NMClient and MRAppMaster will use this to cache connection with node manager. There will be at max one connection per node manager. Ex. configuring it to a value of 5 will make sure that client will at max have 5 connections cached with 5 different node managers. These connections will be timed out if idle for more than system wide idle timeout period. The token if used for authentication then it will be used only at connection creation time. If new token is received then earlier connection should be closed in order to use newer token. This and (yarn.client.nodemanager-client-async.thread-pool-max-size) are related and should be sync (no need for them to be equal). |
yarn.nodemanager.aux-services.mapreduce_shuffle.class | org.apache.hadoop.mapred.ShuffleHandler | |
mapreduce.job.jar | ||
mapreduce.job.hdfs-servers | ${fs.defaultFS} | |
yarn.web-proxy.principal | The kerberos principal for the proxy, if the proxy is not running as part of the RM. | |
yarn.web-proxy.keytab | Keytab for WebAppProxy, if the proxy is not running as part of the RM. | |
yarn.web-proxy.address | The address for the web proxy as HOST:PORT, if this is not given then the proxy will run as part of the RM | |
yarn.application.classpath | $HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/* | CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries |
yarn.client.app-submission.poll-interval | 1000 |
yarn默认配置的更多相关文章
- Flink on yarn的配置及执行
1. 写在前面 Flink被誉为第四代大数据计算引擎组件,即可以用作基于离线分布式计算,也可以应用于实时计算.Flink可以自己搭建集群模式已提供为庞大数据的计算.但在实际应用中.都是计算hdfs上的 ...
- 【原创】大叔经验分享(9)yarn重要配置yarn.nodemanager.local-dirs
yarn中有一个比较重要的配置yarn.nodemanager.local-dirs,如果配置的不好,在饱和状态运行下集群会出现很多问题:1 默认配置${hadoop.tmp.dir}/nm-loca ...
- 第7章 YARN HA配置
目录 7.1 yarn-site.xm文件配置 7.2 测试YARN自动故障转移 ResourceManager (RM)负责跟踪集群中的资源,以及调度应用程序(例如,MapReduce作业).在Ha ...
- MapReduce On Yarn的配置详解和日常维护
MapReduce On Yarn的配置详解和日常维护 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.MapReduce运维概述 MapReduce on YARN的运维主要是 ...
- php获取时间问题,用默认配置读到本地时间。。。。。
用date获取到时间有8小时时差 因为php用date获取到的是格林威治时区的时间,而大陆时间和格林威治时间有8个小时时差,所以.... 修改:网上有各种修改方式: 比如:在程序中添加时间的初始化的语 ...
- Asp.net默认配置下,Session莫名丢失的原因及解决
Asp.net默认配置下,Session莫名丢失的原因及解决 我们平时写的asp.net程序,里面要用到Session来保存一些跨页面的数据.但是Session会经常无故丢失,上网查查,也没找到原因. ...
- hadoop集群默认配置和常用配置【转】
转自http://www.cnblogs.com/ggjucheng/archive/2012/04/17/2454590.html 获取默认配置 配置hadoop,主要是配置core-site.xm ...
- EXT经验--查询EditorGridPanel的tbar的默认配置对象
前言:EXT的API可谓熟悉EXT的葵花宝典,会看API可谓对于配置EXT,学习EXT最重要的基本功,这点相对于学习轻量级的Easyui来说更加明显. 比如下面的一段代码:注:在Ext.grid.Ed ...
- jquery.validate.js默认配置,jquery.validate.js自定义提示信息
jquery.validate.js默认配置,jquery.validate.js自定义提示信息 配置jQuery.validator默认的处理方法 >>>>>>& ...
随机推荐
- Mavne 打包时出现程序包找到不的问题
<plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactI ...
- js 省份城市二级动态联动的例子
<!DOCTYPE html><html lang="en"><head> <meta charset="UTF-8" ...
- module.exports和 exports 方法暴露
在Node.js包管理时需要把方法暴露给外部文件 文件:Hello.js Hello方法是模仿面向对象类的写法 function Hello() { var name; this.setName ...
- 4.net两种交互模式
.net两种交互模式 (1) C/S:客户端(Client)/服务器模式(Server) (2) B/S:浏览器(Browser)/服务器模式(Server) 来自为知笔记(Wiz)
- vue——计算属性和侦听器
一.计算属性(data中的相关数据) 侦听多个属性时——计算属性 comuted. 模板内的表达式非常便利,但是设计它们的初衷是用于简单运算的.在模板中放入太多的逻辑会让模板过重且难以维护.例如: & ...
- rem,em,px
http://pxtoem.com/ px,em,rem单位转换工具 px(Pixel)根据显示屏分辨率,固定大小 em,根据父元素大小进行改变 rem,根据html进行相对改变 em 1. bod ...
- easyui datagrid 显示 footer
1.设置 showFooter为true $grid = $dg.datagrid({ // fit: true, //fitColumns: true, // pagination: false, ...
- canvas的getImageData和putImageDataAPI
- Personal Geodatabase - Can't Create New or Open Existing
来自:http://forums.arcgis.com/threads/32110-Personal-Geodatabase-Can-t-Create-New-or-Open-Existing Per ...
- ANR android
1.android ANR产生原因和解决办法 2.Android ANR异常及解决方法 3.Android ANR 分析解决方法 4.[原创]Android 系统稳定性 - ANR(一) 5.[原创] ...