EMR的fair-scheduler.xml
cat /etc/ecm/hadoop-conf/fair-scheduler.xml
<?xml version="1.0"?>
<allocations>
<aclSubmitApps>*</aclSubmitApps>
<weight>2</weight>
<minResources>10000 mb, 10vcores</minResources>
<maxChildResources>34000 mb,24 vcores</maxChildResources>
<maxRunningApps>50</maxRunningApps>
<maxAMShare>1</maxAMShare>
<maxResources>400000 mb, 200vcores</maxResources> #限制队列最大使用资源
<aclAdministerApps>*</aclAdministerApps>
<schedulingPolicy>fair</schedulingPolicy>
<queue name="default">
<aclSubmitApps>*</aclSubmitApps>
<minResources>10000 mb, 10vcores</minResources>
<aclAdministerApps>*</aclAdministerApps>
<weight>1</weight>
<maxRunningApps>10</maxRunningApps>
<maxAMShare>0.5</maxAMShare>
<maxResources>200000 mb, 100vcores</maxResources>
</queue>
<queue name="collects">
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
<weight>8</weight>
<maxAMShare>0.8</maxAMShare>
<minResources>50 mb, 2vcores</minResources>
<maxResources>400000 mb, 200vcores</maxResources>
<maxRunningApps>50</maxRunningApps>
</queue>
<queue name="data_bi">
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
<weight>4</weight>
<minResources>100 mb, 1vcores</minResources>
<maxResources>30000 mb, 50vcores</maxResources>
<maxRunningApps>5</maxRunningApps>
</queue>
<queue name="opay_collects">
<weight>20</weight>
<minResources>10 mb, 1vcores</minResources>
<maxResources>400000 mb, 200vcores</maxResources>
<maxRunningApps>20</maxRunningApps>
<maxAMShare>0.5</maxAMShare>
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
</queue>
<queue name="opos_collects">
<weight>5</weight>
<minResources>10 mb, 1vcores</minResources>
<maxResources>80000 mb, 50vcores</maxResources>
<maxRunningApps>10</maxRunningApps>
</queue>
<queue name="users" type="parent">
<weight>5</weight>
<minResources>10 mb, 1vcores</minResources>
<maxResources>10000 mb, 150vcores</maxResources>
<maxRunningApps>30</maxRunningApps>
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
</queue>
<queue name="airflow">
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
<weight>8</weight>
<minResources>10 mb, 2vcores</minResources>
<maxResources>200000 mb, 150vcores</maxResources>
<maxRunningApps>30</maxRunningApps>
</queue>
<defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
<userMaxAppsDefault>50</userMaxAppsDefault>
<queueMaxAppsDefault>50</queueMaxAppsDefault>
<queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>
<defaultFairSharePreemptionThreshold>0.5</defaultFairSharePreemptionThreshold>
<queueMaxResourcesDefault>34000 mb,24vcores</queueMaxResourcesDefault>
<defaultFairSharePreemptionTimeout>9223372036854775807</defaultFairSharePreemptionTimeout>
<defaultMinSharePreemptionTimeout>9223372036854775807</defaultMinSharePreemptionTimeout>
</allocations>
#新的xml, 不带root限制: 放emr的yarn-配置-fair-scheduler
<?xml version="1.0" encoding="utf-8"?>
<allocations>
<queue name="root">
<queue name="default">
<aclSubmitApps>*</aclSubmitApps>
<minResources>10000 mb, 10vcores</minResources>
<aclAdministerApps>*</aclAdministerApps>
<weight>1</weight>
<maxRunningApps>10</maxRunningApps>
<maxAMShare>0.5</maxAMShare>
<maxResources>200000 mb, 100vcores</maxResources>
</queue>
<queue name="collects">
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
<weight>8</weight>
<maxAMShare>0.8</maxAMShare>
<minResources>50 mb, 2vcores</minResources>
<maxResources>400000 mb, 200vcores</maxResources>
<maxRunningApps>50</maxRunningApps>
</queue>
<queue name="data_bi">
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
<weight>4</weight>
<minResources>100 mb, 1vcores</minResources>
<maxResources>30000 mb, 50vcores</maxResources>
<maxRunningApps>5</maxRunningApps>
</queue>
<queue name="opay_collects">
<weight>20</weight>
<minResources>10 mb, 1vcores</minResources>
<maxResources>400000 mb, 200vcores</maxResources>
<maxRunningApps>20</maxRunningApps>
<maxAMShare>0.5</maxAMShare>
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
</queue>
<queue name="opos_collects">
<weight>5</weight>
<minResources>10 mb, 1vcores</minResources>
<maxResources>80000 mb, 50vcores</maxResources>
<maxRunningApps>10</maxRunningApps>
</queue>
<queue name="users" type="parent">
<weight>5</weight>
<minResources>10 mb, 1vcores</minResources>
<maxResources>10000 mb, 150vcores</maxResources>
<maxRunningApps>30</maxRunningApps>
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
</queue>
<queue name="airflow">
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>*</aclSubmitApps>
<aclAdministerApps>*</aclAdministerApps>
<weight>8</weight>
<minResources>10 mb, 2vcores</minResources>
<maxResources>200000 mb, 150vcores</maxResources>
<maxRunningApps>30</maxRunningApps>
</queue>
</queue>
<defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
<userMaxAppsDefault>50</userMaxAppsDefault>
<queueMaxAppsDefault>50</queueMaxAppsDefault>
<queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>
<defaultFairSharePreemptionThreshold>0.5</defaultFairSharePreemptionThreshold>
<queueMaxResourcesDefault>34000 mb,24vcores</queueMaxResourcesDefault>
<defaultFairSharePreemptionTimeout>9223372036854775807</defaultFairSharePreemptionTimeout>
<defaultMinSharePreemptionTimeout>9223372036854775807</defaultMinSharePreemptionTimeout>
</allocations>
EMR的fair-scheduler.xml的更多相关文章
- Fair Scheduler 队列设置经验总结
Fair Scheduler 队列设置经验总结 由于公司的hadoop集群的计算资源不是很充足,需要开启yarn资源队列的资源抢占.在使用过程中,才明白资源抢占的一些特点.在这里总结一下. 只有一个队 ...
- 三：Fair Scheduler 公平调度器
参考资料: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html http://h ...
- Fair Scheduler中的Delay Schedule分析
延迟调度的主要目的是提高数据本地性(data locality),减少数据在网络中的传输.对于那些输入数据不在本地的MapTask,调度器将会延迟调度他们,而把slot分配给那些具备本地性的MapTa ...
- Hadoop学习之--Fair Scheduler作业调度分析
Fair Scheduler调度器同步心跳分配任务的过程简单来讲会经历以下环节: 1. 对map/reduce是否已经达到资源上限的循环判断 2. 对pool队列根据Fair算法排序 3.然后循环po ...
- YARN的Fair Scheduler和Capacity Scheduler
关于Scheduler YARN有四种调度机制:Fair Schedule,Capacity Schedule,FIFO以及Priority: 其中Fair Scheduler是资源池机制,进入到里面 ...
- Hadoop的三种调度器FIFO、Capacity Scheduler、Fair Scheduler（转载）
目前Hadoop有三种比较流行的资源调度器:FIFO .Capacity Scheduler.Fair Scheduler.目前Hadoop2.7默认使用的是Capacity Scheduler容量调 ...
- fair scheduler配置
<property> <name>yarn.resourcemanager.scheduler.class</name> <value>or ...
- Yarn参数优化(Fair Scheduler版本)
YARN 自从hadoop2.0之后, 我们可以使用apache yarn 来对集群资源进行管理.yarn把可以把资源(内存,CPU)以Container的方式进行划分隔离.YARN会管理集群中所有机 ...
- Linux 2.6 完全公平调度算法CFS（Completely Fair Scheduler）
分析
转会http://www.ibm.com/developerworks/cn/linux/l-completely-fair-scheduler/index.html? ca=drs-cn-0125 ...
- 利用yarn capacity scheduler在EMR集群上实现大集群的多租户的集群资源隔离和quota限制
转自:https://m.aliyun.com/yunqi/articles/79700 背景 使用过hadoop的人基本都会考虑集群里面资源的调度和优先级的问题,假设你现在所在的公司有一个大hado ...
随机推荐
- BZOJ4406 WC2016 论战捆竹竿
Problem BZOJ Solution 显然是一个同余系最短路问题,转移方案就是所有|S|-border的长度,有 \(O(n)\) 种,暴力跑dijkstra的复杂度为 \(O(n^2\log ...
- Visual Studio2019安装步骤
学校使用的版本是2012版本,而现在讲的版本是2019版本,差别不大,个人认为2019更能胜任学习任务. 另外VS2019是完全免费的,版本越高越好了!毕竟C++都出了C++20对吧. Step 1: ...
- [Linux] 创建、删除用户
系统:Ubuntu useradd:创建新用户或更新默认新用户信息 创建新用户 xxx,/home 目录没有同名文件夹,并且此用户没有 shell 权限 $ sudo useradd xxx 创建新用 ...
- java实现哈夫曼树进行文件加解压
目录 1.哈夫曼树简述 2.构造树的节点 3.构造哈夫曼树的类(压缩) 4.构造哈夫曼树的类(解压) 5.整体工程文件(包括测试类) 6.结果 7.参考链接 1.哈夫曼树简述 给定n个权值作为n个叶子 ...
- Mininet系列实验（四）：基于Mininet测量路径的损耗率
1 实验目的 熟悉Mininet自定义拓扑脚本的编写与损耗率的设定: 熟悉编写POX脚本,测量路径损耗速率 2 实验原理 在SDN环境中,控制器可以通过对交换机下发流表操作来控制交换机的转发行为,此外 ...
- JavaWeb基础知识
一.WEB基本概念 1.1.WEB开发的相关知识 WEB,在英语中web即表示网页的意思,它用于表示Internet主机上供外界访问的资源. Internet上供外界访问的Web资源分为: 静态web ...
- 在不切换分支的情况下，如何在all branches中快速查看指定分支，相对其他分支的状态
假设需要查看的分支为temp 1.git show temp 通过这个可以拿到commit id 2.查看TortoiseGit的日志,左下角勾选所有分支 3.在日志界面搜索commit id,然后右 ...
- Apache RocketMQ 的过去、现在和未来 原创： DataPipeline DataPipeline数见科技 前天
Apache RocketMQ 的过去.现在和未来 原创: DataPipeline DataPipeline数见科技 前天
- 用了pm2之后，如何让console.log('..')出现在控制台
使用 pm2 log 0 可以查看.当然这个 0 应该是应用的 id
- java 虚拟机类加载 及内存结构
http://www.jb51.net/article/105920.htm https://www.cnblogs.com/Qian123/p/5707562.html Java类加载全过程 一个j ...