YARN Architecture

The fundamental idea of YARN is to split up the functionalities of resource management and job scheduling/monitoring into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job or a DAG of jobs.

The ResourceManager and the NodeManager form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The NodeManager is the per-machine framework agent who is responsible for containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the ResourceManager/Scheduler.

The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.

YARN的基本构想是将资源管理器和作业调度器/监控器分开成两个单独的进程。这个想法是为了拥有一个全局的资源管理器（RM）和每一个应用都有一个应用控制器。应用可以是一个单独的作业也可以是一组作业。

ResourceManager和NodeManager构成数据计算框架。RM是最终的权威仲裁系统中的所有应用的资源分配。NodeManager是框架在每台机器中负责containers的代理，监控它们的资源使用（内存、CPU、磁盘和网络）和将其汇报给ResourceManager/调度器。监控它们的资源使用（内存、CPU、磁盘和网络）和将其汇报给ResourceManager/调度器。

每个应用程序的ApplicationMaster实际上是框架指定的库负责从RM谈判获取资源并和MM一起工作来执行和监控任务。

The ResourceManager has two main components: Scheduler and ApplicationsManager.

The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees about restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a resource Container which incorporates elements such as memory, cpu, disk, network etc.

The Scheduler has a pluggable policy which is responsible for partitioning the cluster resources among the various queues, applications etc. The current schedulers such as the CapacityScheduler and the FairScheduler would be some examples of plug-ins.

The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure. The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.

MapReduce in Hadoop-2.x maintains API compatibility with previous stable release (hadoop-1.x). This means that all MapReduce jobs should still run unchanged on top of YARN with just a recompile.

ResourceManager有两个主要的组成部分：调度器和应用管理器。

调度器负责给各个正在运行的拥有相似的约束如容量，队列等的应用分配资源。调度器是一个纯粹的调度器而不负责监控或者跟踪应用的状态。他也不负责恢复由于应用失效或者硬件失效而失败的任务。调度器根据应用的资源需求来执行它的调度。而不是根据一个抽象资源“容器”包含的元素例如内存、CPU、磁盘和网络等

调度器是一个可插拔的组件负责将资源分配给各种各样的队列、应用等。目前的容量调度器和公平调度器将成为一些插件的例子。

应用管理器负责接收作业的提交、选择第一个容器用来运行应用指定的应用控制器和提供当ApplicationMaster容器失效时的重启。每个应用的ApplicationMaster负责从调度器那里谈判获取合适的资源容器，跟踪他们的状态和监控过程。

hadoop-2.x中的MapReduce兼容前面稳定的版本（hadoop-1.x）。这就意味着所有的MapReduce作业只需要再编译一次无需做任何改变就可以运行在YARN上。