Apache Tez Design
http://tez.incubator.apache.org/
http://dongxicheng.org/mapreduce-nextgen/apache-tez/
http://dongxicheng.org/mapreduce-nextgen/apache-tez-newest-progress/
Tez aims to be a general purpose execution runtime that enhances various scenarios that are not well served by classic Map-Reduce.
In the short term the major focus is to support Hive and Pig, specifically to enable performance improvements to batch and ad-hoc interactive queries.
What services will Tez provide
Tez兼容传统的map-reduce jobs, 当然主要focus提供基于DAG的jobs和相应的API以及primitives.
Tez provides runtime components:
- An execution environment that can handle traditional map-reduce jobs
- An execution environment that handles DAG-based jobs comprising various built-in and extendable primitives
- Cluster-side determination of input pieces
- Runtime planning such as task cardinality determination and dynamic modification to the DAG structure
Tez provides APIs to access these services:
- Traditional map-reduce functionality is accessed via java classes written to the Job interface: org.apache.hadoop.mapred.Job and/or org.apache.hadoop.mapreduce.v2.app.job.Job;
and by specifying in yarn-site that the map-reduce framework should be Tez. - DAG-based execution is accessed via the new Tez DAG API: org.apache.tez.dag.api.*, org.apache.tez.engine.api.*.
Tez provides pre-made primitives for use with the DAG API (org.apache.tez.engine.common.*)
- Vertex Input
- Vertex Output
- Sorting
- Shuffling
- Merging
- Data transfer
Tez-YARN architecture
In the above figure Tez is represented by the red components: client-side API, an AppMaster, and multiple containers that execute child processes under the control of the AppMaster.
Three separate software stacks are involved in the execution of a Tez job, each using components from the clientapplication, Tez, and YARN:
DAG topologies and scenarios
The following terminology is used:
Job Vertex: A “stage” in the job plan. 逻辑顶点, 可以理解成stage
Job Edge: The logical connections between Job Vertices. 逻辑边, 关联
Vertex: A materialized stage at runtime comprising a certain number of materialized tasks. 物理顶点, 由并行的tasks节点组成
Edge: Represents actual data movement between tasks. 物理边, 代表实际数据流向
Task: A process performing computation within a YARN container. Task, 一个执行节点
Task cardinality: The number of materialized tasks in a Vertex. Task基数, Vertex的并发度
Static plan: Planning decisions fixed before job submission.
Dynamic plan: Planning decisions made at runtime in the AppMaster process.
Tez API
The Tez API comprises many services that support applications to run DAG-style jobs. An application that makes use of Tez will need to:
1. Create a job plan (the DAG) comprising vertices, edges, and data source references
2. Create task implementations that perform computations and interact with the DAG AppMaster
3. Configure Yarn and Tez appropriately
DAG definition API
抽象DAG的定义接口
public class DAG{
DAG();
void addVertex(Vertex);
void addEdge(Edge);
void addConfiguration(String, String);
void setName(String);
void verify();
DAGPlan createDaG();
} public class Vertex {
Vertex(String vertexName, String processorName, int parallelism);
void setTaskResource();
void setTaskLocationsHint(TaskLocationHint[]);
void setJavaOpts(String);
String getVertexName();
String getProcessorName();
int getParallelism();
Resource getTaskResource();
TaskLocationHint[] getTaskLocationsHint();
String getJavaOpts();
} public class Edge {
Edge(Vertex inputVertex, Vertex outputVertex, EdgeProperty edgeProperty);
String getInputVertex();
String getOutputVertex();
EdgeProperty getEdgeProperty();
String getId();
}
Execution APIs
Task作为Tez的执行者, 遵循input, output, processor的模式
public interface Master
//a context object for task execution. currently only stub public interface Input{
void initialize(Configuration conf, Master master)
boolean hasNext()
Object getNextKey()
Iterable<Object> getNextValues()
float getProgress()
void close()
} public interface Output{
void initialize(Configuration conf, Master master);
void write(Object key, Object value);
OutputContext getOutputContext();
void close();
} public interface Partitioner {
int getPartition(Object key, Object value, int numPartitions);
} public interface Processor {
void initialize(Configuration conf, Master master)
void process(Input[] in, Output[] out)
void close()
} public interface Task{
void initialize(Configuration conf, Master master)
Input[] getInputs();
Processor getProcessor();
Output[] getOutputs();
void run()
void close()
}
Apache Tez Design的更多相关文章
- CentOS 6.5 Maven 编译 Apache Tez 0.8.3 踩坑/报错解决记录
最近准备学习使用Tez,因此从官网下载了最新的Tez 0.8.3源码,按照安装教程编译使用.平时使用的集群环境是离线的,本打算这一次也进行离线编译,无奈一编译就开始报缺少jar包的错,即使手动下载ja ...
- Apache Tez 了解
你可能听说过Apache Tez,它是一个针对Hadoop数据处理应用程序的新分布式执行框架.但是它到底是什么呢?它的工作原理是什么?哪些人应该使用它,为什么?如果你有这些疑问,那么可以看一下Bika ...
- Apache Tez 0.7、0.83、 0.82 安装、调试笔记
———————————————————— 准备 Tez 编译环境 ———————————————————— 1 需要的支持 tez0.7 需要 Hadoop 2.60 以上 2 需要的 linux 相 ...
- Apache Tez on hive
———————————————————— 调配 Hadoop ———————————————————— 1 将 编译好的 TEZ .tar.gz 文件上传到 HDFS 中. hdfs fs -p ...
- Big Data资料汇总
整理和翻新一下自己看过和笔记过的Big Data相关的论文和Blog Streaming & Spark In-Stream Big Data Processing Discretized S ...
- apache开源项目 -- tez
为了更高效地运行存在依赖关系的作业(比如Pig和Hive产生的MapReduce作业),减少磁盘和网络IO,Hortonworks开发了DAG计 算框架Tez.Tez是从MapReduce计算框架演化 ...
- Hadoop2.0/YARN深入浅出(Hadoop2.0、Spark、Storm和Tez)
随着云计算.大数据迅速发展,亟需用hadoop解决大数据量高并发访问的瓶颈.谷歌.淘宝.百度.京东等底层都应用hadoop.越来越多的企 业急需引入hadoop技术人才.由于掌握Hadoop技术的开发 ...
- Apache 项目列表功能分类便于技术选型
big-data (49): Apache Accumulo Apache Airavata Apache Ambari Apache Apex Apache Avro Apache Be ...
- hive on tez配置
1.Tez简介 Tez是Hontonworks开源的支持DAG作业的计算框架,它可以将多个有依赖的作业转换为一个作业从而大幅提升MapReduce作业的性能.Tez并不直接面向最终用户--事实上它允许 ...
随机推荐
- atitit.http get post的原理以及框架实现java php
atitit.http get post的原理以及框架实现java php 1. 相关的设置 1 1.1. urlencode 1 1.2. 输出流的编码 1 1.3. 图片,文件的post 1 2. ...
- Struts2动作
ActionContext 每一个请求的处理都在一个独立的线程中.每一个线程都有一个ActionContext对象.它包括了ValueStack和HttpServletRequest的东西. Stru ...
- JS高程3:JSON
JSON,JavaScript Object Notation,JS对象表示法,是目前最常见的传输结构化数据的数据结构. JSON并非编程语言,而是一种数据结构,像mp4.avi一样,只是一种数据格式 ...
- getCanonicalPath getAbsolutePath区别
1.在winows环境下它们的区别是 getCanonicalPath是标准路径,没有特殊字符,getAbsolutePath是有特殊字符的 2.在AIX系统中它们的区别: 首先编译:javac ...
- (译)Getting Started——1.3.3 Working with Foundation(使用Foundation框架)
在你使用Objective-C语言开发应用时,你会发现在开发中,你会用到很多框架.尤其是Foundation框架,该框架为应用提供了最基础的服务.Foundation框架包括了代表着基本数据类型的va ...
- ArcGIS教程:“流向”的工作原理
获取表面的水文特征的关键之中的一个是可以确定从栅格中的每一个像元流出的方向.这可通过流向工具来完毕. 该工具把表面作为输入,然后输出一个显示从每一个像元流出方向的栅格. 假设选择了输出下降率栅格数据选 ...
- pip 使用技巧
指定豆瓣源安装 pip install configparser -i http://pypi.douban.com/simple/ --trusted-host=pypi.douban.com/si ...
- Windows 动态库创建和使用 part 2
一.Windows动态库的创建: 1.先选择 "DLL" 和 “控项目” 2.添加一个头文件,一个源文件 CppDll.h,CppDll.cpp,一个模块定义文件 CppDll. ...
- Windows下RabbitMQ安装,部署,配置
安装部署 1.当前环境以及参考资料出处 部署环境:windows server 2008 r2 enterprise 官方安装部署文档:http://www.rabbitmq.com/install- ...
- HDU 4970(杭电多校#9 1011题)Killing Monsters(瞎搞)
题目地址:HDU 4970 先进行预处理.在每一个炮塔的火力范围边界标记一个点. 然后对每一个点的伤害值扫一遍就能算出来. 然后在算出每一个点到终点的总伤害值,并保存下来,也是扫一遍就可以. 最后在询 ...