Oozie wordcount实战
一、定义
基本概念
Action: An execution/computation task (Map-Reduce job, Pig job, a shell command). It can also be referred as task or 'action node'.
》》》》Action 也叫 Action Node 用于执行或者运算的任务(如MapRudecr,shell 命令等)
Workflow: A collection of actions arranged in a control dependency DAG (Direct Acyclic Graph)有向无环图. "control dependency" from one action to another means that the second action can't run until the first action has completed.
》》》》WorkFlow 依赖有向无环图控制 actions ,在这个任务结束之前,另一个任务不能运行
Workflow Definition: A programmatic description of a workflow that can be executed.
》》》》用于定义一个Wokrflow
Workflow Definition Language: The language used to define a Workflow Definition.
Workflow Job: An executable instance of a workflow definition.
Workflow Engine: A system that executes workflows jobs. It can also be referred as a DAG engine.
Workflow Definition
A workflow definition is a DAG with control flow nodes (start, end, decision, fork, join, kill) or action nodes (map-reduce, pig, etc.), nodes are connected by transitions arrows.
》》》》一个 Workflow 包括有flow控制节点[control flow nodes (start, end, decision, fork, join, kill)] 和 action nodes (map-reduce, pig, etc.)
The workflow definition language is XML based and it is called hPDL (Hadoop Process Definition Language).
二、如何编写一个 workflow.xml 之 Map-Reduce
1.The map-reduce action starts a Hadoop map/reduce job from a workflow. Hadoop jobs can be Java Map/Reduce jobs or streaming jobs. 》》》可以是一个 JAVA 的Map-ruduce程序,也可以是一个流式计算任务。
2.A map-reduce action can be configured to perform file system cleanup and directory creation before starting the map reduce job. This capability enables Oozie to retry a Hadoop job in the situation of a transient failure (Hadoop checks the non-existence of the job output directory and then creates it when the Hadoop job is starting, thus a retry without cleanup of the job output directory would fail).》》》Mapreduce 程序需要确保输出目录不存在
3.The workflow job will wait until the Hadoop map/reduce job completes before continuing to the next action in the workflow execution path.》》》在继续下一个任务之前确保这个任务已经结束了
4.The counters of the Hadoop job and job exit status (=FAILED=, KILLED or SUCCEEDED ) must be available to the workflow job after the Hadoop jobs ends. This information can be used from within decision nodes and other actions configurations.》》》必须提供一个自己的状态给别人参考,以进行别的任务安排
5.The map-reduce action has to be configured with all the necessary Hadoop JobConf properties to run the Hadoop map/reduce job.》》》这句户的意思是说,我们在编写mapreduce程序的时候只需要 Map 和 Reduce 其他配置信息在 xml 中说明
workfolw.xml(旧版本API,且缺少很多必要的配置参数,毕竟是demo)
<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf">
<start to="mr-node"/>
<action name="mr-node">
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}"/>
</prepare>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>mapred.mapper.class</name>
<value>org.apache.oozie.example.SampleMapper</value>
</property>
<property>
<name>mapred.reducer.class</name>
<value>org.apache.oozie.example.SampleReducer</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>1</value>
</property>
<property>
<name>mapred.input.dir</name>
<value>/user/${wf:user()}/${examplesRoot}/input-data/text</value>
</property>
<property>
<name>mapred.output.dir</name>
<value>/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}</value>
</property>
</configuration>
</map-reduce>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
修改 job.properties 文件
nameNode=hdfs://cen-ubuntu.cenzhongman.com:8020
jobTracker=0.0.0.0:8032
queueName=default
oozieAppRoot=oozie-apps
oozieDataRoot=oozie/datas
oozie.wf.application.path=${nameNode}/user/${user.name}/${oozieAppRoot}/mr-wordcount-wf/workflow.xml
inputDir=mr-wordcount-wf/input
outputDir=mr-wordcount-wf/output
标准的 workflow.xml 文件
参考MapReduce 程序设计中的 driver
<workflow-app xmlns="uri:oozie:workflow:0.5" name="mr-wordcount-wf">
<start to="mr-node-wordcount"/>
<action name="mr-node-wordcount">
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/user/cen/${oozieAppsRoot}/${outputDir}"/>
</prepare>
<configuration>
<property>
<name>mapred.mapper.new-api</name>
<value>true</value>
</property>
<property>
<name>mapred.reducer.new-api</name>
<value>true</value>
</property>
<property>
<name>mapreduce.job.queuename</name>
<value>${queueName}</value>
</property>
<property>
<name>mapreduce.job.map.class</name>
<value>com.cenzhongman.hdfs.WordCount$WordcountMapper</value>
</property>
<property>
<name>mapreduce.job.reduce.class</name>
<value>com.cenzhongman.hdfs.WordCount$WordcountReducer</value>
</property>
<property>
<name>mapreduce.map.output.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapreduce.map.output.value.class</name>
<value>org.apache.hadoop.io.IntWritable</value>
</property>
<property>
<name>mapreduce.job.output.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapreduce.job.output.value.class</name>
<value>org.apache.hadoop.io.IntWritable</value>
</property>
<property>
<name>mapreduce.input.fileinputformat.inputdir</name>
<value>/user/cen/${oozieAppsRoot}/${inputDir}</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.outputdir</name>
<value>/user/cen/${oozieAppsRoot}/${outputDir}</value>
</property>
</configuration>
</map-reduce>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
注意事项:
- 修改版本为0.5
- 修改程序名
- 修改 action 名(两处)
- 修改删除路径
- 修改 map reduce 新api
- 修改Mapclass(注意内部类的写法)
- 修改reduceclass(注意内部类的写法)
- 修改 map-output-key class value
- 修改 job-output-key class value
- 修改 input dir
- 修改 output dir
其他步骤
1.拷贝jar包到lib目录下
2.上传包文件夹到指定目录
3.上传数据文件
4.执行程序
export OOZIE_URL=http://cen-ubuntu:11000/oozie/
bin/oozie job -config /opt/cdh5.3.6/oozie-4.0.0-cdh5.3.6/oozie-apps/mr-wordcount-wf/job.properties -run
Oozie wordcount实战的更多相关文章
- Hadoop生态圈-Oozie部署实战
Hadoop生态圈-Oozie部署实战 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.Oozie简介 1>.什么是Oozie Oozie英文翻译为:驯象人.一个基于工作流 ...
- Apache Beam WordCount编程实战及源码解读
概述:Apache Beam WordCount编程实战及源码解读,并通过intellij IDEA和terminal两种方式调试运行WordCount程序,Apache Beam对大数据的批处理和流 ...
- Hadoop生态圈-Oozie实战之调度shell脚本
Hadoop生态圈-Oozie实战之调度shell脚本 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客展示案例:使用Oozie调度Shell脚本. 1>.解压官方案例 ...
- Hadoop生态圈-Oozie实战之逻辑调度执行多个Job
Hadoop生态圈-Oozie实战之逻辑调度执行多个Job 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 1>.启动hadoop集群 [root@yinzhengjie ha ...
- Apache Beam WordCount编程实战及源代码解读
概述:Apache Beam WordCount编程实战及源代码解读,并通过intellij IDEA和terminal两种方式调试执行WordCount程序,Apache Beam对大数据的批处理和 ...
- 从flink-example分析flink组件(1)WordCount batch实战及源码分析
上一章<windows下flink示例程序的执行> 简单介绍了一下flink在windows下如何通过flink-webui运行已经打包完成的示例程序(jar),那么我们为什么要使用fli ...
- 从flink-example分析flink组件(3)WordCount 流式实战及源码分析
前面介绍了批量处理的WorkCount是如何执行的 <从flink-example分析flink组件(1)WordCount batch实战及源码分析> <从flink-exampl ...
- Hadoop实战5:MapReduce编程-WordCount统计单词个数-eclipse-java-windows环境
Hadoop研发在java环境的拓展 一 背景 由于一直使用hadoop streaming形式编写mapreduce程序,所以目前的hadoop程序局限于python语言.下面为了拓展java语言研 ...
- Hadoop实战3:MapReduce编程-WordCount统计单词个数-eclipse-java-ubuntu环境
之前习惯用hadoop streaming环境编写python程序,下面总结编辑java的eclipse环境配置总结,及一个WordCount例子运行. 一 下载eclipse安装包及hadoop插件 ...
随机推荐
- css3 animatehue属性
-webkit-perspective(-moz,-o,perspective下同)表示透视范围大小: -webkit-transform-style很好理解了,表示变换类型,preserve-3d看 ...
- Android @1x,@2x,@3x 资源文件自动分包工具
version 1.2 1.修改不用输入扩展名 2.输出路径可选.默认会在输入路径下建文件夹 前沿: 现在开发中ios,android会使用一套图,但是ui设计师给的图命名是以@1x,@2x,@3x这 ...
- 易客CRM-3.0.4 (OpenLogic CentOS 6.5)
平台: CentOS 类型: 虚拟机镜像 软件包: apache1.3.8 centos6.5 mysql5.1.72 php5.2.17 commercial crm linux 服务优惠价: 按服 ...
- JavaScript Date对象方法详细总结
Date 对象方法 方法 描述 Date() 返回当日的日期和时间. getDate() 从 Date 对象返回一个月中的某一天 (1 ~ 31). getDay() 从 Date 对象返回一周中的某 ...
- jrtplib源码分析 第一篇 jthread的编译与分析
第一篇 jthread的编译与分析 jrtplib代码依赖库jthread,因此先从jthread开始jrtplib的学习.首先从以下链接下载jthread的源代码http://research.ed ...
- Arduino-串口函数Serial
串口是Arduino与其它设备进行通信的接口,我们需要很好的掌握它的使用.Arduino串口使用相关的函数共有10个(随着版本的升级,新版本加入了更多,具体请参见官网:http://www.ardui ...
- Jmeter入门8 连接microsoft sql server数据库取数据
本文以Jmeter 连接microsoft sql server为例. 1 从微软官网下载Microsoft SQL Server JDBC Driver 地址:http://www.microsof ...
- 最长上升子序列&&最长不下降子序列
百练2757: 题目描述: 对于给定的序列,求出最长上升子序列的长度. 题目链接:http://bailian.openjudge.cn/practice/2757 解题思路 一.动态规划 1. 找子 ...
- 线程属性总结 线程的api属性
http://blog.csdn.net/zsf8701/article/details/7842392 //线程属性结构如下:typedef struct{ int etachstate; //线程 ...
- javaScript校验图片大小、格式
1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.or ...