Hadoop生态圈-Oozie实战之逻辑调度执行多个Job

　　　　　　　　　　Hadoop生态圈-Oozie实战之逻辑调度执行多个Job

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　 作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

1>.启动hadoop集群

[root@yinzhengjie hadoop-2.5.-cdh5.3.6]# sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [s101]
s101: starting namenode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/logs/hadoop-root-namenode-yinzhengjie.out
s102: starting datanode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/logs/hadoop-root-datanode-s102.out
s104: starting datanode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/logs/hadoop-root-datanode-s104.out
s103: starting datanode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/logs/hadoop-root-datanode-s103.out
Starting secondary namenodes [s101]
s101: starting secondarynamenode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/logs/hadoop-root-secondarynamenode-yinzhengjie.out
starting yarn daemons
starting resourcemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/logs/yarn-yinzhengjie-resourcemanager-s101.out
s104: starting nodemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/logs/yarn-root-nodemanager-s104.out
s102: starting nodemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/logs/yarn-root-nodemanager-s102.out
s103: starting nodemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/logs/yarn-root-nodemanager-s103.out
[root@yinzhengjie hadoop-2.5.-cdh5.3.6]# 

启动hadoop集群（[root@yinzhengjie hadoop-2.5.0-cdh5.3.6]# sbin/start-all.sh ）

[root@yinzhengjie hadoop-2.5.-cdh5.3.6]# sbin/mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/logs/mapred-yinzhengjie-historyserver-s101.out
[root@yinzhengjie hadoop-2.5.-cdh5.3.6]# 

启动日志服务（[root@yinzhengjie hadoop-2.5.0-cdh5.3.6]# sbin/mr-jobhistory-daemon.sh start historyserver）

[root@yinzhengjie oozie-4.0.-cdh5.3.6]# bin/oozied.sh start

Setting OOZIE_HOME:          /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6
Setting OOZIE_CONFIG:        /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/conf
Sourcing:                    /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/conf/oozie-env.sh
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Setting OOZIE_CONFIG_FILE:   oozie-site.xml
Setting OOZIE_DATA:          /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/data
Setting OOZIE_LOG:           /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/logs
Setting OOZIE_LOG4J_FILE:    oozie-log4j.properties
Setting OOZIE_LOG4J_RELOAD:
hostname: Name or service not known
Setting OOZIE_HTTP_HOSTNAME:
Setting OOZIE_HTTP_PORT:
Setting OOZIE_ADMIN_PORT:
Setting OOZIE_HTTPS_PORT:
Setting OOZIE_BASE_URL:      http://:11000/oozie
Setting CATALINA_BASE:       /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/oozie-server
Setting OOZIE_HTTPS_KEYSTORE_FILE:     /root/.keystore
Setting OOZIE_HTTPS_KEYSTORE_PASS:     password
Setting OOZIE_INSTANCE_ID:
Setting CATALINA_OUT:        /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/logs/catalina.out
Setting CATALINA_PID:        /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/oozie-server/temp/oozie.pid

Using   CATALINA_OPTS:        -Xmx1024m -Dderby.stream.error.file=/home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/logs/derby.log
Adding to CATALINA_OPTS:     -Doozie.home.dir=/home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6 -Doozie.config.dir=/home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/conf -Doozie.log.dir=/home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/logs -Doozie.data.dir=/home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/data -Doozie.instance.id= -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload= -Doozie.http.hostname= -Doozie.admin.port= -Doozie.http.port= -Doozie.https.port= -Doozie.base.url=http://:11000/oozie -Doozie.https.keystore.file=/root/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=

Using CATALINA_BASE:   /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/oozie-server
Using CATALINA_HOME:   /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/oozie-server
Using CATALINA_TMPDIR: /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/oozie-server/temp
Using JRE_HOME:        /soft/jdk
Using CLASSPATH:       /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/oozie-server/bin/bootstrap.jar
Using CATALINA_PID:    /home/yinzhengjie/download/cdh/oozie-4.0.-cdh5.3.6/oozie-server/temp/oozie.pid
Existing PID file found during start.
Removing/clearing stale PID file.
[root@yinzhengjie oozie-4.0.-cdh5.3.6]# 

启动oozie（[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# bin/oozied.sh start）

[root@yinzhengjie oozie-4.0.-cdh5.3.6]# xcall.sh jps
============= s101 jps ============
 ResourceManager
 SecondaryNameNode
 JobHistoryServer
 Jps
 Bootstrap
 NameNode
命令执行成功
============= s102 jps ============
 Jps
 NodeManager
 DataNode
命令执行成功
============= s103 jps ============
 DataNode
 Jps
 NodeManager
命令执行成功
============= s104 jps ============
 NodeManager
 DataNode
 Jps
命令执行成功
[root@yinzhengjie oozie-4.0.-cdh5.3.6]# 

查看进程是否启动成功（[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# xcall.sh jps）

　　查看oozie界面是否启动成功：

2>.解压官方案例模板

[root@yinzhengjie oozie-4.0.-cdh5.3.6]#
[root@yinzhengjie oozie-4.0.-cdh5.3.6]# tar -zxf oozie-examples.tar.gz
[root@yinzhengjie oozie-4.0.-cdh5.3.6]# 

3>.编写脚本

[root@yinzhengjie oozie-4.0.-cdh5.3.6]# cat yinzhengjie-oozie-jobs/shell/test-.sh
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:y1053419035@qq.com

/bin/date -d today +"%Y-%m-%d %T" > /home/yinzhengjie/data/access-.log
[root@yinzhengjie oozie-4.0.-cdh5.3.6]#
[root@yinzhengjie oozie-4.0.-cdh5.3.6]#
[root@yinzhengjie oozie-4.0.-cdh5.3.6]# cat yinzhengjie-oozie-jobs/shell/test-.sh
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:y1053419035@qq.com

/bin/date -d today +"%Y-%m-%d %T" > /home/yinzhengjie/data/access-.log
[root@yinzhengjie oozie-4.0.-cdh5.3.6]# 

4>.编辑job.properties配置文件

[root@yinzhengjie oozie-4.0.-cdh5.3.6]# more yinzhengjie-oozie-jobs/shell/job.properties
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#HDFS地址
nameNode=hdfs://s101:8020

#ResourceManager地址
jobTracker=s101:

#队列名称
queueName=default
examplesRoot=yinzhengjie-oozie-jobs

#指定oozie的shell脚本存放路径
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/shell

#指定执行的脚本名称
EXEC1=test-.sh
EXEC2=test-.sh
[root@yinzhengjie oozie-4.0.-cdh5.3.6]# 

5>.编辑workflow.xml 配置文件

 [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# cat yinzhengjie-oozie-jobs/shell/workflow.xml
 <workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf">
     <start to="yinzhengjie-shell-node1"/>
     <action name="yinzhengjie-shell-node1">
         <shell xmlns="uri:oozie:shell-action:0.2">
             <job-tracker>${jobTracker}</job-tracker>
             <name-node>${nameNode}</name-node>
             <configuration>
                 <property>
                     <name>mapred.job.queue.name</name>
                     <value>${queueName}</value>
                 </property>
             </configuration>
             <exec>${EXEC1}</exec>
             <file>/user/root/yinzhengjie-oozie-jobs/shell/${EXEC1}#${EXEC1}</file>
             <!-- <argument>my_output=Hello Oozie</argument>-->
             <capture-output/>
         </shell>
         <ok to="yinzhengjie-shell-node2"/>
         <error to="fail"/>
     </action>

     <action name="yinzhengjie-shell-node2">
         <shell xmlns="uri:oozie:shell-action:0.2">
             <job-tracker>${jobTracker}</job-tracker>
             <name-node>${nameNode}</name-node>
             <configuration>
                 <property>
                     <name>mapred.job.queue.name</name>
                     <value>${queueName}</value>
                 </property>
             </configuration>
             <exec>${EXEC2}</exec>
             <file>/user/root/yinzhengjie-oozie-jobs/shell/${EXEC2}#${EXEC2}</file>
             <!-- <argument>my_output=Hello Oozie</argument>-->
             <capture-output/>
         </shell>
         <ok to="end"/>
         <error to="fail"/>
     </action>
     <kill name="fail">
         <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
     </kill>
     <end name="end"/>
 </workflow-app>
 [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# 

6>.上传任务配置到hdfs

[root@yinzhengjie oozie-4.0.-cdh5.3.6]# /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/bin/hdfs dfs -put yinzhengjie-oozie-jobs/shell/ /user/root/yinzhengjie-oozie-jobs/shell
[root@yinzhengjie oozie-4.0.-cdh5.3.6]#
[root@yinzhengjie oozie-4.0.-cdh5.3.6]# /home/yinzhengjie/download/cdh/hadoop-2.5.-cdh5.3.6/bin/hdfs dfs -ls -R /user/root/yinzhengjie-oozie-jobs/shell
-rw-r--r--    root supergroup         -- : /user/root/yinzhengjie-oozie-jobs/shell/blog.sh
-rw-r--r--    root supergroup        -- : /user/root/yinzhengjie-oozie-jobs/shell/job.properties
-rw-r--r--    root supergroup         -- : /user/root/yinzhengjie-oozie-jobs/shell/test-.sh
-rw-r--r--    root supergroup         -- : /user/root/yinzhengjie-oozie-jobs/shell/test-.sh
-rw-r--r--    root supergroup        -- : /user/root/yinzhengjie-oozie-jobs/shell/workflow.xml
[root@yinzhengjie oozie-4.0.-cdh5.3.6]#

[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -put yinzhengjie-oozie-jobs/shell/ /user/root/yinzhengjie-oozie-jobs/shell

7>.执行任务

[root@yinzhengjie oozie-4.0.-cdh5.3.6]# bin/oozie job -oozie http://s101:11000/oozie -config yinzhengjie-oozie-jobs/shell/job.properties -run
job: --oozie-root-W
[root@yinzhengjie oozie-4.0.-cdh5.3.6]# 

8>.

9>.