Azkaban 2.5.0 job type 插件安装

一、环境及软件

安装环境：

• 安装目录： /usr/local/ae/ankaban
• Hadoop 安装目录 export HADOOP_HOME=/usr/local/ae/hadoop-1.2.1
• azkaban-executor-2.5.0安装目录：/usr/local/ae/azkaban/azkaban-executor-2.5.0

安装软件：

• azkaban-jobtype-2.5.0.tar.gz

Azkaban jobtype plugin source：github.com/azkaban/azkaban-plugins/tree/master/plugins/jobtype

二、配置jobtype

1. 解压 azkaban-jobtype-2.5.0.tar.gz 到 $AZKABAN-EXECUTOR_HOME/plugins/jobtypes 目录下

   user@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5./plugins/jobtypes$ tar -zxvx azkaban--jobtype-2.5..tar.gz

2. jobtype properties 配置
   common.properties

   ## everything that the user job can know
   
   hadoop.home=/usr/local/ae/hadoop-1.2.
   #hive.home=
   #pig.home=
   
   azkaban.should.proxy=false
   jobtype.global.classpath=${hadoop.home}/hadoop-core-1.2..jar,${hadoop.home}/*,${hadoop.home}/conf/,${hadoop.home}/lib/*

   commonprivate.properties

   ## hadoop security manager setting common to all hadoop jobs
   hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_1_0
   
   ## hadoop security related settings
   
   # proxy.keytab.location=YOUR_AZKABAN_KERBEROS_PRINCIPAL
   #  proxy.user=KEYTAB_LOCATION
   
   azkaban.should.proxy=false
   obtain.binary.token=false
   # obtain.namenode.token=true
   # obtain.jobtracker.token=true
   
   # global classpath items for all jobs. e.g. hadoop-core jar, hadoop conf
   #jobtype.global.classpath=${hadoop.home}/*,${hadoop.home}/conf
   
   # global jvm args for all jobs. e.g. java.io.temp.dir, java.library.path
   #jobtype.global.jvm.args=
   
   # hadoop
   hadoop.home=/usr/local/ae/hadoop-1.2.1
   #pig.home=
   #hive.home=

三、配置 jobtype 插件安装目录到 Azkaban-executor

1. 配置 azkaban.properties
   在 azkaban.properties 文件中添加以下配置

   # Azkaban JobTypes Plugins
   azkaban.jobtype.plugin.dir=plugins/jobtypes

四、重启 Azkaban-executor

user@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.$ sh bin/azkaban-executor-shutdown.sh
user@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.$ sh bin/azkaban-executor-start.sh

五、HadoopJava jobtype profile

type=hadoopJava
#jvm.args
job.class=×××××××××××××××××××××****
main.args=×××××××××
method.run=run
#method.cancel
classpath=./lib/*,/usr/local/ae/hadoop-1.2.1/*,/usr/local/ae/hadoop-1.2.1/lib/*,/usr/local/ae/hbase-0.94.8/*,/usr/local/ae/hbase-0.94.8/lib/*
#dependencies
#getJobGeneratedProperties

Note：1. 若 ${job.class} 继承自 AbstractHadoopJob 则需要设置 FileIutPutFormat 与 FileOutPutFormat

2. Azkaban 在启动是会生成两个日志文件azkaban-access.log/azkaban-webserver.log，他们的生成位置是在你执行脚本的目录，所以建议你最好还是在AZKABAN_HOME 目录下执行启动脚本，如果你喜欢在 ./bin 目录下启动，你需要将上文第3.1步骤的红色标记处修改目录位置为 ../${dir}