azkaban的安装部署

一、所需环境

1，JDK

2，HADOOP

4，mysql

3，HIVE

二、安装azkaban

1,安装git命令: yum install git

2,下载azkaban源码：git clone https://github.com/azkaban/azkaban.git

3,安装到 azkaban-tar目录下

4,更改配置文件：

vi 安装目录/azkaban-tar/azkaban/git clone https://github.com/azkaban/azkaban.git

将final boolean isExecuteAsUser = this.sysProps.getBoolean(EXECUTE_AS_USER, true);

true修改false

5,在azkaban目录下执行：./gradlew
distTar

此步骤会生成5个压缩包，分别放在下图目录中：

azkaban-db-3.37.0-10-gdd67b02.tar.gz

azkaban-exec-server-3.37.0-10-gdd67b02.tar.gz

azkaban-hadoop-security-plugin-3.37.0-10-gdd67b02.tar.gz

azkaban-solo-server-3.37.0-10-gdd67b02.tar.gz

azkaban-web-server-3.37.0-10-gdd67b02.tar.gz

6,在当前azkaban文件夹下，将该五个压缩包copy出来，放到 安装目录/azkaban-tar文件夹中。

cp
 azkaban-*/build/distributions/*.tar.gz  安装目录/azkaban-tar

7,将这5个压缩包解压，然后重命令，将这5个压缩包放在另外的 安装目录/azkaban下

8,登录mysql mysql –u root –p

9,数据库中执行：create database azkaban;

10,数据库中执行：use azkaban;

11, 将azkaban sql表结构导入mysql，数据库中执行：source 安装目录/azkaban/azkaban-db-3.46.0-2-gbb24a72/create-all-sql-3.46.0-2-gbb24a72.sql

如果执行失败，则将sql语句复制出来执行

12，部署azkaban-web-server

将 安装目录/azkaban/azkaban-solo-server/ 下的conf目录拷贝到  安装目录/azkaban/ azkaban-web-server/conf

命令：cp -r  /安装目录/azkaban/azkaban-solo-server/conf   /安装目录/azkaban/azkaban-web-server/conf

配置：azkaban.properties

如下（标红的需要更改，其他的可以对比一下）：

# Azkaban Personalization Settings

azkaban.name=Test

azkaban.label=My Local Azkaban

azkaban.color=#FF3601

azkaban.default.servlet.path=/index

web.resource.dir=/usr/local/azkaban/azkaban-web-server/web

default.timezone.id=Asia/Shanghai

# Azkaban UserManager class

user.manager.class=azkaban.user.XmlUserManager

user.manager.xml.file=/usr/local/azkaban/azkaban-web-server/conf/azkaban-users.xml

# Loader for projects

executor.global.properties=/usr/local/azkaban/azkaban-web-server/conf/global.properties

azkaban.project.dir=projects

database.type=mysql

mysql.port=3306

mysql.host=localhost

mysql.database=azkaban

mysql.user=root

mysql.password=root

mysql.numconnections=100

 

# Velocity dev mode

velocity.dev.mode=false

# Azkaban Jetty server properties.

jetty.use.ssl=false

jetty.maxThreads=25

jetty.port=8081

# Azkaban Executor settings

executor.maxThreads=50

executor.port=12321

executor.flow.threads=30

# mail settings

mail.sender=

mail.host=

# User facing web server configurations used to
construct the user facing server URLs. They are useful when there is a reverse
proxy between Azkaban web servers and users.

# enduser -> myazkabanhost:443 -> proxy
-> localhost:8081

# when this parameters set then these parameters
are used to generate email links.

# if these parameters are not set then
jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.

#
azkaban.webserver.external_hostname=myazkabanhost.com

# azkaban.webserver.external_ssl_port=443

# azkaban.webserver.external_port=8081

job.failure.email=

job.success.email=

lockdown.create.projects=false

cache.directory=cache

# JMX stats

jetty.connector.stats=true

executor.connector.stats=true

 

azkaban.native.lib=false

# Azkaban plugin settings

azkaban.jobtype.plugin.dir=plugins/jobtypes

 

13,在conf目录下新增配置文件log4j.properties,内容如下

log4j.rootLogger=INFO,C

log4j.appender.C=org.apache.log4j.ConsoleAppender

log4j.appender.C.Target=System.err

log4j.appender.C.layout=org.apache.log4j.PatternLayout

log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd
HH:mm:ss} %-5p %c{1}:%L - %m%n

14，部署 azkaban-exec-server

将 /安装目录/azkaban/ azkaban-solo-server下的conf文件拷贝到 /安装文件/azkaban/ azkaban-exec-server

命令：cp –r /安装目录/azkaban/azkaban-solo-server/conf
 /安装目录/azkaban/azkaban-exec-server/conf

配置azkaban.properties:（标红的需要更改，其他的可以对比一下）

# Azkaban Personalization Settings

azkaban.name=Test

azkaban.label=My Local Azkaban

azkaban.color=#FF3601

azkaban.default.servlet.path=/index

web.resource.dir=web/

default.timezone.id=Asia/Shanghai

# Azkaban UserManager class

user.manager.class=azkaban.user.XmlUserManager

user.manager.xml.file=conf/azkaban-users.xml

# Loader for projects

executor.global.properties=/usr/local/azkaban/azkaban-exec-server/conf/global.properties

azkaban.project.dir=projects

database.type=mysql

mysql.port=3306

mysql.host=localhost

mysql.database=azkaban

mysql.user=root

mysql.password=root

mysql.numconnections=100

# Velocity dev mode

velocity.dev.mode=false

# Azkaban Jetty server properties.

jetty.use.ssl=false

jetty.maxThreads=25

jetty.port=8081

# Azkaban Executor settings

executor.maxThreads=50

executor.port=12321

executor.flow.threads=30

# mail settings

mail.sender=

mail.host=

# User facing web server configurations used to
construct the user facing server URLs. They are useful when there is a reverse
proxy between Azkaban web servers and users.

# enduser -> myazkabanhost:443 -> proxy
-> localhost:8081

# when this parameters set then these parameters
are used to generate email links.

# if these parameters are not set then
jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.

#
azkaban.webserver.external_hostname=myazkabanhost.com

# azkaban.webserver.external_ssl_port=443

# azkaban.webserver.external_port=8081

job.failure.email=

job.success.email=

lockdown.create.projects=false

cache.directory=cache

# JMX stats

jetty.connector.stats=true

executor.connector.stats=true

# Azkaban plugin settings

azkaban.jobtype.plugin.dir=plugins/jobtypes

 

azkaban.native.lib=false

在conf目录下，新建配置log4j.properties文件，内容如下：

log4j.rootLogger=INFO,C

log4j.appender.C=org.apache.log4j.ConsoleAppender

log4j.appender.C.Target=System.err

log4j.appender.C.layout=org.apache.log4j.PatternLayout

log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd
HH:mm:ss} %-5p %c{1}:%L - %m%n

15,在hadoop中的/share/Hadoop/common 路径下找到hadoop-common-2.6.4.jar

在jdk中的/jdk1.8.0_91/db/lib/路径下找到derby.jar

将这俩个jar包拷贝到azkaban-exec-server和azkaban-web-server

安装目录下的extlib目录，没有则创建一个，找不到就下载

16,启动

进入azkaban-exec-server目录，执行start-exec.sh

进入azkaban-web-server目录，执行start-web.sh

查看进程：jps

成功执行

通过http://localhost:8081可访问到web-ui

16,此时的web界面：

需要更改配置文件解决：

将上图中的web目录的路径加到azkaban.properties配置文件中，如下图

再进入web目录可以看到界面变得美观

17, 用户名密码在 azkaban-web-server/conf/azkaban-users.xml 中配置