Hadoop生态圈-Azkaban部署实战

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。　　　　　　

 一.Azkaban部署流程

1>.上传azkaban程序并创建解压目录

[yinzhengjie@s101 data]$ pwd

/home/yinzhengjie/data

[yinzhengjie@s101 data]$ ll

total

-rw-r--r--  yinzhengjie yinzhengjie   Apr  : azkaban-exec-server-3.46..tar.gz

-rw-r--r--  yinzhengjie yinzhengjie      Apr  : azkaban-hadoop-security-plugin-3.46..tar.gz

-rw-r--r--  yinzhengjie yinzhengjie  Apr  : azkaban-plugins-master.zip

-rw-r--r--  yinzhengjie yinzhengjie   Apr  : azkaban-solo-server-3.46..tar.gz

-rw-r--r--  yinzhengjie yinzhengjie   Apr  : azkaban-web-server-3.46..tar.gz

-rw-r--r--  yinzhengjie yinzhengjie      Nov    create-all-sql-3.46..sql

[yinzhengjie@s101 data]$

[yinzhengjie@s101 data]$ mkdir /soft/azkaban

[yinzhengjie@s101 data]$

2>.分别解压文件到/soft/azkaban中

[yinzhengjie@s101 data]$ tar -xzf azkaban-exec-server-3.46..tar.gz -C /soft/azkaban/

[yinzhengjie@s101 data]$ tar -xzf azkaban-web-server-3.46..tar.gz -C /soft/azkaban/

[yinzhengjie@s101 data]$

3>.进入到mysql命令行并创建数据库

[yinzhengjie@s101 data]$ mysql -uroot -pyinzhengjie

Warning: Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is

Server version: 5.6. MySQL Community Server (GPL)

Copyright (c) , , Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database yinzhengjie_azkaban;

Query OK,  row affected (0.00 sec)

mysql> use yinzhengjie_azkaban;

Database changed

mysql> source /home/yinzhengjie/data/create-all-sql-3.46..sql

Query OK,  rows affected (0.03 sec)

Query OK,  rows affected (0.00 sec)

Query OK,  rows affected (0.01 sec)

Query OK,  rows affected (0.00 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.01 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.00 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.01 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.06 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.13 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.22 sec)

Query OK,  rows affected (0.15 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.18 sec)

Query OK,  rows affected (0.13 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.17 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.03 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.06 sec)

Query OK,  rows affected (0.33 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.01 sec)

Query OK,  rows affected (0.01 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.22 sec)

Query OK,  rows affected (0.12 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.09 sec)

Query OK,  rows affected (0.10 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.17 sec)

Query OK,  rows affected (0.18 sec)

Query OK,  rows affected (0.14 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.12 sec)

Query OK,  rows affected (0.04 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.08 sec)

Query OK,  rows affected (0.17 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.16 sec)

Query OK,  rows affected (0.13 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.13 sec)

Query OK,  rows affected (0.10 sec)

Records:   Duplicates:   Warnings: 

Query OK,  rows affected (0.04 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected,  warning (0.00 sec)

Query OK,  rows affected (0.19 sec)

Query OK,  rows affected (0.23 sec)

Query OK,  rows affected (0.08 sec)

Query OK,  rows affected (0.13 sec)

Query OK,  rows affected (0.01 sec)

Query OK,  rows affected (0.06 sec)

Query OK,  rows affected (0.19 sec)

Query OK,  rows affected (0.27 sec)

Query OK,  rows affected (0.18 sec)

Query OK,  rows affected (0.16 sec)

Query OK,  rows affected (0.17 sec)

Query OK,  rows affected (0.00 sec)

Query OK,  rows affected (0.19 sec)

mysql> exit;

Bye

[yinzhengjie@s101 data]$

4>.创建SSL配置（完成下述工作后,将在当前目录生成 keystore 证书文件，需要输出初始密码，选择国家，在输入确认字母y其他按回车即可）

[yinzhengjie@s101 data]$ keytool -keystore keystore -alias jetty -genkey -keyalg RSA

Enter keystore password:

Re-enter new password:

What is your first and last name?

  [Unknown]:

What is the name of your organizational unit?

  [Unknown]:

What is the name of your organization?

  [Unknown]:

What is the name of your City or Locality?

  [Unknown]:

What is the name of your State or Province?

  [Unknown]:

What is the two-letter country code for this unit?

  [Unknown]:  CN

Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=CN correct?

  [no]:  y

Enter key password for <jetty>

    (RETURN if same as keystore password):

[yinzhengjie@s101 data]$

[yinzhengjie@s101 data]$ ll | grep keystore

-rw-rw-r--  yinzhengjie yinzhengjie       Jun  : keystore

[yinzhengjie@s101 data]$

5>.将keystore文件拷贝到web文件夹下

[yinzhengjie@s101 data]$ ll | grep keystore

-rw-rw-r--  yinzhengjie yinzhengjie       Jun  : keystore

[yinzhengjie@s101 data]$ cp keystore /soft/azkaban/azkaban-web-server-3.46./

[yinzhengjie@s101 data]$

6>.编辑配置文件【azkaban.properties】

[yinzhengjie@s101 data]$ more /soft/azkaban/azkaban-web-server-3.46./conf/azkaban.properties

#Azkaban Personalization Settings

#服务器UI名称,用于服务器上方显示的名字

azkaban.name=yinzhengjie

#描述

azkaban.label=My Local Azkaban

#UI颜色

azkaban.color=#FF3601

#指定首页路径

azkaban.default.servlet.path=/index

#默认根web目录

web.resource.dir=/soft/azkaban/azkaban-web-server-3.46./web/

#默认时区,已改为亚洲/上海 默认为美国

default.timezone.id=Asia/Shanghai

#Azkaban UserManager class

#用户权限管理默认类

user.manager.class=azkaban.user.XmlUserManager

#用户配置,具体配置参加下文

user.manager.xml.file=/soft/azkaban/azkaban-web-server-3.46./conf/azkaban-users.xml

#Loader for projects

# global配置文件所在位置

executor.global.properties=/soft/azkaban/azkaban-web-server-3.46./conf/global.properties

azkaban.project.dir=projects

#数据库类型

database.type=mysql

#端口号

mysql.port=

#数据库连接IP

mysql.host=s101

#数据库实例名

mysql.database=azkaban

#数据库用户名

mysql.user=root

#数据库密码

mysql.password=yinzhengjie

#最大连接数

mysql.numconnections=

# Velocity dev mode

#关闭快速开发模式

velocity.dev.mode=false

# Jetty服务器属性.

#最大线程数

jetty.maxThreads=

#Jetty SSL端口

jetty.ssl.port=

#Jetty端口

jetty.port=

#SSL文件名

jetty.keystore=keystore

#SSL文件密码

jetty.password=yinzhengjie

#Jetty主密码 与 keystore文件相同

jetty.keypassword=yinzhengjie

#SSL文件名

jetty.truststore=keystore

# SSL文件密码

jetty.trustpassword=yinzhengjie

# 执行服务器属性

#执行服务器端口

executor.port=

# 邮件设置

#发送邮箱

mail.sender=xxxxxxxx@.com

#发送邮箱smtp地址

mail.host=smtp..com

#发送邮件时显示的名称

mail.user=xxxxxxxx

#邮箱密码

mail.password=**********

#任务失败时发送邮件的地址

job.failure.email=xxxxxxxx@.com

#任务成功时发送邮件的地址

job.success.email=xxxxxxxx@.com

lockdown.create.projects=false

#缓存目录

cache.directory=cache

# JMX stats

jetty.connector.stats=true

executor.connector.stats=true

# Azkaban JobTypes 插件配置

azkaban.jobtype.plugin.dir=plugins/jobtypes

[yinzhengjie@s101 data]$

　　以上代码只是对该配置文件的一个详细说明，不建议在这个配置文件中写中文！在实际配置中，需要将这个文件拷贝两份，分别放在web和exec目录中，实际使用配置如下：

[yinzhengjie@s101 ~]$ more /soft/azkaban/azkaban-web-server-3.46./conf/azkaban.properties

# Azkaban Personalization Settings

azkaban.name=yinzhengjie

azkaban.label=My Local Azkaban

azkaban.color=#FF3601

azkaban.default.servlet.path=/index

web.resource.dir=/soft/azkaban/azkaban-web-server-3.46./web/

default.timezone.id=Asia/Shanghai

# Azkaban UserManager class

user.manager.class=azkaban.user.XmlUserManager

user.manager.xml.file=/soft/azkaban/azkaban-web-server-3.46./conf/azkaban-users.xml

# Loader for projects

executor.global.properties=/soft/azkaban/azkaban-web-server-3.46./conf/global.properties

azkaban.project.dir=projects

database.type=mysql

mysql.port=

mysql.host=s101

mysql.database=yinzhengjie_azkaban

mysql.user=root

mysql.password=yinzhengjie

mysql.numconnections=

# Velocity dev mode

velocity.dev.mode=false

# Azkaban Jetty server properties.

jetty.use.ssl=false

jetty.maxThreads=

jetty.port=

# Azkaban Executor settings

executor.port=

# mail settings

mail.sender=

mail.host=

# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.

# enduser -> myazkabanhost: -> proxy -> localhost:

# when this parameters set then these parameters are used to generate email links.

# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.

# azkaban.webserver.external_hostname=myazkabanhost.com

# azkaban.webserver.external_ssl_port=

# azkaban.webserver.external_port=

job.failure.email=

job.success.email=

lockdown.create.projects=false

cache.directory=cache

# JMX stats

jetty.connector.stats=true

executor.connector.stats=true

# Azkaban plugin settings

azkaban.jobtype.plugin.dir=plugins/jobtypes

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ more /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban.properties

[yinzhengjie@s101 ~]$ more /soft/azkaban/azkaban-exec-server-3.46./conf/azkaban.properties

# Azkaban Personalization Settings

azkaban.name=yinzhengjie

azkaban.label=My Local Azkaban

azkaban.color=#FF3601

azkaban.default.servlet.path=/index

web.resource.dir=/soft/azkaban/azkaban-web-server-3.46./web/

default.timezone.id=Asia/Shanghai

# Azkaban UserManager class

user.manager.class=azkaban.user.XmlUserManager

user.manager.xml.file=/soft/azkaban/azkaban-web-server-3.46./conf/azkaban-users.xml

# Loader for projects

executor.global.properties=/soft/azkaban/azkaban-web-server-3.46./conf/global.properties

azkaban.project.dir=projects

database.type=mysql

mysql.port=

mysql.host=s101

mysql.database=yinzhengjie_azkaban

mysql.user=root

mysql.password=yinzhengjie

mysql.numconnections=

# Velocity dev mode

velocity.dev.mode=false

# Azkaban Jetty server properties.

jetty.use.ssl=false

jetty.maxThreads=

jetty.port=

# Azkaban Executor settings

executor.port=

# mail settings

mail.sender=

mail.host=

# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.

# enduser -> myazkabanhost: -> proxy -> localhost:

# when this parameters set then these parameters are used to generate email links.

# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.

# azkaban.webserver.external_hostname=myazkabanhost.com

# azkaban.webserver.external_ssl_port=

# azkaban.webserver.external_port=

job.failure.email=

job.success.email=

lockdown.create.projects=false

cache.directory=cache

# JMX stats

jetty.connector.stats=true

executor.connector.stats=true

# Azkaban plugin settings

azkaban.jobtype.plugin.dir=plugins/jobtypes

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ more /soft/azkaban/azkaban-exec-server-3.46.0/conf/azkaban.properties

7>.修改用户配置文件【azkaban-users.xml】

[yinzhengjie@s101 data]$ more /soft/azkaban/azkaban-web-server-3.46./conf/azkaban-users.xml

<azkaban-users>

    <user username="azkaban" password="yinzhengjie" roles="admin" groups="azkaban" />

    <user username="metrics" password="yinzhengjie" roles="metrics"/>

    <user username="admin" password="yinzhengjie" roles="admin,metrics" />

    <role name="admin" permissions="ADMIN" />

    <role name="metrics" permissions="METRICS"/>

</azkaban-users>

[yinzhengjie@s101 data]$

8>.创建global.properties

[yinzhengjie@s101 data]$ touch /soft/azkaban/azkaban-web-server-3.46./conf/global.properties

[yinzhengjie@s101 data]$

9>.编写启动脚本

[yinzhengjie@s101 ~]$ more /usr/local/bin/azweb.sh

#!/bin/bash

#@author :yinzhengjie

#blog:http://www.cnblogs.com/yinzhengjie

#EMAIL:y1053419035@qq.com

#判断用户是否传参

if [ $# -ne  ];then

    echo "无效参数，用法为: $0  {start|stop}"

    exit

fi

#获取用户输入的命令

cmd=$

case $cmd in

    start)

        cd /soft/azkaban

        /soft/azkaban/azkaban-web-server-3.46./bin/start-web.sh

        ;;

    stop)

        cd /soft/azkaban

        /soft/azkaban/azkaban-web-server-3.46./bin/shutdown-web.sh

        ;;

    *)

        echo "无效参数，用法为: $0  {start|stop}"

        exit

        ;;

esac

echo ========= azweb $cmd =============

[yinzhengjie@s101 ~]$

web端启动脚本（/usr/local/bin/azweb.sh）

[yinzhengjie@s101 ~]$ more /usr/local/bin/azexec.sh

#!/bin/bash

#@author :yinzhengjie

#blog:http://www.cnblogs.com/yinzhengjie

#EMAIL:y1053419035@qq.com

#判断用户是否传参

if [ $# -ne  ];then

    echo "无效参数，用法为: $0  {start|stop}"

    exit

fi

#获取用户输入的命令

cmd=$

case $cmd in

    start)

        cd /soft/azkaban

        /soft/azkaban/azkaban-exec-server-3.46./bin/start-exec.sh

        ;;

    stop)

        cd /soft/azkaban

        /soft/azkaban/azkaban-exec-server-3.46./bin/shutdown-exec.sh

        ;;

    *)

        echo "无效参数，用法为: $0  {start|stop}"

        exit

        ;;

esac

echo ========= azexec $cmd =============

[yinzhengjie@s101 ~]$

exec端启动脚本（/usr/local/bin/azexec.sh）

10>.启动服务（启动web和executor）

[yinzhengjie@s101 ~]$ jps

 Jps

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ azweb.sh start

========= azweb start =============

[yinzhengjie@s101 ~]$ azexec.sh start

========= azexec start =============

[yinzhengjie@s101 ~]$ jps

 AzkabanExecutorServer

 AzkabanWebServer

 Jps

[yinzhengjie@s101 ~]$ netstat -untalp | grep

(Not all processes could be identified, non-owned process info

 will not be shown, you would have to be root to see it all.)

tcp6              :::                 :::*                    LISTEN      /java

[yinzhengjie@s101 ~]$

11>.测试azkaban连接

　　输入用户名和密码信息：

　　登录成功会有以下页面：

 二. Azkaban防坑小技巧

1>.job文件有空格

　　解决方案：

　　　　　　修改command.job配置文件，将里面多余的空格删除掉！然后重新上传job任务。

2>.Missing required property 'azkaban.native.lib'

　　解决方案：

[yinzhengjie@s101 ~]$ grep azkaban.jobtype.plugin.dir /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban.properties

azkaban.jobtype.plugin.dir=/soft/azkaban/azkaban-exec-server-3.46.0/plugins/jobtypes/

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ grep azkaban.jobtype.plugin.dir  /soft/azkaban/azkaban-exec-server-3.46.0/conf/azkaban.properties

azkaban.jobtype.plugin.dir=/soft/azkaban/azkaban-exec-server-3.46.0/plugins/jobtypes/

[yinzhengjie@s101 ~]$

编辑“azkaban.properties” 配置文件，然后需要重启Azkaban服务！

3>.

4>.

Hadoop生态圈-Azkaban部署实战的更多相关文章

  1. Hadoop生态圈-Oozie部署实战

    Hadoop生态圈-Oozie部署实战 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.Oozie简介 1>.什么是Oozie Oozie英文翻译为:驯象人.一个基于工作流 ...

  2. Hadoop生态圈-Azkaban实战之Command类型执行指定脚本

    Hadoop生态圈-Azkaban实战之Command类型执行指定脚本 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 1>.服务端测试代码(别忘记添加权限哟!) [yinzh ...

  3. Hadoop生态圈-Azkaban实战之Command类型多job工作流flow

    Hadoop生态圈-Azkaban实战之Command类型多job工作流flow 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. Azkaban内置的任务类型支持command.ja ...

  4. Hadoop生态圈-Azkaban实现文件上传到hdfs并执行MR数据清洗

    Hadoop生态圈-Azkaban实现文件上传到hdfs并执行MR数据清洗 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 如果你没有Hadoop集群的话也没有关系,我这里给出当时我 ...

  5. Hadoop生态圈-Azkaban实现hive脚本执行

    Hadoop生态圈-Azkaban实现hive脚本执行 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客中在HDFS分布式系统取的数据,而这个数据的是有之前我通过MapRed ...

  6. Hadoop生态圈-Sqoop部署以及基本使用方法

    Hadoop生态圈-Sqoop部署以及基本使用方法 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. Sqoop(发音:skup)是一款开源的工具,主要用于在Hadoop(Hive)与 ...

  7. Hadoop生态圈-zookeeper完全分布式部署

    Hadoop生态圈-zookeeper完全分布式部署 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客部署是建立在Hadoop高可用基础之上的,关于Hadoop高可用部署请参 ...

  8. Hadoop生态圈-单点登录框架之CAS（Central Authentication Service）部署

    Hadoop生态圈-单点登录框架之CAS(Central Authentication Service)部署 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.CAS简介 CAS( ...

  9. Hadoop生态圈-通过CDH5.15.1部署spark1.6与spark2.3.0的版本兼容运行

    Hadoop生态圈-通过CDH5.15.1部署spark1.6与spark2.3.0的版本兼容运行 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 在我的CDH5.15.1集群中,默 ...

随机推荐

  1. 模拟赛 sutoringu

    sutoringu 题意: 询问有多少一个字符串内有多少个个子区间,满足可以分成k个相同的串. 分析: 首先可以枚举一个长度len,表示分成的k个长为len的串.然后从1开始,每len的长度分成一块, ...

  2. Unity XLua 官方教程学习

    一.Lua 文件加载 1. 执行字符串 using UnityEngine; using XLua; public class ByString : MonoBehaviour { LuaEnv lu ...

  3. "Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network" 解读

    简介:这是一篇17年的CVPR,作者提出使用现有的人脸识别深度神经网络Resnet101来得到一个具有鲁棒性的人脸模型. 原文链接:https://www.researchgate.net/publi ...

  4. Mac下搭建lamp

    Mac下搭建lamp Mac 自带了Apache,并默认支持PHP环境,只需要配置Apache和PHP即可使用.需要单独安装mysql服务端. Apache 基础配置 Apache支持PHP配置 Ap ...

  5. c# WndProc事件 消息类型

    转载:https://www.cnblogs.com/idben/p/3783997.html WM_NULL = 0x0000; WM_CREATE = 0x0001;应用程序创建一个窗口 WM_D ...

  6. PHP 爬虫——QueryList

    前言: 来了个任务说要做个电影网站,要写个壳,数据直接从别人那扒.行吧!那就要学习下PHP爬虫了.占个博客,以后补充.http://study.querylist.cc/archives/6/ 之前开 ...

  7. Alpha冲刺——day7

    Alpha冲刺--day7 作业链接 Alpha冲刺随笔集 github地址 团队成员 031602636 许舒玲(队长) 031602237 吴杰婷 031602220 雷博浩 031602634 ...

  8. PAT 甲级 1086 Tree Traversals Again

    https://pintia.cn/problem-sets/994805342720868352/problems/994805380754817024 An inorder binary tree ...

  9. Django时间时区问题（received a naive datetime while time zone support is active）

    在django1.4以后,存在两个概念 naive time 与 active time. 简单点讲,naive time就是不带时区的时间,Active time就是带时区的时间. 举例来说,使用d ...

  10. 洛谷P3703 [SDOI2017]树点涂色（LCT，dfn序，线段树，倍增LCA）

    洛谷题目传送门 闲话 这是所有LCT题目中的一个异类. 之所以认为是LCT题目,是因为本题思路的瓶颈就在于如何去维护同颜色的点的集合. 只不过做着做着,感觉后来的思路(dfn序,线段树,LCA)似乎要 ...