Hadoop生态圈-Azkaban实现文件上传到hdfs并执行MR数据清洗

　　　　　　　　　　　　　　　　　Hadoop生态圈-Azkaban实现文件上传到hdfs并执行MR数据清洗

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

　　如果你没有Hadoop集群的话也没有关系，我这里给出当时我部署Hadoop集群的笔记：https://www.cnblogs.com/yinzhengjie/p/9154265.html。当然想要了解更多还是请参考官网的部署方案，我部署的环境只是测试开发环境。

一.启动Hadoop集群

1>.启动脚本信息

[yinzhengjie@s101 ~]$ more /usr/local/bin/xzk.sh

#!/bin/bash

#@author :yinzhengjie

#blog:http://www.cnblogs.com/yinzhengjie

#EMAIL:y1053419035@qq.com

#判断用户是否传参

if [ $# -ne  ];then

    echo "无效参数，用法为: $0  {start|stop|restart|status}"

    exit

fi

#获取用户输入的命令

cmd=$

#定义函数功能

function zookeeperManger(){

    case $cmd in

    start)

        echo "启动服务"

        remoteExecution start

        ;;

    stop)

        echo "停止服务"

        remoteExecution stop

        ;;

    restart)

        echo "重启服务"

        remoteExecution restart

        ;;

    status)

        echo "查看状态"

        remoteExecution status

        ;;

    *)

        echo "无效参数，用法为: $0  {start|stop|restart|status}"

        ;;

    esac

}

#定义执行的命令

function remoteExecution(){

    for (( i= ; i<= ; i++ )) ; do

            tput setaf

            echo ========== s$i zkServer.sh  $ ================

            tput setaf

            ssh s$i  "source /etc/profile ; zkServer.sh $1"

    done

}

#调用函数

zookeeperManger

[yinzhengjie@s101 ~]$

zookeeper启动脚本（[yinzhengjie@s101 ~]$ more /usr/local/bin/xzk.sh ）

[yinzhengjie@s101 ~]$ more /usr/local/bin/xcall.sh

#!/bin/bash

#@author :yinzhengjie

#blog:http://www.cnblogs.com/yinzhengjie

#EMAIL:y1053419035@qq.com

#判断用户是否传参

if [ $# -lt  ];then

        echo "请输入参数"

        exit

fi

#获取用户输入的命令

cmd=$@

for (( i=;i<=;i++ ))

do

        #使终端变绿色

        tput setaf

        echo ============= s$i $cmd ============

        #使终端变回原来的颜色，即白灰色

        tput setaf

        #远程执行命令

        ssh s$i $cmd

        #判断命令是否执行成功

        if [ $? ==  ];then

                echo "命令执行成功"

        fi

done

[yinzhengjie@s101 ~]$

批量执行命令的脚本（[yinzhengjie@s101 ~]$ more /usr/local/bin/xcall.sh）

[yinzhengjie@s101 ~]$ more /soft/hadoop/sbin/start-dfs.sh

#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Start hadoop dfs daemons.

# Optinally upgrade or rollback dfs state.

# Run this on master node.

usage="Usage: start-dfs.sh [-upgrade|-rollback] [other options such as -clusterId]"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/hdfs-config.sh

# get arguments

if [[ $# -ge  ]]; then

  startOpt="$1"

  shift

  case "$startOpt" in

    -upgrade)

      nameStartOpt="$startOpt"

    ;;

    -rollback)

      dataStartOpt="$startOpt"

    ;;

    *)

      echo $usage

      exit

    ;;

  esac

fi

#Add other possible options

nameStartOpt="$nameStartOpt $@"

#---------------------------------------------------------

# namenodes

NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -namenodes)

echo "Starting namenodes on [$NAMENODES]"

"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

  --config "$HADOOP_CONF_DIR" \

  --hostnames "$NAMENODES" \

  --script "$bin/hdfs" start namenode $nameStartOpt

#---------------------------------------------------------

# datanodes (using default slaves file)

if [ -n "$HADOOP_SECURE_DN_USER" ]; then

  echo \

    "Attempting to start secure cluster, skipping datanodes. " \

    "Run start-secure-dns.sh as root to complete startup."

else

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

    --config "$HADOOP_CONF_DIR" \

    --script "$bin/hdfs" start datanode $dataStartOpt

fi

#---------------------------------------------------------

# secondary namenodes (if any)

SECONDARY_NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -secondarynamenodes >/dev/null)

if [ -n "$SECONDARY_NAMENODES" ]; then

  echo "Starting secondary namenodes [$SECONDARY_NAMENODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

      --config "$HADOOP_CONF_DIR" \

      --hostnames "$SECONDARY_NAMENODES" \

      --script "$bin/hdfs" start secondarynamenode

fi

#---------------------------------------------------------

# quorumjournal nodes (if any)

SHARED_EDITS_DIR=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.namenode.shared.edits.dir >&-)

case "$SHARED_EDITS_DIR" in

qjournal://*)

  JOURNAL_NODES=$(echo "$SHARED_EDITS_DIR" | sed 's,qjournal://\([^/]*\)/.*,\1,g; s/;/ /g; s/:[0-9]*//g')

  echo "Starting journal nodes [$JOURNAL_NODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

      --config "$HADOOP_CONF_DIR" \

      --hostnames "$JOURNAL_NODES" \

      --script "$bin/hdfs" start journalnode ;;

esac

#---------------------------------------------------------

# ZK Failover controllers, if auto-HA is enabled

AUTOHA_ENABLED=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.ha.automatic-failover.enabled)

if [ "$(echo "$AUTOHA_ENABLED" | tr A-Z a-z)" = "true" ]; then

  echo "Starting ZK Failover Controllers on NN hosts [$NAMENODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

    --config "$HADOOP_CONF_DIR" \

    --hostnames "$NAMENODES" \

    --script "$bin/hdfs" start zkfc

fi

# eof

[yinzhengjie@s101 ~]$

HDFS启动脚本（[yinzhengjie@s101 ~]$ more /soft/hadoop/sbin/start-dfs.sh）

[yinzhengjie@s101 ~]$ more /soft/hadoop/sbin/start-yarn.sh

#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Start all yarn daemons.  Run this on master node.

echo "starting yarn daemons"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/yarn-config.sh

# start resourceManager

#"$bin"/yarn-daemon.sh --config $YARN_CONF_DIR  start resourcemanager

"$bin"/yarn-daemons.sh --config $YARN_CONF_DIR --hosts masters start resourcemanager

# start nodeManager

"$bin"/yarn-daemons.sh --config $YARN_CONF_DIR  start nodemanager

# start proxyserver

#"$bin"/yarn-daemon.sh --config $YARN_CONF_DIR  start proxyserver

[yinzhengjie@s101 ~]$

资源调度器Yarn启动脚本（[yinzhengjie@s101 ~]$ more /soft/hadoop/sbin/start-yarn.sh ）

2>.启动zookeeper集群

[yinzhengjie@s101 ~]$ xzk.sh start

启动服务

========== s102 zkServer.sh start ================

ZooKeeper JMX enabled by default

Using config: /soft/zk/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

========== s103 zkServer.sh start ================

ZooKeeper JMX enabled by default

Using config: /soft/zk/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

========== s104 zkServer.sh start ================

ZooKeeper JMX enabled by default

Using config: /soft/zk/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 AzkabanWebServer

 AzkabanExecutorServer

 Jps

命令执行成功

============= s102 jps ============

 Jps

 QuorumPeerMain

命令执行成功

============= s103 jps ============

 Jps

 QuorumPeerMain

命令执行成功

============= s104 jps ============

 Jps

 QuorumPeerMain

命令执行成功

============= s105 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

3>.启动HDFS分布式文件系统

[yinzhengjie@s101 ~]$ start-dfs.sh

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/soft/hadoop-2.7./share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.-bin/lib/log4j-slf4j-impl-2.4..jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Starting namenodes on [s101 s105]

s101: starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s101.out

s105: starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s105.out

s105: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s105.out

s103: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s103.out

s104: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s104.out

s102: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s102.out

Starting journal nodes [s102 s103 s104]

s103: starting journalnode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-journalnode-s103.out

s102: starting journalnode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-journalnode-s102.out

s104: starting journalnode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-journalnode-s104.out

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/soft/hadoop-2.7./share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.-bin/lib/log4j-slf4j-impl-2.4..jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Starting ZK Failover Controllers on NN hosts [s101 s105]

s101: starting zkfc, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-zkfc-s101.out

s105: starting zkfc, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-zkfc-s105.out

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

 DFSZKFailoverController

 AzkabanWebServer

 AzkabanExecutorServer

 NameNode

命令执行成功

============= s102 jps ============

 JournalNode

 DataNode

 Jps

 QuorumPeerMain

命令执行成功

============= s103 jps ============

 DataNode

 QuorumPeerMain

 JournalNode

 Jps

命令执行成功

============= s104 jps ============

 JournalNode

 DataNode

 Jps

 QuorumPeerMain

命令执行成功

============= s105 jps ============

 DFSZKFailoverController

 DataNode

 NameNode

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

4>.启动YARN资源调度器

[yinzhengjie@s101 ~]$ start-yarn.sh

starting yarn daemons

s101: starting resourcemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-resourcemanager-s101.out

s105: starting resourcemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-resourcemanager-s105.out

s105: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s105.out

s103: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s103.out

s104: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s104.out

s102: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s102.out

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 DFSZKFailoverController

 AzkabanWebServer

 Jps

 AzkabanExecutorServer

 ResourceManager

 NameNode

命令执行成功

============= s102 jps ============

 JournalNode

 Jps

 DataNode

 NodeManager

 QuorumPeerMain

命令执行成功

============= s103 jps ============

 NodeManager

 Jps

 DataNode

 QuorumPeerMain

 JournalNode

命令执行成功

============= s104 jps ============

 JournalNode

 Jps

 DataNode

 QuorumPeerMain

 NodeManager

命令执行成功

============= s105 jps ============

 DFSZKFailoverController

 Jps

 DataNode

 NameNode

 NodeManager

命令执行成功

[yinzhengjie@s101 ~]$

5>.检查web服务是否可用正常访问

二.文件上传到hdfs

1>.创建job文件并将其压缩

2>.将压缩后的文件上传至Azkaban的WEB服务器上

3>.执行jpb程序

4>.点击继续

5>.查看细节

6>.查看日志信息

7>.查看HDFS的webUI是否在根下成功创建azkaban目录

8>.查看执行ID

三.执行MR数据清洗

1>.编辑的配置文件内容

[yinzhengjie@s101 ~]$ more /home/yinzhengjie/yinzhengjie.txt

Security is off.

Safemode is off.

 files and directories,  blocks =  total filesystem object(s).

Heap Memory used 74.06 MB of 261.5 MB Heap Memory. Max Heap Memory is  MB.

Non Heap Memory used 56.54 MB of 57.94 MB Commited Non Heap Memory. Max Non Heap Memory is - B.

[yinzhengjie@s101 ~]$

源文件内容（[yinzhengjie@s101 ~]$ more /home/yinzhengjie/yinzhengjie.txt）

#putFileToHdfs Add by yinzhengjie

type=command

command=hdfs dfs -put /home/yinzhengjie/yinzhengjie.txt /azkaban

将需要进行单词统计的数据上传到HDFS中（putFileToHdfs.job）

#mapreduce.job ADD by yinzhengjie

type=command

command=hadoop jar /soft/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7..jar wordcount /azkaban /azkaban_out

执行MapReduce任务对上传的到hdfs的文件进行单词统计（mapreduce.job）

2>.编辑job任务并压缩

3>.将任务上传到azkaban的WEB界面中

4>.选择执行顺序

5>.查看MapReduce的运行状态

6>.查看MapReduce的运行结果

Hadoop生态圈-Azkaban实现文件上传到hdfs并执行MR数据清洗的更多相关文章

Ubuntu本地文件上传至HDFS文件系统出现的乱码问题及解决方案
1.问题来源及原因用shell命令上传到HDFS中之后出现中文乱码,在shell命令窗口查看如图: 在eclipse中的文件HDFS查看工具查看如图: 原因:上传至HDFS文件系统的文本文件(这里是 ...
FTP文件上传到HDFS上
在做测试数据时,往往会有ftp数据上传到hdfs的需求,一般需要手动操作,这样做太费事,于是有了下边代码实现的方式: ftp数据上传到hdfs函数: import java.io.InputStrea ...
HDwiki文件上传导致远程代码执行漏洞
漏洞版本: HDwiki(2011) 漏洞描述: 互动维客开源系统(HDwiki)作为中国第一家拥有自主知识产权的中文维基(Wiki)系统,由互动在线(北京)科技有限公司于2006 年11月28日正式 ...
Python Paramiko实现sftp文件上传下载以及远程执行命令
一.简介 Paramiko模块是基于Python实现的SSH远程安全连接,用于SSH远程执行命令.文件传输等功能. 安装模块默认Python没有自带,需要手动安装: pip3 install par ...
Hadoop_13_Hadoop Shell脚本采集日志上传到HDFS
案例1:开发shell采集脚本 1.点击流日志每天都10T,在业务应用服务器上,需要准实时上传至数据仓库(Hadoop HDFS)上 2.一般上传文件都是在凌晨24点操作,由于很多种类的业务数据都要在 ...
Hadoop 代码实现文件上传
本项目主要实现Windows下利用代码实现Hadoop中文件上传至HDFS 实现上传文本文件中单词个数的计数 1.项目结构 2.相关代码 CopyFromLocalFile 1 package com ...
WEB文件上传漏洞介绍解决办法
引用:http://blog.csdn.net/kobejayandy/article/details/35861499 问题: -1. 上传文件WEB脚本语言,服务器的WEB容器解释并执行了用户上传 ...
中国电信某站点JBOSS任意文件上传漏洞
1.目标站点 http://125.69.112.239/login.jsp 2.简单测试发现是jboss,HEAD请求头绕过失败,猜测弱口令失败,发现没有删除 http://125.69.112. ...
WEB安全：文件上传漏洞
文件上传漏洞过程用户上传了一个可执行的脚本文件,并通过此脚本文件获得了执行服务器端命令的能力. 一般的情况有: 上传文件WEB脚本语言,服务器的WEB容器解释并执行了用户上传的脚本,导致代码执行: ...

随机推荐

【HNOI2018】游戏
题面题解这道题目到底有没有靠谱一点的解法啊... 有很多种$\color{green}{\mathrm{AC}}$的方法,设$L[i],R[i]$表示点$i$最左边和最右边能够到达的位 ...
CS50.3
1,int()取整函数 2,RPG(role playing game )角色扮演游戏 3,代码写了,要跑,需要compiler (编译器) 4,CLI(command-line interface) ...
linux中wget的使用方法介绍
wget是在Linux下开发的开放源代码的软件,作者是Hrvoje Niksic,后来被移植到包括Windows在内的各个平台上.它有以下功能和特点:(1)支持断点下传功能:这一点,也是网络蚂蚁和Fl ...
java实现基于关键字的文件夹(文件）的搜索、文件夹（文件）的复制、删除
最近在做一个项目,需要实现这几项功能,上网查了很多资料,自己研究了好几天终于实现了,现在与大家分享一下. 一.JAVA实现文件夹的搜索在百度搜索N个技术文章,从哪些大牛们共享的资料中终于写出了我 ...
jinkens 构建java及vue 项目
能帮我们学习吉他的音乐软件——Guitar Pro
Guitar Pro是一款十分好用的吉他软件,也是目前广大音乐爱好者最喜欢的多音轨的音谱编辑软件.支持MIDI.MusicXML.PTB.GTP等多种格式文件的导入/导出. Guitar Pro 7. ...
软件测试----H模型
H模型将测试活动完全独立出来,形成一个完整的流程,同时将测试准备和测试执行清晰表现出来. 测试流程: --测试准备:所有测试活动的准备判断是否到测试就绪点. --测试就绪点:测试准入准则,即是否可以开 ...
《杜增强讲Unity之Tanks坦克大战》11-游戏流程控制
11 游戏流程控制使用协程来控制游戏流程 11.1 添加MessageText 首先添加一个Text来显示文字 image 设置GameMgr image 11.2 游戏整体流程下面Gam ...
win10引导错误的修复（内容系转载）
#!尊重原作者,再此声明此内容属于网络转载,只是为了能保留下来方便日后查阅!!! win10误删引导文件,0xc0000098的解决方案,bcd引导文件受损情况分析一.※相对简单的解决方法,对应的情 ...
OpenMPI源码剖析：网络通信原理(二) 如何选择网络协议?
因为比较常用的是 TCP 协议,所以在 opal/mca/btl/tcp/btl_tcp.h 头文件中找到对应的 struct mca_btl_tcp_component_t { mca_btl_ba ...

Hadoop生态圈-Azkaban实现文件上传到hdfs并执行MR数据清洗

Hadoop生态圈-Azkaban实现文件上传到hdfs并执行MR数据清洗的更多相关文章

随机推荐

热门专题