Hadoop启动脚本分析

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

　　　　能看到这篇博客的你估计对Hadoop已经有一个系统的了解了，最起码各种搭建方式你应该是会的，不会也没有关系，可以参考我的笔记，里面有各种搭建方式，哈哈哈~

[yinzhengjie@s101 ~]$ cat `which xcall.sh`

#!/bin/bash

#@author :yinzhengjie

#blog:http://www.cnblogs.com/yinzhengjie

#EMAIL:y1053419035@qq.com

#判断用户是否传参

if [ $# -lt  ];then

        echo "请输入参数"

        exit

fi

#获取用户输入的命令

cmd=$@

for (( i=;i<=;i++ ))

do

        #使终端变绿色

        tput setaf

        echo ============= s$i $cmd ============

        #使终端变回原来的颜色，即白灰色

        tput setaf

        #远程执行命令

        ssh s$i $cmd

        #判断命令是否执行成功

        if [ $? ==  ];then

                echo "命令执行成功"

        fi

done

[yinzhengjie@s101 ~]$

xcall.sh 脚本内容，我在测试的时候经常使用它

一.start-all.sh脚本分析

[yinzhengjie@s101 ~]$ cat `which start-all.sh`  | grep -v ^# | grep -v ^$

echo "This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/hadoop-config.sh

if [ -f "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh ]; then

  "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh --config $HADOOP_CONF_DIR

fi

if [ -f "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh ]; then

  "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh --config $HADOOP_CONF_DIR

fi

[yinzhengjie@s101 ~]$

　　从这个脚本中的第一行我们可以看出来，这个脚本已经过时了，取而代之的是：“This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh”，也就是 “start-dfs.sh”和“start-yarn.sh”。

二.start-dfs.sh 脚本分析

[yinzhengjie@s101 ~]$ more `which start-dfs.sh` | grep -v ^# | grep -v ^$

usage="Usage: start-dfs.sh [-upgrade|-rollback] [other options such as -clusterId]"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/hdfs-config.sh

if [[ $# -ge  ]]; then

  startOpt="$1"

  shift

  case "$startOpt" in

    -upgrade)

      nameStartOpt="$startOpt"

    ;;

    -rollback)

      dataStartOpt="$startOpt"

    ;;

    *)

      echo $usage

      exit

    ;;

  esac

fi

nameStartOpt="$nameStartOpt $@"

NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -namenodes)

echo "Starting namenodes on [$NAMENODES]"

"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

  --config "$HADOOP_CONF_DIR" \

  --hostnames "$NAMENODES" \

  --script "$bin/hdfs" start namenode $nameStartOpt

if [ -n "$HADOOP_SECURE_DN_USER" ]; then

  echo \

    "Attempting to start secure cluster, skipping datanodes. " \

    "Run start-secure-dns.sh as root to complete startup."

else

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

    --config "$HADOOP_CONF_DIR" \

    --script "$bin/hdfs" start datanode $dataStartOpt

fi

SECONDARY_NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -secondarynamenodes >/dev/null)

if [ -n "$SECONDARY_NAMENODES" ]; then

  echo "Starting secondary namenodes [$SECONDARY_NAMENODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

      --config "$HADOOP_CONF_DIR" \

      --hostnames "$SECONDARY_NAMENODES" \

      --script "$bin/hdfs" start secondarynamenode

fi

SHARED_EDITS_DIR=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.namenode.shared.edits.dir >&-)

case "$SHARED_EDITS_DIR" in

qjournal://*)

  JOURNAL_NODES=$(echo "$SHARED_EDITS_DIR" | sed 's,qjournal://\([^/]*\)/.*,\1,g; s/;/ /g; s/:[0-9]*//g')

  echo "Starting journal nodes [$JOURNAL_NODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

      --config "$HADOOP_CONF_DIR" \

      --hostnames "$JOURNAL_NODES" \

      --script "$bin/hdfs" start journalnode ;;

esac

AUTOHA_ENABLED=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.ha.automatic-failover.enabled)

if [ "$(echo "$AUTOHA_ENABLED" | tr A-Z a-z)" = "true" ]; then

  echo "Starting ZK Failover Controllers on NN hosts [$NAMENODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

    --config "$HADOOP_CONF_DIR" \

    --hostnames "$NAMENODES" \

    --script "$bin/hdfs" start zkfc

fi

[yinzhengjie@s101 ~]$

　　以上的注释已经被我过滤掉了，从这个脚本中大致可以看出这个脚本是用来启动hdfs进程的，即分别是：NameNode，DataNode以及secondaryNameNode。

1>.单独启动NameNode脚本用法如下：

[yinzhengjie@s101 ~]$ hadoop-daemon.sh --hostnames s101 start namenode

starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s101.out

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

 NameNode

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

　　以上是单独启动NameNode节点的脚本用法，如果你想要批量启动的话可以使用hadoop-daemons.sh命令，只不过由于我部署的集群环境只有一个NameNode节点，因此看不出来有任何效果。

[yinzhengjie@s101 ~]$ hadoop-daemons.sh --hostnames  ` hdfs getconf -namenodes` start namenode

s101: starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

 NameNode

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ hadoop-daemons.sh --hostnames ` hdfs getconf -namenodes` start namenode

2>.单独启动DataNode脚本如下：

[yinzhengjie@s101 ~]$ hadoop-daemon.sh start datanode

starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

 DataNode

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

　　以上是单独启动DataNode的脚本用法，想要执行如果你想要批量启动的话可以使用hadoop-daemons.sh命令，由于我有三个节点，看起来效果就很明显了。

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$ hadoop-daemons.sh start datanode

s102: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s102.out

s104: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s104.out

s103: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s103.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

 DataNode

命令执行成功

============= s103 jps ============

 Jps

 DataNode

命令执行成功

============= s104 jps ============

 Jps

 DataNode

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ hadoop-daemons.sh start datanode

3>.单独启动secondaryNameNode

[yinzhengjie@s101 ~]$ hadoop-daemon.sh --hostnames s101 start secondarynamenode

starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 SecondaryNameNode

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

　　以上是单独启动secondaryNameNode的脚本用法，想要执行如果你想要批量启动的话可以使用hadoop-daemons.sh命令，由于我有三个节点，看起来效果就很明显了。

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ for i in `cat /soft/hadoop/etc/hadoop/slaves | grep -v ^#` ;do  hadoop-daemons.sh --hostnames $i start secondarynamenode ;done

s102: starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s102.out

s103: starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s103.out

s104: starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s104.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

 SecondaryNameNode

命令执行成功

============= s103 jps ============

 Jps

 SecondaryNameNode

命令执行成功

============= s104 jps ============

 Jps

 SecondaryNameNode

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ for i in `cat /soft/hadoop/etc/hadoop/slaves | grep -v ^#` ;do hadoop-daemons.sh --hostnames $i start secondarynamenode ;done

三.start-yarn.sh 脚本分析

[yinzhengjie@s101 ~]$ cat /soft/hadoop/sbin/start-yarn.sh | grep -v ^# | grep -v ^$

echo "starting yarn daemons"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/yarn-config.sh

"$bin"/yarn-daemon.sh --config $YARN_CONF_DIR  start resourcemanager

"$bin"/yarn-daemons.sh --config $YARN_CONF_DIR  start nodemanager

[yinzhengjie@s101 ~]$

　　其实用法跟上面的类似，单独启动进程如下：

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ yarn-daemon.sh start  nodemanager

starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 NodeManager

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ yarn-daemon.sh start nodemanager

　　如果想要想要批量启动的，实操如下：

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$ yarn-daemons.sh start  nodemanager

s102: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s102.out

s104: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s104.out

s103: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s103.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

 NodeManager

命令执行成功

============= s103 jps ============

 Jps

 NodeManager

命令执行成功

============= s104 jps ============

 Jps

 NodeManager

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ yarn-daemons.sh start nodemanager

二.stop-all.sh脚本分析

[yinzhengjie@s101 ~]$ cat `which stop-all.sh` | grep -v ^#  | grep -v ^$

echo "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/hadoop-config.sh

if [ -f "${HADOOP_HDFS_HOME}"/sbin/stop-dfs.sh ]; then

  "${HADOOP_HDFS_HOME}"/sbin/stop-dfs.sh --config $HADOOP_CONF_DIR

fi

if [ -f "${HADOOP_HDFS_HOME}"/sbin/stop-yarn.sh ]; then

  "${HADOOP_HDFS_HOME}"/sbin/stop-yarn.sh --config $HADOOP_CONF_DIR

fi

[yinzhengjie@s101 ~]$

　　看到第一行时：echo "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"，估计你已经明白是怎么回事了把，就是把上面的所有start参数换成了stop参数。从这个脚本中的第一行我们可以看出来，这个脚本已经过时了，取而代之的是：“This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh”，也就是 “stop-dfs.sh”和“stop-yarn.sh”。

三.小结

　　综上所述，我们可以得到以下四个等式：

1>.start-all.sh = start-dfs.sh + start-yarn.sh

2>.stop-all.sh = stop-dfs.sh + stop-yarn.sh

3>.hadoop-damons.sh = hadoop-damon.sh + slaves

4>.yarn-damons.sh = yarn-damon.sh + slaves

Hadoop启动脚本分析的更多相关文章

  1. 4. hadoop启动脚本分析

    4. hadoop启动脚本分析 1. hadoop的端口 ``` 50070 //namenode http port 50075 //datanode http port 50090 //2name ...

  2. hadoop启动脚本分析及常见命令

    进程------------------ [hdfs]start-dfs.sh NameNode NN DataNode DN SecondaryNamenode 2NN [yarn]start-ya ...

  3. Hadoop 启动脚本分析与实战经验

    start-all.sh脚本现在已经废弃,推荐使用start-dfs.sh和start-yarn.sh分别启动HDFS和YARN. 在新一代的Hadoop里面HDFS称为了统一存储的平台,而YARN成 ...

  4. HBase配置&启动脚本分析

    本文档基于hbase-0.96.1.1-cdh5.0.2,对HBase配置&启动脚本进行分析 date:2016/8/4 author:wangxl HBase配置&启动脚本分析 剔除 ...

  5. Spark配置&启动脚本分析

    本文档基于Spark2.0,对spark启动脚本进行分析. date:2016/8/3 author:wangxl Spark配置&启动脚本分析 我们主要关注3类文件,配置文件,启动脚本文件以 ...

  6. hadoop启动脚本

    记录一下一个简单的hadoop启动脚本 就是启动zookeeper集群,hadoop的HDFS和YRAN的脚本 start-cluster.sh 关于关闭的脚本,只需要顺序换一下,然后将start改为 ...

  7. busybox rootfs 启动脚本分析(二）

    上次分析了busybox的启动脚本,这次分析一下init.d中一些脚本的内容. 参考链接 http://www.cnblogs.com/helloworldtoyou/p/6169678.html h ...

  8. busybox rootfs 启动脚本分析(一）

    imx6文件系统启动脚本分析.开机运行/sbin/init,读取/etc/inittab文件,进行初始化. 参考链接 http://blog.163.com/wghbeyond@126/blog/st ...

  9. Hadoop 2.0+YARN启动脚本分析与实战经验

    start-all.sh脚本现在已经废弃,推荐使用start-dfs.sh和start-yarn.sh分别启动HDFS和YARN. 在新一代的Hadoop里面HDFS称为了统一存储的平台,而YARN成 ...

随机推荐

  1. Django 数据流程图

    根据学习Django并且通过几个作业,发现Django制作网站的数据流程有些比较难懂,所以制作一个数据流程图,帮助自己理解,也希望对正学习的人有所帮助! 别的不多说,上美图:

  2. hive笔记：复杂数据类型-map结构

    map 结构 1. 语法:map(k1,v1,k2,v2,…)   操作类型:map ,map类型的数据可以通过'列名['key']的方式访问 案例: select deductions['Feder ...

  3. swift修改UITextfield的Placeholder字体大小和颜色

    第一种方法: self.userNumber.setValue(UIColor.lightGray, forKey: "_placeholderLabel.textColor") ...

  4. LVS负载均衡基础介绍及NET、DR模式配置

    LVS:术语: CIP:Client IP:客户端IP: VIP:Virtual Server IP:虚拟主机对外IP: RIP:Real Server IP:真实主机IP: DIP:Director ...

  5. hello随笔

    初次来到博客园,都试一下 我明白了,随笔就是博客咯.日记自对自己可见.再试试分类吧

  6. django自定义分页器

    一 django 的分页器 1 批量创建数据 批量导入数据: Booklist=[] for i in range(100): Booklist.append(Book(title="boo ...

  7. Javascrip 入门第三节课

    一.location对象 location.href 获取当前网页的URLlocation.search() 获取?之后的请求信息 location.href="URL" // 跳 ...

  8. 【Python 01】Python可以做什么

    Python学习未来方向: 1.数据分析 2.自然语言处理 3.社交网络分析 4.人工智能 5.深度学习 6.计算机视觉 7.网络爬虫 8.量化交易

  9. 日志学习系列（四）——NLog实例

    具体不想介绍了,新建一个解决方案 ,直接用NuGet安装就行了 具体项目代码可以在https://github.com/qiuxianhu/SimpleNLog下载

  10. Zookeeper集群为什么要是单数

    (原) 在zookeeper集群中,会有三种角色,leader. follower. observer分别对应着总统.议员.观察者. 半数以上投票通过:可以这样理解.客户端的增删改操作无论访问到了哪台 ...