Hadoop启动脚本分析

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

　　　　能看到这篇博客的你估计对Hadoop已经有一个系统的了解了，最起码各种搭建方式你应该是会的，不会也没有关系，可以参考我的笔记，里面有各种搭建方式，哈哈哈~

[yinzhengjie@s101 ~]$ cat `which xcall.sh`

#!/bin/bash

#@author :yinzhengjie

#blog:http://www.cnblogs.com/yinzhengjie

#EMAIL:y1053419035@qq.com

#判断用户是否传参

if [ $# -lt  ];then

        echo "请输入参数"

        exit

fi

#获取用户输入的命令

cmd=$@

for (( i=;i<=;i++ ))

do

        #使终端变绿色

        tput setaf

        echo ============= s$i $cmd ============

        #使终端变回原来的颜色，即白灰色

        tput setaf

        #远程执行命令

        ssh s$i $cmd

        #判断命令是否执行成功

        if [ $? ==  ];then

                echo "命令执行成功"

        fi

done

[yinzhengjie@s101 ~]$

xcall.sh 脚本内容，我在测试的时候经常使用它

一.start-all.sh脚本分析

[yinzhengjie@s101 ~]$ cat `which start-all.sh`  | grep -v ^# | grep -v ^$

echo "This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/hadoop-config.sh

if [ -f "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh ]; then

  "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh --config $HADOOP_CONF_DIR

fi

if [ -f "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh ]; then

  "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh --config $HADOOP_CONF_DIR

fi

[yinzhengjie@s101 ~]$

　　从这个脚本中的第一行我们可以看出来，这个脚本已经过时了，取而代之的是：“This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh”，也就是 “start-dfs.sh”和“start-yarn.sh”。

二.start-dfs.sh 脚本分析

[yinzhengjie@s101 ~]$ more `which start-dfs.sh` | grep -v ^# | grep -v ^$

usage="Usage: start-dfs.sh [-upgrade|-rollback] [other options such as -clusterId]"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/hdfs-config.sh

if [[ $# -ge  ]]; then

  startOpt="$1"

  shift

  case "$startOpt" in

    -upgrade)

      nameStartOpt="$startOpt"

    ;;

    -rollback)

      dataStartOpt="$startOpt"

    ;;

    *)

      echo $usage

      exit

    ;;

  esac

fi

nameStartOpt="$nameStartOpt $@"

NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -namenodes)

echo "Starting namenodes on [$NAMENODES]"

"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

  --config "$HADOOP_CONF_DIR" \

  --hostnames "$NAMENODES" \

  --script "$bin/hdfs" start namenode $nameStartOpt

if [ -n "$HADOOP_SECURE_DN_USER" ]; then

  echo \

    "Attempting to start secure cluster, skipping datanodes. " \

    "Run start-secure-dns.sh as root to complete startup."

else

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

    --config "$HADOOP_CONF_DIR" \

    --script "$bin/hdfs" start datanode $dataStartOpt

fi

SECONDARY_NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -secondarynamenodes >/dev/null)

if [ -n "$SECONDARY_NAMENODES" ]; then

  echo "Starting secondary namenodes [$SECONDARY_NAMENODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

      --config "$HADOOP_CONF_DIR" \

      --hostnames "$SECONDARY_NAMENODES" \

      --script "$bin/hdfs" start secondarynamenode

fi

SHARED_EDITS_DIR=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.namenode.shared.edits.dir >&-)

case "$SHARED_EDITS_DIR" in

qjournal://*)

  JOURNAL_NODES=$(echo "$SHARED_EDITS_DIR" | sed 's,qjournal://\([^/]*\)/.*,\1,g; s/;/ /g; s/:[0-9]*//g')

  echo "Starting journal nodes [$JOURNAL_NODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

      --config "$HADOOP_CONF_DIR" \

      --hostnames "$JOURNAL_NODES" \

      --script "$bin/hdfs" start journalnode ;;

esac

AUTOHA_ENABLED=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.ha.automatic-failover.enabled)

if [ "$(echo "$AUTOHA_ENABLED" | tr A-Z a-z)" = "true" ]; then

  echo "Starting ZK Failover Controllers on NN hosts [$NAMENODES]"

  "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \

    --config "$HADOOP_CONF_DIR" \

    --hostnames "$NAMENODES" \

    --script "$bin/hdfs" start zkfc

fi

[yinzhengjie@s101 ~]$

　　以上的注释已经被我过滤掉了，从这个脚本中大致可以看出这个脚本是用来启动hdfs进程的，即分别是：NameNode，DataNode以及secondaryNameNode。

1>.单独启动NameNode脚本用法如下：

[yinzhengjie@s101 ~]$ hadoop-daemon.sh --hostnames s101 start namenode

starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s101.out

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

 NameNode

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

　　以上是单独启动NameNode节点的脚本用法，如果你想要批量启动的话可以使用hadoop-daemons.sh命令，只不过由于我部署的集群环境只有一个NameNode节点，因此看不出来有任何效果。

[yinzhengjie@s101 ~]$ hadoop-daemons.sh --hostnames  ` hdfs getconf -namenodes` start namenode

s101: starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

 NameNode

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ hadoop-daemons.sh --hostnames ` hdfs getconf -namenodes` start namenode

2>.单独启动DataNode脚本如下：

[yinzhengjie@s101 ~]$ hadoop-daemon.sh start datanode

starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

 DataNode

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

　　以上是单独启动DataNode的脚本用法，想要执行如果你想要批量启动的话可以使用hadoop-daemons.sh命令，由于我有三个节点，看起来效果就很明显了。

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$ hadoop-daemons.sh start datanode

s102: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s102.out

s104: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s104.out

s103: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s103.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

 DataNode

命令执行成功

============= s103 jps ============

 Jps

 DataNode

命令执行成功

============= s104 jps ============

 Jps

 DataNode

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ hadoop-daemons.sh start datanode

3>.单独启动secondaryNameNode

[yinzhengjie@s101 ~]$ hadoop-daemon.sh --hostnames s101 start secondarynamenode

starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 SecondaryNameNode

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

　　以上是单独启动secondaryNameNode的脚本用法，想要执行如果你想要批量启动的话可以使用hadoop-daemons.sh命令，由于我有三个节点，看起来效果就很明显了。

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ for i in `cat /soft/hadoop/etc/hadoop/slaves | grep -v ^#` ;do  hadoop-daemons.sh --hostnames $i start secondarynamenode ;done

s102: starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s102.out

s103: starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s103.out

s104: starting secondarynamenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-secondarynamenode-s104.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

 SecondaryNameNode

命令执行成功

============= s103 jps ============

 Jps

 SecondaryNameNode

命令执行成功

============= s104 jps ============

 Jps

 SecondaryNameNode

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ for i in `cat /soft/hadoop/etc/hadoop/slaves | grep -v ^#` ;do hadoop-daemons.sh --hostnames $i start secondarynamenode ;done

三.start-yarn.sh 脚本分析

[yinzhengjie@s101 ~]$ cat /soft/hadoop/sbin/start-yarn.sh | grep -v ^# | grep -v ^$

echo "starting yarn daemons"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/yarn-config.sh

"$bin"/yarn-daemon.sh --config $YARN_CONF_DIR  start resourcemanager

"$bin"/yarn-daemons.sh --config $YARN_CONF_DIR  start nodemanager

[yinzhengjie@s101 ~]$

　　其实用法跟上面的类似，单独启动进程如下：

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ yarn-daemon.sh start  nodemanager

starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s101.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 NodeManager

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ yarn-daemon.sh start nodemanager

　　如果想要想要批量启动的，实操如下：

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

命令执行成功

============= s103 jps ============

 Jps

命令执行成功

============= s104 jps ============

 Jps

命令执行成功

[yinzhengjie@s101 ~]$ yarn-daemons.sh start  nodemanager

s102: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s102.out

s104: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s104.out

s103: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s103.out

[yinzhengjie@s101 ~]$ xcall.sh jps

============= s101 jps ============

 Jps

命令执行成功

============= s102 jps ============

 Jps

 NodeManager

命令执行成功

============= s103 jps ============

 Jps

 NodeManager

命令执行成功

============= s104 jps ============

 Jps

 NodeManager

命令执行成功

[yinzhengjie@s101 ~]$

[yinzhengjie@s101 ~]$ yarn-daemons.sh start nodemanager

二.stop-all.sh脚本分析

[yinzhengjie@s101 ~]$ cat `which stop-all.sh` | grep -v ^#  | grep -v ^$

echo "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"

bin=`dirname "${BASH_SOURCE-$0}"`

bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec

HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}

. $HADOOP_LIBEXEC_DIR/hadoop-config.sh

if [ -f "${HADOOP_HDFS_HOME}"/sbin/stop-dfs.sh ]; then

  "${HADOOP_HDFS_HOME}"/sbin/stop-dfs.sh --config $HADOOP_CONF_DIR

fi

if [ -f "${HADOOP_HDFS_HOME}"/sbin/stop-yarn.sh ]; then

  "${HADOOP_HDFS_HOME}"/sbin/stop-yarn.sh --config $HADOOP_CONF_DIR

fi

[yinzhengjie@s101 ~]$

　　看到第一行时：echo "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"，估计你已经明白是怎么回事了把，就是把上面的所有start参数换成了stop参数。从这个脚本中的第一行我们可以看出来，这个脚本已经过时了，取而代之的是：“This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh”，也就是 “stop-dfs.sh”和“stop-yarn.sh”。

三.小结

　　综上所述，我们可以得到以下四个等式：

1>.start-all.sh = start-dfs.sh + start-yarn.sh

2>.stop-all.sh = stop-dfs.sh + stop-yarn.sh

3>.hadoop-damons.sh = hadoop-damon.sh + slaves

4>.yarn-damons.sh = yarn-damon.sh + slaves

Hadoop启动脚本分析的更多相关文章

  1. 4. hadoop启动脚本分析

    4. hadoop启动脚本分析 1. hadoop的端口 ``` 50070 //namenode http port 50075 //datanode http port 50090 //2name ...

  2. hadoop启动脚本分析及常见命令

    进程------------------ [hdfs]start-dfs.sh NameNode NN DataNode DN SecondaryNamenode 2NN [yarn]start-ya ...

  3. Hadoop 启动脚本分析与实战经验

    start-all.sh脚本现在已经废弃,推荐使用start-dfs.sh和start-yarn.sh分别启动HDFS和YARN. 在新一代的Hadoop里面HDFS称为了统一存储的平台,而YARN成 ...

  4. HBase配置&启动脚本分析

    本文档基于hbase-0.96.1.1-cdh5.0.2,对HBase配置&启动脚本进行分析 date:2016/8/4 author:wangxl HBase配置&启动脚本分析 剔除 ...

  5. Spark配置&启动脚本分析

    本文档基于Spark2.0,对spark启动脚本进行分析. date:2016/8/3 author:wangxl Spark配置&启动脚本分析 我们主要关注3类文件,配置文件,启动脚本文件以 ...

  6. hadoop启动脚本

    记录一下一个简单的hadoop启动脚本 就是启动zookeeper集群,hadoop的HDFS和YRAN的脚本 start-cluster.sh 关于关闭的脚本,只需要顺序换一下,然后将start改为 ...

  7. busybox rootfs 启动脚本分析(二）

    上次分析了busybox的启动脚本,这次分析一下init.d中一些脚本的内容. 参考链接 http://www.cnblogs.com/helloworldtoyou/p/6169678.html h ...

  8. busybox rootfs 启动脚本分析(一）

    imx6文件系统启动脚本分析.开机运行/sbin/init,读取/etc/inittab文件,进行初始化. 参考链接 http://blog.163.com/wghbeyond@126/blog/st ...

  9. Hadoop 2.0+YARN启动脚本分析与实战经验

    start-all.sh脚本现在已经废弃,推荐使用start-dfs.sh和start-yarn.sh分别启动HDFS和YARN. 在新一代的Hadoop里面HDFS称为了统一存储的平台,而YARN成 ...

随机推荐

  1. Oracle 12c RAC 安装文档

    参考文档: https://docs.oracle.com/en/database/oracle/oracle-database/12.2/cwlin/index.html https://docs. ...

  2. GCD多线程的一个全面的题目

    GCD多线程的一个全面的题目  

  3. 理解ECS的概念和Unity中的ECS设计

    组合优于继承 ecs的概念很早就有了,最初的主要目的应该还是为了改善设计. e-c-s三者都有其意义,e-c是组合优于继承,主要用以改善oo的继承耦合过重以及多继承菱形问题. oop常见设计里,每个g ...

  4. AnyDesk远程连接及异常处理

    远程协助工具,用得最普遍的非QQ莫属,毕竟用户量在这里摆着的.不过,用户体验效果还不太理想,你懂得.接下来分享两个工具,一个是TeamViewer,另一个是AnyDesk.你更倾向于哪一款呢? 一.T ...

  5. SQLServer之创建唯一非聚集索引

    创建唯一非聚集索引典型实现 唯一索引可通过以下方式实现: PRIMARY KEY 或 UNIQUE 约束 在创建 PRIMARY KEY 约束时,如果不存在该表的聚集索引且未指定唯一非聚集索引,则将自 ...

  6. bootatrsp datetimepicker的初始化和阻止模态窗关闭(事件冒泡)

    1.github下载资源包  http://www.bootcss.com/p/bootstrap-datetimepicker/ 2.引入bootstrap-datetimepicker.min.c ...

  7. ASP.NET MVC学习系列（4）——MVC过滤器FilterAttribute

    1.概括 MVC提供的几种过滤器其实也是一种特性(Attribute),MVC支持的过滤器类型有四种,分别是:AuthorizationFilter(授权),ActionFilter(行为),Resu ...

  8. C. Maximal Intersection（STL）

    这道题,关键在于怎么求多个区间的交集,使用multiset就可以 分别将 r , l 存在不同的mutiset中. 然后,我们来看一下 是不是 交集的 l 是最大的, 交集的 r 是最小的 #incl ...

  9. android H5支付 网络环境未能通过安全验证,请稍后再试

    android做混合开发微信H5支付时碰到的一个问题. 解决办法:把所使用的WebView中重新如下方法即可 webView.setWebViewClient(new WebViewClient() ...

  10. left join inner join 区别

    left 以左表为准,左表在右表没有对应的记录,也为显示(右表字段填空). inner 需要满足两张表都有记录. 不管哪种join 一对多最终的结局 只会是多条记录