Apache Hadoop 2.9.2 的YARN High Available 模式部署

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

一.环境准备

1>.官方文档（http://hadoop.apache.org/docs/r2.9.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html）

2>.实验环境

　　环境在上一篇博客基础之上，参考链接：https://www.cnblogs.com/yinzhengjie/p/10720586.html。

二.修改核心配置

1>.修改yarn-site.xml

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-2.9./etc/hadoop/yarn-site.xml

<configuration>

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <!--启用resourcemanager ha-->

    <property>

        <name>yarn.resourcemanager.ha.enabled</name>

        <value>true</value>

    </property>

    <!--声明两台resourcemanager的地址-->

    <property>

        <name>yarn.resourcemanager.cluster-id</name>

        <value>yinzhengjie-yarn</value>

    </property>

    <property>

        <name>yarn.resourcemanager.ha.rm-ids</name>

        <value>resourcemanager1,resourcemanager2</value>

    </property>

    <property>

        <name>yarn.resourcemanager.hostname.resourcemanager1</name>

        <value>node101.yinzhengjie.org.cn</value>

    </property>

    <property>

        <name>yarn.resourcemanager.hostname.resourcemanager2</name>

        <value>node103.yinzhengjie.org.cn</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address.resourcemanager1</name>

        <value>node101.yinzhengjie.org.cn:</value>

    </property>

   <property>

        <name>yarn.resourcemanager.webapp.address.resourcemanager2</name>

        <value>node103.yinzhengjie.org.cn:</value>

    </property>

    <!--指定zookeeper集群的地址-->

    <property>

        <name>yarn.resourcemanager.zk-address</name>

        <value>node101.yinzhengjie.org.cn:,node102.yinzhengjie.org.cn:,node101.yinzhengjie.org.cn:</value>

    </property>

    <!--启用自动恢复-->

    <property>

        <name>yarn.resourcemanager.recovery.enabled</name>

        <value>true</value>

    </property>

    <!--指定resourcemanager的状态信息存储在zookeeper集群-->

    <property>

        <name>yarn.resourcemanager.store.class</name>

        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>

</property>

</configuration>

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml

2>.将修改的配置文件同步到其他节点中

[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9./etc/hadoop/yarn-site.xml  node102.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9./etc/hadoop/

yarn-site.xml                                                                                                                                                                                                             %      .8MB/s   :

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml node102.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/

[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9./etc/hadoop/yarn-site.xml  node103.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9./etc/hadoop/

yarn-site.xml                                                                                                                                                                                                             %      .0MB/s   :

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml node103.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/

3>.启动hdfs集群

[root@node101.yinzhengjie.org.cn ~]# start-dfs.sh

Starting namenodes on [node101.yinzhengjie.org.cn node103.yinzhengjie.org.cn]

node101.yinzhengjie.org.cn: starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-namenode-node101.yinzhengjie.org.cn.out

node103.yinzhengjie.org.cn: starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-namenode-node103.yinzhengjie.org.cn.out

node103.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-datanode-node103.yinzhengjie.org.cn.out

node102.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-datanode-node102.yinzhengjie.org.cn.out

node101.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-datanode-node101.yinzhengjie.org.cn.out

Starting journal nodes [node103.yinzhengjie.org.cn node102.yinzhengjie.org.cn node101.yinzhengjie.org.cn]

node103.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-journalnode-node103.yinzhengjie.org.cn.out

node102.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-journalnode-node102.yinzhengjie.org.cn.out

node101.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-journalnode-node101.yinzhengjie.org.cn.out

Starting ZK Failover Controllers on NN hosts [node101.yinzhengjie.org.cn node103.yinzhengjie.org.cn]

node101.yinzhengjie.org.cn: starting zkfc, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-zkfc-node101.yinzhengjie.org.cn.out

node103.yinzhengjie.org.cn: starting zkfc, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-zkfc-node103.yinzhengjie.org.cn.out

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# hdfs haadmin -getServiceState namenode1　　　　　　　　　　　　#查看当前节点的状态！很显然，和下面的Web UI展示的结果一致！

active

[root@node101.yinzhengjie.org.cn ~]#

4>.启动yarn集群

[root@node101.yinzhengjie.org.cn ~]# start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /yinzhengjie/softwares/hadoop-2.9./logs/yarn-root-resourcemanager-node101.yinzhengjie.org.cn.out

node102.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.9./logs/yarn-root-nodemanager-node102.yinzhengjie.org.cn.out

node103.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.9./logs/yarn-root-nodemanager-node103.yinzhengjie.org.cn.out

node101.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.9./logs/yarn-root-nodemanager-node101.yinzhengjie.org.cn.out

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# start-yarn.sh　　　　　　　　　　　　　　　　　　　　　　　　#在resourcemanager1节点上启动YARN

[root@node101.yinzhengjie.org.cn ~]# jps

 ZooKeeperMain

 Jps

 QuorumPeerMain

 DFSZKFailoverController

 JournalNode

 NameNode

 NodeManager

 ResourceManager

 DataNode

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# jps

[root@node103.yinzhengjie.org.cn ~]# yarn-daemon.sh start resourcemanager

starting resourcemanager, logging to /yinzhengjie/softwares/hadoop-2.9./logs/yarn-root-resourcemanager-node103.yinzhengjie.org.cn.out

[root@node103.yinzhengjie.org.cn ~]#

[root@node103.yinzhengjie.org.cn ~]# jps

 Jps

 ResourceManager

 JournalNode

 NodeManager

 QuorumPeerMain

 NameNode

 DFSZKFailoverController

 DataNode

[root@node103.yinzhengjie.org.cn ~]#

[root@node103.yinzhengjie.org.cn ~]# yarn-daemon.sh start resourcemanager　　　　　　　　　　#在resourcemanager2节点上启动rm进程

[root@node103.yinzhengjie.org.cn ~]# jps

 ResourceManager

 JournalNode

 NodeManager

 QuorumPeerMain

 NameNode

 DFSZKFailoverController

 Jps

 DataNode

[root@node103.yinzhengjie.org.cn ~]#

[root@node103.yinzhengjie.org.cn ~]# jps

[root@node103.yinzhengjie.org.cn ~]# yarn rmadmin -getServiceState resourcemanager1　　　　　　　　#很显然，我们先启动的是resourcemanager1节点，因此它优先被选为Active状态！

active

[root@node103.yinzhengjie.org.cn ~]#

[root@node103.yinzhengjie.org.cn ~]# yarn rmadmin -getServiceState resourcemanager2

standby

[root@node103.yinzhengjie.org.cn ~]#

三.验证YARN的高可用性

1>.将resourcemanager1节点进程杀掉，查看服务是否可以正常提供服务

[root@node101.yinzhengjie.org.cn ~]# jps

 ZooKeeperMain

 QuorumPeerMain

 DFSZKFailoverController

 JournalNode

 NameNode

 NodeManager

 ResourceManager

 DataNode

 Jps

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kill - `jps | grep ResourceManager | awk '{print $1}' `

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# jps

 ZooKeeperMain

 QuorumPeerMain

 Jps

 DFSZKFailoverController

 JournalNode

 NameNode

 NodeManager

 DataNode

[root@node101.yinzhengjie.org.cn ~]#  

2>.启动yarn集群后，注意观察zookeeper集群中生成的目录结构的变化哟！

[root@node101.yinzhengjie.org.cn ~]# zkCli.sh -server node103.yinzhengjie.org.cn:

Connecting to node103.yinzhengjie.org.cn:

-- ::, [myid:] - INFO  [main:Environment@] - Client environment:zookeeper.version=3.4.-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on // : GMT

-- ::, [myid:] - INFO  [main:Environment@] - Client environment:host.name=node101.yinzhengjie.org.cn

-- ::, [myid:] - INFO  [main:Environment@] - Client environment:java.version=1.8.0_201

-- ::, [myid:] - INFO  [main:Environment@] - Client environment:java.vendor=Oracle Corporation

-- ::, [myid:] - INFO  [main:Environment@] - Client environment:java.home=/usr/lib/jvm/java-1.8.-openjdk-1.8.0.201.b09-.el7_6.x86_64/jre

-- ::, [myid:] - INFO  [main:Environment@] - Client environment:java.class.path=/yinzhengjie/softwares/zookeeper-3.4./bin/../zookeeper-server/target/classes:/yinzhengjie/softwares/zookeeper-3.4./bin/../build/classes:/yinzhengjie/softwares/zookeeper-3.4./bin/../zookeeper-server/target/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf:

2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp

2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>

2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux

2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64

2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-957.el7.x86_64

2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root

2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root

2019-04-19 10:22:52,912 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/root

2019-04-19 10:22:52,913 [myid:] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=node103.yinzhengjie.org.cn:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89

Welcome to ZooKeeper!

2019-04-19 10:22:52,938 [myid:] - INFO  [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server node103.yinzhengjie.org.cn/172.30.1.103:2181. Will not attempt to authenticate using SASL (unknown error)

JLine support is enabled

2019-04-19 10:22:52,998 [myid:] - INFO  [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@879] - Socket connection established to node103.yinzhengjie.org.cn/172.30.1.103:2181, initiating session

[zk: node103.yinzhengjie.org.cn:2181(CONNECTING) 0] 2019-04-19 10:22:53,026 [myid:] - INFO  [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server node103.yinzhengjie.org.cn/172.30.1.103:2181, sessionid = 0x6700035fda3a0000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 0] ls /

[zookeeper,hadoop-ha]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 1]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 1] ls /

[zookeeper, yarn-leader-election, hadoop-ha, rmstore]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 2]
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ] get /yarn-leader-election

cZxid = 0x300000008

ctime = Fri Apr  :: CST

mZxid = 0x300000008

mtime = Fri Apr  :: CST

pZxid = 0x300000009

cversion =

dataVersion =

aclVersion =

ephemeralOwner = 0x0

dataLength =

numChildren =

[zk: node103.yinzhengjie.org.cn:(CONNECTED) ]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 2] get /yarn-leader-election

[zk: node103.yinzhengjie.org.cn:(CONNECTED) ] ls /yarn-leader-election

[yinzhengjie-yarn]

[zk: node103.yinzhengjie.org.cn:(CONNECTED) ]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 3] ls /yarn-leader-election

[zk: node103.yinzhengjie.org.cn:(CONNECTED) ] ls /yarn-leader-election/yinzhengjie-yarn

[ActiveBreadCrumb, ActiveStandbyElectorLock]

[zk: node103.yinzhengjie.org.cn:(CONNECTED) ]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 4] ls /yarn-leader-election/yinzhengjie-yarn

[zk: node103.yinzhengjie.org.cn:(CONNECTED) ] get /yarn-leader-election/yinzhengjie-yarn/ActiveBreadCrumb       

yinzhengjie-yarnresourcemanager2

cZxid = 0x30000000b

ctime = Fri Apr  :: CST

mZxid = 0x30000002c

mtime = Fri Apr  :: CST

pZxid = 0x30000000b

cversion =

dataVersion =

aclVersion =

ephemeralOwner = 0x0

dataLength =

numChildren =

[zk: node103.yinzhengjie.org.cn:(CONNECTED) ]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 7] get /yarn-leader-election/yinzhengjie-yarn/ActiveBreadCrumb

[zk: node103.yinzhengjie.org.cn:(CONNECTED) ] get /yarn-leader-election/yinzhengjie-yarn/ActiveStandbyElectorLock

yinzhengjie-yarnresourcemanager2

cZxid = 0x30000002b

ctime = Fri Apr  :: CST

mZxid = 0x30000002b

mtime = Fri Apr  :: CST

pZxid = 0x30000002b

cversion =

dataVersion =

aclVersion =

ephemeralOwner = 0x660003eeeeb10001

dataLength =

numChildren =

[zk: node103.yinzhengjie.org.cn:(CONNECTED) ]

[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 8] get /yarn-leader-election/yinzhengjie-yarn/ActiveStandbyElectorLock

Apache Hadoop 2.9.2 的YARN High Available 模式部署的更多相关文章

  1. Apache Hadoop 2.9.2 的HDFS High Available模式部署

    Apache Hadoop 2.9.2 的HDFS High Available 模式部署 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 我们知道,当NameNode进程挂掉后,可 ...

  2. Apache Hadoop下一代MapReduce框架（YARN）简介 （Apache Hadoop NextGen MapReduce (YARN)）

    英文看着头大,先试着翻译一下. E文原文:http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/YARN ...

  3. kylin_异常_02_java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf 解决办法

    一.异常现象 在kylin的web管理界面,设置hive数据源时,报错: 查找kylin的日志时发现,弹出提示框的原因是因为出现错误: ERROR [http-bio-7070-exec-10] co ...

  4. hadoop错误org.apache.hadoop.yarn.exceptions.YarnException Unauthorized request to start container

    错误: 14/04/29 02:45:07 INFO mapreduce.Job: Job job_1398704073313_0021 failed with state FAILED due to ...

  5. Apache Hadoop YARN： 背景及概述

    从2012年8月开始Apache Hadoop YARN(YARN = Yet Another Resource Negotiator)成了Apache Hadoop的一项子工程.自此Apache H ...

  6. Apache Hadoop集群安装（NameNode HA + YARN HA + SPARK + 机架感知）

    1.主机规划 序号 主机名 IP地址 角色 1 nn-1 192.168.9.21 NameNode.mr-jobhistory.zookeeper.JournalNode 2 nn-2 192.16 ...

  7. Apache hadoop namenode ha和yarn ha ---HDFS高可用性

    HDFS高可用性Hadoop HDFS 的两大问题:NameNode单点:虽然有StandbyNameNode,但是冷备方案,达不到高可用--阶段性的合并edits和fsimage,以缩短集群启动的时 ...

  8. Apache Hadoop集群离线安装部署(二)——Spark-2.1.0 on Yarn安装

    Apache Hadoop集群离线安装部署(一)——Hadoop(HDFS.YARN.MR)安装:http://www.cnblogs.com/pojishou/p/6366542.html Apac ...

  9. Apache Hadoop集群离线安装部署(一)——Hadoop(HDFS、YARN、MR)安装

    虽然我已经装了个Cloudera的CDH集群(教程详见:http://www.cnblogs.com/pojishou/p/6267616.html),但实在太吃内存了,而且给定的组件版本是不可选的, ...

随机推荐

  1. 基于windows环境的Flask网站搭建（mysql + conda + redis）

    1下载mysql-installer-community-5.7.24.0.msi (https://dev.mysql.com/downloads/windows/installer/8.0.htm ...

  2. Handler,Looper,MessageQueue流程梳理

    目的:handle的出现主要是为了解决线程间通讯. 举个例子,android是不允许在主线程中访问网络,因为这样会阻塞主线程,影响性能,所以访问网络都是放在子线程中执行,对于网络返回的结果则需要显示在 ...

  3. 如何学好java

    今天发现这么一篇文章,对于笔者谈的:"一方面很努力学习,一方面又觉得不踏实",我感同身受.觉得文章写得不错,在此献给一些在java中努力的朋友们,希望能有所收获. 文章原内容: 近 ...

  4. scp远程拷贝文件及文件夹

    scp : 远程copy 命令 -r : 递归copy 从Linux Copy 到 Linux 从Linux Copy 到 Windows (当前目录使用. 就可以了) scp -r root@10. ...

  5. 【转】Dockerfile详解

    Dockerfile详解 https://blog.csdn.net/wo18237095579/article/details/80540571 --------------------- 作者:大 ...

  6. python☞自动发送邮件

    一.SMTP 协议 SMTP(Simple Mail Transfer Protocol)是简单邮件传输协议,它是一组用于由源地址到目的地址传送邮件的规则,由它来控制信件的中转方式 二.smtplib ...

  7. Shell企业案例实战和企业面试题

    shell企业面试题 1.批量创建带有随机小写字符文件程序 使用for循环在/pizza目录下创建10个html文件,其中每个文件包含10个随机小写字母加固定字母_pizza 1.思路分析: 核心是: ...

  8. 微信小程序-表单笔记

    发布页——向云端数据库上传多行文字和4张图片 第6,8行注释掉和不注释掉都可以实现数据上传 var _this = this; wx.cloud.callFunction({ name: 'searc ...

  9. 【学亮编程手记】Spring Cloud三大组件Eureka/Feign/Histrix的原理及使用

  10. 02——Solr学习之Solr安装与配置（linux上的安装）

    借鉴博客:https://www.jianshu.com/p/1100f54fcbd8 https://www.cnblogs.com/jepson6669/p/9134652.html 1.准备一个 ...