Apache Hadoop 2.9.2 的YARN High Available 模式部署
Apache Hadoop 2.9.2 的YARN High Available 模式部署
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.环境准备
1>.官方文档（http://hadoop.apache.org/docs/r2.9.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html）
2>.实验环境
环境在上一篇博客基础之上，参考链接：https://www.cnblogs.com/yinzhengjie/p/10720586.html。
二.修改核心配置
1>.修改yarn-site.xml
[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-2.9./etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property> <!--启用resourcemanager ha-->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property> <!--声明两台resourcemanager的地址-->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yinzhengjie-yarn</value>
</property> <property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>resourcemanager1,resourcemanager2</value>
</property> <property>
<name>yarn.resourcemanager.hostname.resourcemanager1</name>
<value>node101.yinzhengjie.org.cn</value>
</property> <property>
<name>yarn.resourcemanager.hostname.resourcemanager2</name>
<value>node103.yinzhengjie.org.cn</value>
</property> <property>
<name>yarn.resourcemanager.webapp.address.resourcemanager1</name>
<value>node101.yinzhengjie.org.cn:</value>
</property> <property>
<name>yarn.resourcemanager.webapp.address.resourcemanager2</name>
<value>node103.yinzhengjie.org.cn:</value>
</property> <!--指定zookeeper集群的地址-->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>node101.yinzhengjie.org.cn:,node102.yinzhengjie.org.cn:,node101.yinzhengjie.org.cn:</value>
</property> <!--启用自动恢复-->
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property> <!--指定resourcemanager的状态信息存储在zookeeper集群-->
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property> </configuration>
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml
2>.将修改的配置文件同步到其他节点中
[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9./etc/hadoop/yarn-site.xml node102.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9./etc/hadoop/
yarn-site.xml % .8MB/s :
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml node102.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/
[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9./etc/hadoop/yarn-site.xml node103.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9./etc/hadoop/
yarn-site.xml % .0MB/s :
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# scp /yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/yarn-site.xml node103.yinzhengjie.org.cn:/yinzhengjie/softwares/hadoop-2.9.2/etc/hadoop/
3>.启动hdfs集群
[root@node101.yinzhengjie.org.cn ~]# start-dfs.sh
Starting namenodes on [node101.yinzhengjie.org.cn node103.yinzhengjie.org.cn]
node101.yinzhengjie.org.cn: starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-namenode-node101.yinzhengjie.org.cn.out
node103.yinzhengjie.org.cn: starting namenode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-namenode-node103.yinzhengjie.org.cn.out
node103.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-datanode-node103.yinzhengjie.org.cn.out
node102.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-datanode-node102.yinzhengjie.org.cn.out
node101.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-datanode-node101.yinzhengjie.org.cn.out
Starting journal nodes [node103.yinzhengjie.org.cn node102.yinzhengjie.org.cn node101.yinzhengjie.org.cn]
node103.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-journalnode-node103.yinzhengjie.org.cn.out
node102.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-journalnode-node102.yinzhengjie.org.cn.out
node101.yinzhengjie.org.cn: starting journalnode, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-journalnode-node101.yinzhengjie.org.cn.out
Starting ZK Failover Controllers on NN hosts [node101.yinzhengjie.org.cn node103.yinzhengjie.org.cn]
node101.yinzhengjie.org.cn: starting zkfc, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-zkfc-node101.yinzhengjie.org.cn.out
node103.yinzhengjie.org.cn: starting zkfc, logging to /yinzhengjie/softwares/hadoop-2.9./logs/hadoop-root-zkfc-node103.yinzhengjie.org.cn.out
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# hdfs haadmin -getServiceState namenode1 #查看当前节点的状态！很显然，和下面的Web UI展示的结果一致！
active
[root@node101.yinzhengjie.org.cn ~]#
4>.启动yarn集群
[root@node101.yinzhengjie.org.cn ~]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /yinzhengjie/softwares/hadoop-2.9./logs/yarn-root-resourcemanager-node101.yinzhengjie.org.cn.out
node102.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.9./logs/yarn-root-nodemanager-node102.yinzhengjie.org.cn.out
node103.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.9./logs/yarn-root-nodemanager-node103.yinzhengjie.org.cn.out
node101.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.9./logs/yarn-root-nodemanager-node101.yinzhengjie.org.cn.out
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# start-yarn.sh #在resourcemanager1节点上启动YARN
[root@node101.yinzhengjie.org.cn ~]# jps
ZooKeeperMain
Jps
QuorumPeerMain
DFSZKFailoverController
JournalNode
NameNode
NodeManager
ResourceManager
DataNode
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# jps
[root@node103.yinzhengjie.org.cn ~]# yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /yinzhengjie/softwares/hadoop-2.9./logs/yarn-root-resourcemanager-node103.yinzhengjie.org.cn.out
[root@node103.yinzhengjie.org.cn ~]#
[root@node103.yinzhengjie.org.cn ~]# jps
Jps
ResourceManager
JournalNode
NodeManager
QuorumPeerMain
NameNode
DFSZKFailoverController
DataNode
[root@node103.yinzhengjie.org.cn ~]#
[root@node103.yinzhengjie.org.cn ~]# yarn-daemon.sh start resourcemanager #在resourcemanager2节点上启动rm进程
[root@node103.yinzhengjie.org.cn ~]# jps
ResourceManager
JournalNode
NodeManager
QuorumPeerMain
NameNode
DFSZKFailoverController
Jps
DataNode
[root@node103.yinzhengjie.org.cn ~]#
[root@node103.yinzhengjie.org.cn ~]# jps
[root@node103.yinzhengjie.org.cn ~]# yarn rmadmin -getServiceState resourcemanager1 #很显然，我们先启动的是resourcemanager1节点，因此它优先被选为Active状态！
active
[root@node103.yinzhengjie.org.cn ~]#
[root@node103.yinzhengjie.org.cn ~]# yarn rmadmin -getServiceState resourcemanager2
standby
[root@node103.yinzhengjie.org.cn ~]#
三.验证YARN的高可用性
1>.将resourcemanager1节点进程杀掉，查看服务是否可以正常提供服务
[root@node101.yinzhengjie.org.cn ~]# jps
ZooKeeperMain
QuorumPeerMain
DFSZKFailoverController
JournalNode
NameNode
NodeManager
ResourceManager
DataNode
Jps
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# kill - `jps | grep ResourceManager | awk '{print $1}' `
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# jps
ZooKeeperMain
QuorumPeerMain
Jps
DFSZKFailoverController
JournalNode
NameNode
NodeManager
DataNode
[root@node101.yinzhengjie.org.cn ~]#
2>.启动yarn集群后，注意观察zookeeper集群中生成的目录结构的变化哟！
[root@node101.yinzhengjie.org.cn ~]# zkCli.sh -server node103.yinzhengjie.org.cn:
Connecting to node103.yinzhengjie.org.cn:
-- ::, [myid:] - INFO [main:Environment@] - Client environment:zookeeper.version=3.4.-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on // : GMT
-- ::, [myid:] - INFO [main:Environment@] - Client environment:host.name=node101.yinzhengjie.org.cn
-- ::, [myid:] - INFO [main:Environment@] - Client environment:java.version=1.8.0_201
-- ::, [myid:] - INFO [main:Environment@] - Client environment:java.vendor=Oracle Corporation
-- ::, [myid:] - INFO [main:Environment@] - Client environment:java.home=/usr/lib/jvm/java-1.8.-openjdk-1.8.0.201.b09-.el7_6.x86_64/jre
-- ::, [myid:] - INFO [main:Environment@] - Client environment:java.class.path=/yinzhengjie/softwares/zookeeper-3.4./bin/../zookeeper-server/target/classes:/yinzhengjie/softwares/zookeeper-3.4./bin/../build/classes:/yinzhengjie/softwares/zookeeper-3.4./bin/../zookeeper-server/target/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../build/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/yinzhengjie/softwares/zookeeper-3.4.14/bin/../conf:
2019-04-19 10:22:52,912 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-04-19 10:22:52,912 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-04-19 10:22:52,912 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-04-19 10:22:52,912 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-04-19 10:22:52,912 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-04-19 10:22:52,912 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-957.el7.x86_64
2019-04-19 10:22:52,912 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2019-04-19 10:22:52,912 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2019-04-19 10:22:52,912 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/root
2019-04-19 10:22:52,913 [myid:] - INFO [main:ZooKeeper@442] - Initiating client connection, connectString=node103.yinzhengjie.org.cn:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
Welcome to ZooKeeper!
2019-04-19 10:22:52,938 [myid:] - INFO [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server node103.yinzhengjie.org.cn/172.30.1.103:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-04-19 10:22:52,998 [myid:] - INFO [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@879] - Socket connection established to node103.yinzhengjie.org.cn/172.30.1.103:2181, initiating session
[zk: node103.yinzhengjie.org.cn:2181(CONNECTING) 0] 2019-04-19 10:22:53,026 [myid:] - INFO [main-SendThread(node103.yinzhengjie.org.cn:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server node103.yinzhengjie.org.cn/172.30.1.103:2181, sessionid = 0x6700035fda3a0000, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 0] ls /
[zookeeper,hadoop-ha]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 1]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 1] ls /
[zookeeper, yarn-leader-election, hadoop-ha, rmstore]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 2]
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ] get /yarn-leader-election cZxid = 0x300000008
ctime = Fri Apr :: CST
mZxid = 0x300000008
mtime = Fri Apr :: CST
pZxid = 0x300000009
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 2] get /yarn-leader-election
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ] ls /yarn-leader-election
[yinzhengjie-yarn]
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 3] ls /yarn-leader-election
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ] ls /yarn-leader-election/yinzhengjie-yarn
[ActiveBreadCrumb, ActiveStandbyElectorLock]
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 4] ls /yarn-leader-election/yinzhengjie-yarn
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ] get /yarn-leader-election/yinzhengjie-yarn/ActiveBreadCrumb yinzhengjie-yarnresourcemanager2
cZxid = 0x30000000b
ctime = Fri Apr :: CST
mZxid = 0x30000002c
mtime = Fri Apr :: CST
pZxid = 0x30000000b
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x0
dataLength =
numChildren =
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 7] get /yarn-leader-election/yinzhengjie-yarn/ActiveBreadCrumb
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ] get /yarn-leader-election/yinzhengjie-yarn/ActiveStandbyElectorLock yinzhengjie-yarnresourcemanager2
cZxid = 0x30000002b
ctime = Fri Apr :: CST
mZxid = 0x30000002b
mtime = Fri Apr :: CST
pZxid = 0x30000002b
cversion =
dataVersion =
aclVersion =
ephemeralOwner = 0x660003eeeeb10001
dataLength =
numChildren =
[zk: node103.yinzhengjie.org.cn:(CONNECTED) ]
[zk: node103.yinzhengjie.org.cn:2181(CONNECTED) 8] get /yarn-leader-election/yinzhengjie-yarn/ActiveStandbyElectorLock
Apache Hadoop 2.9.2 的YARN High Available 模式部署的更多相关文章
- Apache Hadoop 2.9.2 的HDFS High Available模式部署
Apache Hadoop 2.9.2 的HDFS High Available 模式部署 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 我们知道,当NameNode进程挂掉后,可 ...
- Apache Hadoop下一代MapReduce框架（YARN）简介 （Apache Hadoop NextGen MapReduce (YARN)）
英文看着头大,先试着翻译一下. E文原文:http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/YARN ...
- kylin_异常_02_java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf 解决办法
一.异常现象 在kylin的web管理界面,设置hive数据源时,报错: 查找kylin的日志时发现,弹出提示框的原因是因为出现错误: ERROR [http-bio-7070-exec-10] co ...
- hadoop错误org.apache.hadoop.yarn.exceptions.YarnException Unauthorized request to start container
错误: 14/04/29 02:45:07 INFO mapreduce.Job: Job job_1398704073313_0021 failed with state FAILED due to ...
- Apache Hadoop YARN： 背景及概述
从2012年8月开始Apache Hadoop YARN(YARN = Yet Another Resource Negotiator)成了Apache Hadoop的一项子工程.自此Apache H ...
- Apache Hadoop集群安装（NameNode HA + YARN HA + SPARK + 机架感知）
1.主机规划 序号 主机名 IP地址 角色 1 nn-1 192.168.9.21 NameNode.mr-jobhistory.zookeeper.JournalNode 2 nn-2 192.16 ...
- Apache hadoop namenode ha和yarn ha ---HDFS高可用性
HDFS高可用性Hadoop HDFS 的两大问题:NameNode单点:虽然有StandbyNameNode,但是冷备方案,达不到高可用--阶段性的合并edits和fsimage,以缩短集群启动的时 ...
- Apache Hadoop集群离线安装部署(二)——Spark-2.1.0 on Yarn安装
Apache Hadoop集群离线安装部署(一)——Hadoop(HDFS.YARN.MR)安装:http://www.cnblogs.com/pojishou/p/6366542.html Apac ...
- Apache Hadoop集群离线安装部署(一)——Hadoop(HDFS、YARN、MR)安装
虽然我已经装了个Cloudera的CDH集群(教程详见:http://www.cnblogs.com/pojishou/p/6267616.html),但实在太吃内存了,而且给定的组件版本是不可选的, ...
随机推荐
- oppo6.0系统设备(亲测有效)激活xposed框架的流程
对于喜欢研究手机的伙伴来说,常常会用到xposed框架以及种类繁多功能强大的模块,对于5.0以下的系统版本,只要手机能获得Root权限,安装和激活xposed框架是异常简易的,但随着系统版本的不断升级 ...
- json-sever 配置与应用
先安装 node.js. 全局安装 json-server: npm install json-server -g 检查是否安装成功: json-server -h 安装好后,json-server的 ...
- Centos7 使用 kubeadm 安装Kubernetes 1.13.3
目录 目录 什么是Kubeadm? 什么是容器存储接口(CSI)? 什么是CoreDNS? 1.环境准备 1.1.网络配置 1.2.更改 hostname 1.3.配置 SSH 免密码登录登录 1.4 ...
- DataPipeline | 享物说产品负责人夏凯：数据驱动的用户增长实战
夏凯 卡内基梅隆大学计算机系毕业,曾供职于Evernote数据团队和微软Bing.com搜索引擎广告部门.回国后作为早期成员加入小红书,先后从事大数据,用户增长,项目和团队管理等工作. 我最初是在美国 ...
- 利用Navicat高效率postgresql转mysql数据库
本人很喜欢postgresql数据库,也一直认为postgresql比mysql要更好更强大. 可生态环境太差了,无奈,最近要把一个小站转成mysql数据库. 小站主要表数据110万,pg_dump备 ...
- Linux（一）—— Unix&Linux 历史
Linux(一)-- Unix&Linux 历史 Unix =Unix内核+Unix实用工具 Unix Unix 的诞生 Unix的历史可以追溯到20世纪60年代中期,当时麻省理工学院,AT& ...
- python学习——读取染色体长度（一、简化问题）
# 读取fasta # 解析每条序列的长度 chr1_len = 10 chr2_len = 20 chr3_len = 30 chr4_len = 40 chr5_len = 50 # 求和 tot ...
- 超链接标签绑定JS事件&&不加"javascript:;"导致的杯具
很久以来,在写Html和JS时,经常会给超链接<a>标签,绑定JS事件. 我们经常看到这样的写法,<a href="javascript:;" onclick=& ...
- 说说我心中的Linux系统
我不知道在阅读此篇文章的你,是一个什么样的人,或许你只是偶然看到此篇文章的路人,或许是对linux有兴趣但没接触过linux的圈外人,或许是已经入行没多久的菜鸟,或许是喜欢linux却学习不下去预备放 ...
- MongoDB之常用操作
最近经常使用MongoDB来进行数据的操作,特此记录总结一下