hadoop NameNode HA 和ResouceManager HA

官网配置地址：

HDFS HA : http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html

ResourceManager HA : http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

安装jdk

关闭防火墙

hadoop自动HA借助于zookeeper实现，整体架构如下：

m2和m3作为NameNode节点应该配置与其他所有节点的SSH无密码登录

m4和m5应该与m6、m7、m8配置SSH无密码登录

core-site.xml具体配置

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

  <name>fs.defaultFS</name>

  <value>hdfs://cluster</value>

</property>

<property>

  <name>hadoop.tmp.dir</name>

  <value>/home/hadoop/app/hadoop-2.7.3/tmp/data</value>

</property>

<property>

   <name>ha.zookeeper.quorum</name>

   <value>m6:2181,m7:2181,m8:2181</value>

 </property>

</configuration>

hdfs-site.xml具体配置

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

  <name>dfs.nameservices</name>

  <value>cluster</value>

</property>

<property>

  <name>dfs.ha.namenodes.cluster</name>

  <value>nn1,nn2</value>

</property>

<property>

  <name>dfs.namenode.rpc-address.cluster.nn1</name>

  <value>m2:9820</value>

</property>

<property>

  <name>dfs.namenode.rpc-address.cluster.nn2</name>

  <value>m3:9820</value>

</property>

<property>

  <name>dfs.namenode.http-address.cluster.nn1</name>

  <value>m2:9870</value>

</property>

<property>

  <name>dfs.namenode.http-address.cluster.nn2</name>

  <value>m3:9870</value>

</property>

<property>

  <name>dfs.namenode.shared.edits.dir</name>

  <value>qjournal://m6:8485;m7:8485;m8:8485;/cluster</value>

</property>

<property>

  <name>dfs.client.failover.proxy.provider.cluster</name>

  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<property>

   <name>dfs.ha.fencing.methods</name>

   <value>

        sshfence

        shell(shell(/bin/true))

   </value>

</property>

<property>

   <name>dfs.ha.fencing.ssh.private-key-files</name>

   <value>/home/hadoop/.ssh/id_rsa</value>

</property>

<property>

  <name>dfs.journalnode.edits.dir</name>

  <value>/home/hadoop/app/hadoop-2.7.3/journalnode/data</value>

</property>

<property>

  <name>dfs.replication</name>

  <value>3</value>

</property>

 <property>

   <name>dfs.ha.automatic-failover.enabled</name>

   <value>true</value>

 </property>

</configuration>

yarn-site.xml具体配置

<?xml version="1.0"?>

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

  <name>yarn.resourcemanager.ha.enabled</name>

  <value>true</value>

</property>

<property>

  <name>yarn.resourcemanager.cluster-id</name>

  <value>cluster</value>

</property>

<property>

  <name>yarn.resourcemanager.ha.rm-ids</name>

  <value>rm1,rm2</value>

</property>

<property>

  <name>yarn.resourcemanager.hostname.rm1</name>

  <value>m4</value>

</property>

<property>

  <name>yarn.resourcemanager.hostname.rm2</name>

  <value>m5</value>

</property>

<property>

  <name>yarn.resourcemanager.webapp.address.rm1</name>

  <value>m4:8088</value>

</property>

<property>

  <name>yarn.resourcemanager.webapp.address.rm2</name>

  <value>m5:8088</value>

</property>

<property>

  <name>yarn.resourcemanager.zk-address</name>

  <value>m6:2181,m7:2181,m8:2181</value>

</property>

<property>

  <name>yarn.nodemanager.aux-services</name>

  <value>mapreduce_shuffle</value>

</property>

</configuration>

mapred-site.xml具体配置

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

    <name>mapreduce.framework.name</name>

    <value>yarn</value>

</property>

</configuration>

slaves具体配置

m6

m7

m8

拷贝hadoop到m3、m4、m5、m6、m7、m8

scp -r hadoop-2.7.3/ m3:/home/hadoop/app/

scp -r hadoop-2.7.3/ m4:/home/hadoop/app/

scp -r hadoop-2.7.3/ m5:/home/hadoop/app/

scp -r hadoop-2.7.3/ m6:/home/hadoop/app/

scp -r hadoop-2.7.3/ m7:/home/hadoop/app/

scp -r hadoop-2.7.3/ m8:/home/hadoop/app/

zookeeper配置zoo.cfg（m6 m7 m8）　

# The number of milliseconds of each tick

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

dataDir=/home/hadoop/app/zookeeper-3.3.6/data

# the port at which the clients will connect

clientPort=2181

server.1=m6:2888:3888

server.2=m7:2888:3888

server.3=m8:2888:3888

配置好后的启动顺序：

1、启动zookeeper ./bin/zkServer.sh start

2、分别在m6 m7 m8上启动journalnode， ./hadoop-daemon.sh start journalnode只有第一次才需要手动启动journalnode，以后启动hdfs的时候会自动启动journalnode

3、在m2上格式化namenode，格式化成功后拷贝元数据到m3节点上

4、格式化zkfc ./bin/hdfs zkfc -formatZK 只需要一次

5、启动hdfs

6、启动yarn

验证：

通过kill命令杀死namenode进程观察namenode节点是否会自动切换

yarn rmadmin -getServiceState rm1查看那个resourceManager是active那个是standby

单独启动namenode: ./sbin/hadoop-daemon.sh start namenode