hadoop NameNode HA 和ResouceManager HA

官网配置地址：

HDFS HA : http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html

ResourceManager HA : http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

安装jdk

关闭防火墙

hadoop自动HA借助于zookeeper实现，整体架构如下：

m2和m3作为NameNode节点应该配置与其他所有节点的SSH无密码登录

m4和m5应该与m6、m7、m8配置SSH无密码登录

core-site.xml具体配置

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://cluster</value>
</property>

<property>
  <name>hadoop.tmp.dir</name>
  <value>/home/hadoop/app/hadoop-2.7.3/tmp/data</value>
</property>

<property>
   <name>ha.zookeeper.quorum</name>
   <value>m6:2181,m7:2181,m8:2181</value>
 </property>
</configuration>

hdfs-site.xml具体配置

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
  <name>dfs.nameservices</name>
  <value>cluster</value>
</property>

<property>
  <name>dfs.ha.namenodes.cluster</name>
  <value>nn1,nn2</value>
</property>

<property>
  <name>dfs.namenode.rpc-address.cluster.nn1</name>
  <value>m2:9820</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.cluster.nn2</name>
  <value>m3:9820</value>
</property>

<property>
  <name>dfs.namenode.http-address.cluster.nn1</name>
  <value>m2:9870</value>
</property>
<property>
  <name>dfs.namenode.http-address.cluster.nn2</name>
  <value>m3:9870</value>
</property>

<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://m6:8485;m7:8485;m8:8485;/cluster</value>
</property>

<property>
  <name>dfs.client.failover.proxy.provider.cluster</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

<property>
   <name>dfs.ha.fencing.methods</name>
   <value>
        sshfence
        shell(shell(/bin/true))
   </value>
</property>

<property>
   <name>dfs.ha.fencing.ssh.private-key-files</name>
   <value>/home/hadoop/.ssh/id_rsa</value>
</property>

<property>
  <name>dfs.journalnode.edits.dir</name>
  <value>/home/hadoop/app/hadoop-2.7.3/journalnode/data</value>
</property>

<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>

 <property>
   <name>dfs.ha.automatic-failover.enabled</name>
   <value>true</value>
 </property>

</configuration>

yarn-site.xml具体配置

<?xml version="1.0"?>

<configuration>

<!-- Site specific YARN configuration properties -->

<property>
  <name>yarn.resourcemanager.ha.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.cluster-id</name>
  <value>cluster</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.rm-ids</name>
  <value>rm1,rm2</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm1</name>
  <value>m4</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm2</name>
  <value>m5</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm1</name>
  <value>m4:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm2</name>
  <value>m5:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.zk-address</name>
  <value>m6:2181,m7:2181,m8:2181</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>

</configuration>

mapred-site.xml具体配置

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
</configuration>

slaves具体配置

m6
m7
m8

拷贝hadoop到m3、m4、m5、m6、m7、m8

scp -r hadoop-2.7.3/ m3:/home/hadoop/app/
scp -r hadoop-2.7.3/ m4:/home/hadoop/app/
scp -r hadoop-2.7.3/ m5:/home/hadoop/app/
scp -r hadoop-2.7.3/ m6:/home/hadoop/app/
scp -r hadoop-2.7.3/ m7:/home/hadoop/app/
scp -r hadoop-2.7.3/ m8:/home/hadoop/app/　

zookeeper配置zoo.cfg（m6 m7 m8）　

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/home/hadoop/app/zookeeper-3.3.6/data
# the port at which the clients will connect
clientPort=2181
server.1=m6:2888:3888
server.2=m7:2888:3888
server.3=m8:2888:3888

配置好后的启动顺序：

1、启动zookeeper   ./bin/zkServer.sh start

2、分别在m6  m7  m8上启动journalnode， ./hadoop-daemon.sh start journalnode只有第一次才需要手动启动journalnode，以后启动hdfs的时候会自动启动journalnode

3、在m2上格式化namenode，格式化成功后拷贝元数据到m3节点上

4、格式化zkfc  ./bin/hdfs zkfc -formatZK  只需要一次

5、启动hdfs

6、启动yarn

验证：

通过kill命令杀死namenode进程观察namenode节点是否会自动切换

yarn rmadmin -getServiceState rm1查看那个resourceManager是active那个是standby

单独启动namenode: ./sbin/hadoop-daemon.sh start namenode