hadoop 2.x 完全分布式搭建

HDFS HA 集群搭建：

　　DN（DataNode）：3个；NN（NameNode）：2；ZK(ZooKeeper)：3（大于1的奇数个）；ZKFC:和NN在同一台机器；JN：3；RM(ResourceManager):1；DM(DataManager):3个；与DN在同一台，就近原则

　　√表示在该机器上有该进程。

	NN	DN	ZK	ZKFC	JN	RM	DM
Node1	√		√	√		√
Node2	√	√	√	√	√		√
Node3		√	√		√		√
Node4		√			√		√

1.解压  hadoop-2.5.2.tar.gz

[hadoop@node1 software]$ tar -zxvf hadoop-2.5..tar.gz 

其中 -zxvf 含义如下：

-z, gzip : 对归档文件使用 gzip 压缩

-x, --extract, --get : 释放归档文件中文件及目录

-v, --verbose : 显示命令整个执行过程

-f, --file=ARCHIVE : 指定 (将要创建或已存在的) 归档文件名

这里注意，我们的环境为CentOS7 64位系统，这里的tar包也需要为64位，可以使用如下方法查看hadoop tar包是32位还是64位：

/hadoop-2.5./lib/native
[hadoop@node1 native]$ ls
libhadoop.a       libhadoop.so        libhadooputils.a  libhdfs.so
libhadooppipes.a  libhadoop.so.1.0.  libhdfs.a         libhdfs.so.0.0.

[hadoop@node1 native]$ file libhadoop.so.1.0.
libhadoop.so.1.0.: ELF -bit LSB shared object, x86-, version  (SYSV), dynamically linked, BuildID[sha1]=29e15e4c9d9840a7b96b5af3e5732e5935d91847, not stripped

2.进入hadoop解压后的目录修改hadoop-env.sh，主要修改JAVA_HOME

[hadoop@node1 hadoop]$ echo $JAVA_HOME
/usr/java/jdk1..0_75
[hadoop@node1 hadoop]$ vim hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1..0_75

2. 修改hdfs-site.xml,可以参照官档 http://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html

<configuration>
<!--配置服务名称，名称可以随意-->
<property>
  <name>dfs.nameservices</name>
  <value>mycluster</value>
</property>
<!--配置所有NameNode名称,这里名称随意-->
<property>
  <name>dfs.ha.namenodes.mycluster</name>
  <value>nn1,nn2</value>
</property>
<!--配置所有NameNode的RPC协议端口-->
<property>
  <name>dfs.namenode.rpc-address.mycluster.nn1</name>
  <value>node1.example.com:</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster.nn2</name>
  <value>node2.example.com:i8020</value>
</property>
<!--配置Http协议端口和主机-->
<property>
  <name>dfs.namenode.http-address.mycluster.nn1</name>
  <value>node1.example.com:</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.nn2</name>
  <value>node2.example.com:</value>
</property>
<!--配置JournalNodes 的地址-->
<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://node2:8485;node3:8485;node4:8485/mycluster</value>
</property>
<!--配置客户端要使用的类，客户端使用这个类找到Active NodeName-->
<property>
  <name>dfs.client.failover.proxy.provider.mycluster</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!--配置sshfence-->
<property>
  <name>dfs.ha.fencing.methods</name>
  <value>sshfence</value>
</property>
<property>
  <name>dfs.ha.fencing.ssh.private-key-files</name>
  <value>/home/hadoop/.ssh/id_dsa</value>
</property>

<!--配置JournalNodes的工作目录-->
<property>
  <name>dfs.journalnode.edits.dir</name>
  <value>/opt/jn/data</value>
</property>

<!--开启自动切换,当然手动切换也是可用用的-->
<property>
   <name>dfs.ha.automatic-failover.enabled</name>
   <value>true</value>
</property>

</configuration>

3.配置core-site.xml

<configuration>
<!--配置NameNode入口,这里配置集群名称，不能配置具体的ip-->
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://mycluster</value>
</property>
<!--配置zk，表明zk在哪些机器上有-->
<property>
  <name>ha.zookeeper.quorum</name>
  <value>node1:,node2:,node3:</value>
</property>
<!--修改hadoop的临时目录，默认目录在系统的tmp目录下-->
<property>
  <name>hadoop.tmp.dir</name>
  <value>/opt/hadoop</value>
</property>

</configuration>

4.配置zk

initLimit=
syncLimit=
dataDir=/opt/zookeeper
clientPort=

server.=node1::
server.=node2::
server.=node3::

在配置的dataDir目录下创建myid文件，在node1上配置，稍后还要在node2，node3做相应配置

[hadoop@node1 zookeeper]$ cat /opt/zookeeper/myid

将node1上的zookeeper目录拷贝的node2，及node3上

[root@node1 opt]# scp -r zookeeper/ root@node2:/opt/
[root@node1 opt]# scp -r zookeeper/ root@node3:/opt/

并在node2及node3上分别修改myid文件，node2修改为2，node3修改为3

将node1上的zk的目录拷贝到node2，node3上

[hadoop@node1 software]$ scp -r zookeeper-3.4. hadoop@node2:/home/hadoop/software/
[hadoop@node1 software]$ scp -r zookeeper-3.4. hadoop@node3:/home/hadoop/software/

配置3台节点的ZK_HOME的环境变量：

[root@node1 bin]# vim /etc/profile
export ZK_HOME=/home/hadoop/software/zookeeper-3.4.
export PATH=$PATH:$ZK_HOME/bin

关闭防火墙，并在3台机器上分别启动zk

# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/hadoop/software/zookeeper-3.4./bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

# jps
 Jps
 QuorumPeerMain

5.配置slaves

[root@node1 hadoop]# vim slaves 

node2
node3
node4

6，将配置好的hadoop目录拷贝到其他节点

[hadoop@node1 hadoop]$ scp * hadoop@node2:/home/hadoop/software/hadoop-2.5./etc/hadoop/
[hadoop@node1 hadoop]$ scp * hadoop@node3:/home/hadoop/software/hadoop-2.5./etc/hadoop/
[hadoop@node1 hadoop]$ scp * hadoop@node4:/home/hadoop/software/hadoop-2.5./etc/hadoop/

7. 启动JournalNode

分别在node2，node3，node4机器上上启动JournalNode

[hadoop@node2 sbin]$ ./hadoop-daemon.sh start journalnode
starting journalnode, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-journalnode-node2.out
[hadoop@node2 sbin]$ jps
 JournalNode
 Jps
 QuorumPeerMain
[hadoop@node2 sbin]$ 

8.在其中一台含有Namenode的机器上进行格式化

[hadoop@node1 bin]$ ./hdfs namenode -format

9.将刚才格式化好的元数据文件拷贝到其他的namenode节点上

　　9.1 先启动刚才格式化后的Namenode节点(只启动NameNode)

[hadoop@node1 sbin]$ ./hadoop-daemon.sh start namenode

[hadoop@node1 sbin]$ jps
 QuorumPeerMain
 NameNode
 Jps

　　9.2 再在未格式化的节点上执行以下命令：

[hadoop@node2 bin]$ ./hdfs namenode -bootstrapStandby

　　检查是否有相应目录生成

10.先停止所有的服务，除了ZK

[hadoop@node1 sbin]$ ./stop-dfs.sh
Stopping namenodes on [node1 node2]
node2: no namenode to stop
node1: stopping namenode
node2: no datanode to stop
node4: no datanode to stop
node3: no datanode to stop
Stopping journal nodes [node2 node3 node4]
node3: stopping journalnode
node2: stopping journalnode
node4: stopping journalnode
Stopping ZK Failover Controllers on NN hosts [node1 node2]
node2: no zkfc to stop
node1: no zkfc to stop
[hadoop@node1 sbin]$ 

11.格式化zkfc ，在任意一台有Namenode机器上进行格式化

[hadoop@node1 bin]$ ./hdfs zkfc -formatZK

12.启动hdfs

[hadoop@node1 sbin]$ ./start-dfs.sh
Starting namenodes on [node1 node2]
node1: starting namenode, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-namenode-node1.out
node2: starting namenode, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-namenode-node2.out
node4: starting datanode, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-datanode-node4.out
node2: starting datanode, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-datanode-node2.out
node3: starting datanode, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-datanode-node3.out
Starting journal nodes [node2 node3 node4]
node3: starting journalnode, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-journalnode-node3.out
node2: starting journalnode, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-journalnode-node2.out
node4: starting journalnode, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-journalnode-node4.out
Starting ZK Failover Controllers on NN hosts [node1 node2]
node1: starting zkfc, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-zkfc-node1.out
node2: starting zkfc, logging to /home/hadoop/software/hadoop-2.5./logs/hadoop-hadoop-zkfc-node2.out
[hadoop@node1 sbin]$ jps
 Jps
 QuorumPeerMain
 NameNode
 DFSZKFailoverController
[hadoop@node1 sbin]$ 

通过jps查看需要的节点是否启动成功

[hadoop@node2 opt]$  jps
 JournalNode
 DataNode
 DFSZKFailoverController
 NameNode
 Jps
 QuorumPeerMain
[hadoop@node2 opt]$ 

[hadoop@node3 opt]$ jps
 JournalNode
 QuorumPeerMain
 DataNode
 Jps
[hadoop@node3 opt]$ 

[hadoop@node4 sbin]$ jps
 Jps
 DataNode
 JournalNode
[hadoop@node4 sbin]$ 

13，通过浏览器访问

哪个节点为 Standby，哪个为active是通过CPU竞争机制。　　

测试创建目录和上传文件：

[hadoop@node1 bin]$ ./hdfs dfs -mkdir -p /usr/file
[hadoop@node1 bin]$ ./hdfs dfs -put /home/hadoop/software/jdk-7u75-linux-x64.rpm /usr/file/

到此 Hadoop2.x HA 就搭建完成。