Hadoop 2.7.3 完全分布式维护-动态增加datanode篇

原有环境

http://www.cnblogs.com/ilifeilong/p/7406944.html

IP	host	JDK	linux	hadop	role
172.16.101.55	sht-sgmhadoopnn-01	1.8.0_111	CentOS release 6.5	hadoop-2.7.3	NameNode,SecondaryNameNode,ResourceManager
172.16.101.58	sht-sgmhadoopdn-01	1.8.0_111	CentOS release 6.5	hadoop-2.7.3	DataNode,NodeManager
172.16.101.59	sht-sgmhadoopdn-02	1.8.0_111	CentOS release 6.5	hadoop-2.7.3	DataNode,NodeManager
172.16.101.60	sht-sgmhadoopdn-03	1.8.0_111	CentOS release 6.5	hadoop-2.7.3	DataNode,NodeManager
172.16.101.66	sht-sgmhadoopdn-04	1.8.0_111	CentOS release 6.5	hadoop-2.7.3	DataNode,NodeManager

现计划向集群新增一台datanode,如表格所示

1. 配置系统环境

主机名,ssh互信,环境变量等

2. 修改namenode节点的slave文件,增加新节点信息

$ cat slaves
sht-sgmhadoopdn-
sht-sgmhadoopdn-
sht-sgmhadoopdn-
sht-sgmhadoopdn-

3. 在namenode节点上,将hadoop-2.7.3复制到新节点上,并在新节点上删除data和logs目录中的文件

$ hostname
sht-sgmhadoopnn-
$ rsync -az --progress /usr/local/hadoop-2.7./* hduser@sht-sgmhadoopdn-04:/usr/local/hadoop-2.7.3/

$ hostname
sht-sgmhadoopdn-04
$ rm -rf /usr/local/hadoop-2.7.3/logs/*
$ rm -rf /usr/local/hadoop-2.7.3/data/*

4. 启动新datanode的datanode进程

$ hadoop-daemon.sh start datanode
starting datanode, logging to /usr/local/hadoop-2.7./logs/hadoop-hduser-datanode-sht-sgmhadoopdn-.out
$ jps
 Jps
 DataNode

5. 在namenode查看当前集群情况,确认信节点已经正常加入

5.1 以命令行方式

$ hdfs dfsadmin -report
Configured Capacity:  (282.49 GB)
Present Capacity:  (77.98 GB)
DFS Remaining:  (77.38 GB)
DFS Used:  (618.02 MB)
DFS Used%: 0.77%
Under replicated blocks:
Blocks with corrupt replicas:
Missing blocks:
Missing blocks (with replication factor ): 

-------------------------------------------------
Live datanodes ():

Name: 172.16.101.66: (sht-sgmhadoopdn-)
Hostname: sht-sgmhadoopdn-
Decommission Status : Normal
Configured Capacity:  (70.62 GB)
DFS Used:  ( KB)
Non DFS Used:  (33.13 GB)
DFS Remaining:  (37.49 GB)
DFS Used%: 0.00%
DFS Remaining%: 53.09%
Configured Cache Capacity:  ( B)
Cache Used:  ( B)
Cache Remaining:  ( B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers:
Last contact: Fri Sep  :: CST 

Name: 172.16.101.60: (sht-sgmhadoopdn-)
Hostname: sht-sgmhadoopdn-
Decommission Status : Normal
Configured Capacity:  (70.62 GB)
DFS Used:  ( MB)
Non DFS Used:  (57.48 GB)
DFS Remaining:  (12.95 GB)
DFS Used%: 0.28%
DFS Remaining%: 18.33%
Configured Cache Capacity:  ( B)
Cache Used:  ( B)
Cache Remaining:  ( B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers:
Last contact: Fri Sep  :: CST 

Name: 172.16.101.59: (sht-sgmhadoopdn-)
Hostname: sht-sgmhadoopdn-
Decommission Status : Normal
Configured Capacity:  (70.62 GB)
DFS Used:  ( MB)
Non DFS Used:  (57.80 GB)
DFS Remaining:  (12.63 GB)
DFS Used%: 0.28%
DFS Remaining%: 17.88%
Configured Cache Capacity:  ( B)
Cache Used:  ( B)
Cache Remaining:  ( B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers:
Last contact: Fri Sep  :: CST 

Name: 172.16.101.58: (sht-sgmhadoopdn-)
Hostname: sht-sgmhadoopdn-
Decommission Status : Normal
Configured Capacity:  (70.62 GB)
DFS Used:  ( MB)
Non DFS Used:  (56.11 GB)
DFS Remaining:  (14.31 GB)
DFS Used%: 0.28%
DFS Remaining%: 20.26%
Configured Cache Capacity:  ( B)
Cache Used:  ( B)
Cache Remaining:  ( B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers:
Last contact: Fri Sep  :: CST 

5.2 以web方式

6. 在namenoe上设置 hdfs 的负载均衡

$ hdfs dfsadmin -setBalancerBandwidth
Balancer bandwidth is set to
$ start-balancer.sh -threshold
starting balancer, logging to /usr/local/hadoop-2.7./logs/hadoop-hduser-balancer-sht-sgmhadoopnn-.out

7. 查看hdfs负载信息(有时候节点数据量较小,看出来数据量变化,可以上传大文件测试)

8. 启动新节点的nodemanager进程

$ yarn-daemon.sh start nodemanager
starting nodemanager, logging to /usr/local/hadoop-2.7./logs/yarn-hduser-nodemanager-sht-sgmhadoopdn-.out
$ jps
 NodeManager
 Jps
 DataNode