hadoop 管理命令dfsadmin

dfsadmin 命令用于管理HDFS集群，这些命令常用于管理员。

1. （Safemode）安全模式

动作 命令
把集群切换到安全模式 bin/hdfs dfsadmin -safemode [enter/get/leave]
数据节点状态列表 bin/hadoop dfsadmin -report
添加或删除数据节点 bin/hadoop dfsadmin -refreshNodes
打印网络拓扑  bin/hadoop dfsadmin -printTopology
官当网站 http://hadoop.apache.org/docs/r2.8.3/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#dfsadmin
  1.  
    [hadoop@master bin]$ ./hdfs dfsadmin -safemode enter #进入安全模式
  2.  
    Safe mode is ON
  3.  
     
  4.  
    [hadoop@master bin]$ ./hdfs dfsadmin -safemode get #获取当前状态
  5.  
    Safe mode is ON
  6.  
     
  7.  
    [hadoop@master bin]$ ./hdfs dfsadmin -safemode leave #离开safemode状态
  8.  
    Safe mode is OFF
  9.  
    [hadoop@master bin]$ 
  10.  
     
  11.  
     

安全模式：On startup, the NameNode enters a special state called Safemode. Replication of data blocks does not occur when the NameNode is in the Safemode state. The NameNode receives Heartbeat and Blockreport messages from the DataNodes. A Blockreport contains the list of data blocks that a DataNode is hosting. Each block has a specified minimum number of replicas. A block is considered safely replicated when the minimum number of replicas of that data block has checked in with the NameNode. After a configurable percentage of safely replicated data blocks checks in with the NameNode (plus an additional 30 seconds), the NameNode exits the Safemode state. It then determines the list of data blocks (if any) that still have fewer than the specified number of replicas. The NameNode then replicates these blocks to other DataNodes.

hadoop启动时，会处于一种特殊的状态称做安全模式，这种模式下，数据块的复制是不能发生的，主节点会收到各数据节点的心跳(Heartbeat）和块报告（Blockreport）信息。块报告信息包含该数据节点包含的块信息，每一个数据块都有一个最小被副本数，当一个块最小副本数与NN记录的相符合时就被认为是安全的复制，在配置一个安全副本百分数与NN相符后（再加30s）（意思就是副本数*N%后与NN记录的相符就认为是安全，N可配置），NN就退出Safemode状态。之后（safemode 时是不能发生数据复制的）如果列表中仍然有少数的副本数比已备份少，NN将会把这些块复制到其他数据节点。

根据上面说明：

1.Safemode 主要是校验数据节点的块信息。

2.safemode 不能发生块复制（Replication ）。

3.hadoop 的维护工作是在模式下进行的。

Safemode状态时创建目录报错：

Cannot create directory /hxw. Name node is in safe mode. It was turned on manually. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.

2.集群信息状态报告

以及集群资源占用情况，以及各数据节点信息。

  1.  
    [hadoop@master logs]$ hadoop dfsadmin -report
  2.  
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
  3.  
    Instead use the hdfs command for it.
  4.  
     
  5.  
     
  6.  
    Configured Capacity: 37492883456 (34.92 GB)
  7.  
    Present Capacity: 22908968960 (21.34 GB)
  8.  
    DFS Remaining: 21126250496 (19.68 GB)
  9.  
    DFS Used: 1782718464 (1.66 GB)
  10.  
    DFS Used%: 7.78%
  11.  
    Under replicated blocks: 18
  12.  
    Blocks with corrupt replicas: 0
  13.  
    Missing blocks: 0
  14.  
    Missing blocks (with replication factor 1): 0
  15.  
    Pending deletion blocks: 0
  16.  
     
  17.  
     
  18.  
    -------------------------------------------------
  19.  
    Live datanodes (2):
  20.  
     
  21.  
     
  22.  
    Name: 10.0.1.226:50010 (slave-2)
  23.  
    Hostname: slave-2
  24.  
    Decommission Status : Normal
  25.  
    Configured Capacity: 18746441728 (17.46 GB)
  26.  
    DFS Used: 891359232 (850.07 MB)
  27.  
    Non DFS Used: 7806763008 (7.27 GB)
  28.  
    DFS Remaining: 10048319488 (9.36 GB)
  29.  
    DFS Used%: 4.75%
  30.  
    DFS Remaining%: 53.60%
  31.  
    Configured Cache Capacity: 0 (0 B)
  32.  
    Cache Used: 0 (0 B)
  33.  
    Cache Remaining: 0 (0 B)
  34.  
    Cache Used%: 100.00%
  35.  
    Cache Remaining%: 0.00%
  36.  
    Xceivers: 1
  37.  
    Last contact: Wed Jan 17 17:09:23 CST 2018
  38.  
     
  39.  
     
  40.  
     
  41.  
     
  42.  
    Name: 10.0.1.227:50010 (slave-1)
  43.  
    Hostname: slave-1
  44.  
    Decommission Status : Normal
  45.  
    Configured Capacity: 18746441728 (17.46 GB)
  46.  
    DFS Used: 891359232 (850.07 MB)
  47.  
    Non DFS Used: 6777151488 (6.31 GB)
  48.  
    DFS Remaining: 11077931008 (10.32 GB)
  49.  
    DFS Used%: 4.75%
  50.  
    DFS Remaining%: 59.09%
  51.  
    Configured Cache Capacity: 0 (0 B)
  52.  
    Cache Used: 0 (0 B)
  53.  
    Cache Remaining: 0 (0 B)
  54.  
    Cache Used%: 100.00%
  55.  
    Cache Remaining%: 0.00%
  56.  
    Xceivers: 1
  57.  
    Last contact: Wed Jan 17 17:09:24 CST 2018

3.节点刷新

当集群有新增或删除节点时使用。

  1.  
    [hadoop@master bin]$ ./hdfs dfsadmin -refreshNodes
  2.  
    Refresh nodes successful
  3.  
     
  4.  
    [hadoop@slave-1 sbin]$ ./hadoop-daemon.sh  stop datanode
  5.  
    stopping datanode
  6.  
     
  7.  
    [hadoop@master bin]$ hadoop dfsadmin -report
  8.  
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
  9.  
    Instead use the hdfs command for it.
  10.  
     
  11.  
     
  12.  
    Configured Capacity: 37492883456 (34.92 GB)
  13.  
    Present Capacity: 22914383872 (21.34 GB)
  14.  
    DFS Remaining: 21131665408 (19.68 GB)
  15.  
    DFS Used: 1782718464 (1.66 GB)
  16.  
    DFS Used%: 7.78%
  17.  
    Under replicated blocks: 18
  18.  
    Blocks with corrupt replicas: 0
  19.  
    Missing blocks: 0
  20.  
    Missing blocks (with replication factor 1): 0
  21.  
    Pending deletion blocks: 0
  22.  
     
  23.  
     
  24.  
    -------------------------------------------------
  25.  
    Live datanodes (2):
  26.  
     
  27.  
     
  28.  
    Name: 10.0.1.226:50010 (slave-2)
  29.  
    Hostname: slave-2
  30.  
    Decommission Status : Normal
  31.  
    Configured Capacity: 18746441728 (17.46 GB)
  32.  
    DFS Used: 891359232 (850.07 MB)
  33.  
    Non DFS Used: 7801290752 (7.27 GB)
  34.  
    DFS Remaining: 10053791744 (9.36 GB)
  35.  
    DFS Used%: 4.75%
  36.  
    DFS Remaining%: 53.63%
  37.  
    Configured Cache Capacity: 0 (0 B)
  38.  
    Cache Used: 0 (0 B)
  39.  
    Cache Remaining: 0 (0 B)
  40.  
    Cache Used%: 100.00%
  41.  
    Cache Remaining%: 0.00%
  42.  
    Xceivers: 1
  43.  
    Last contact: Wed Jan 17 18:16:06 CST 2018
  44.  
     
  45.  
     
  46.  
     
  47.  
     
  48.  
    Name: 10.0.1.227:50010 (slave-1)
  49.  
    Hostname: slave-1
  50.  
    Decommission Status : Normal
  51.  
    Configured Capacity: 18746441728 (17.46 GB)
  52.  
    DFS Used: 891359232 (850.07 MB)
  53.  
    Non DFS Used: 6777208832 (6.31 GB)
  54.  
    DFS Remaining: 11077873664 (10.32 GB)
  55.  
    DFS Used%: 4.75%
  56.  
    DFS Remaining%: 59.09%
  57.  
    Configured Cache Capacity: 0 (0 B)
  58.  
    Cache Used: 0 (0 B)
  59.  
    Cache Remaining: 0 (0 B)
  60.  
    Cache Used%: 100.00%
  61.  
    Cache Remaining%: 0.00%
  62.  
    Xceivers: 1
  63.  
    Last contact: Wed Jan 17 18:13:43 CST 2018
  64.  
     
  65.  
    [hadoop@master bin]$ ./hdfs dfsadmin  -refreshNodes
  66.  
    Refresh nodes successful
  67.  
     
  68.  
     
  69.  
    [hadoop@master bin]$ ./hdfs dfsadmin  -refreshNodes
  70.  
    Refresh nodes successful
  71.  
    [hadoop@master bin]$ hadoop dfsadmin -report       
  72.  
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
  73.  
    Instead use the hdfs command for it.
  74.  
     
  75.  
     
  76.  
    Configured Capacity: 37492883456 (34.92 GB)
  77.  
    Present Capacity: 22914379776 (21.34 GB)
  78.  
    DFS Remaining: 21131661312 (19.68 GB)
  79.  
    DFS Used: 1782718464 (1.66 GB)
  80.  
    DFS Used%: 7.78%
  81.  
    Under replicated blocks: 18
  82.  
    Blocks with corrupt replicas: 0
  83.  
    Missing blocks: 0
  84.  
    Missing blocks (with replication factor 1): 0
  85.  
    Pending deletion blocks: 0
  86.  
     
  87.  
     
  88.  
    -------------------------------------------------
  89.  
    Live datanodes (2):
  90.  
     
  91.  
     
  92.  
    Name: 10.0.1.226:50010 (slave-2)
  93.  
    Hostname: slave-2
  94.  
    Decommission Status : Normal
  95.  
    Configured Capacity: 18746441728 (17.46 GB)
  96.  
    DFS Used: 891359232 (850.07 MB)
  97.  
    Non DFS Used: 7801294848 (7.27 GB)
  98.  
    DFS Remaining: 10053787648 (9.36 GB)
  99.  
    DFS Used%: 4.75%
  100.  
    DFS Remaining%: 53.63%
  101.  
    Configured Cache Capacity: 0 (0 B)
  102.  
    Cache Used: 0 (0 B)
  103.  
    Cache Remaining: 0 (0 B)
  104.  
    Cache Used%: 100.00%
  105.  
    Cache Remaining%: 0.00%
  106.  
    Xceivers: 1
  107.  
    Last contact: Wed Jan 17 18:18:54 CST 2018
  108.  
     
  109.  
     
  110.  
     
  111.  
     
  112.  
    Name: 10.0.1.227:50010 (slave-1)
  113.  
    Hostname: slave-1
  114.  
    Decommission Status : Normal
  115.  
    Configured Capacity: 18746441728 (17.46 GB)
  116.  
    DFS Used: 891359232 (850.07 MB)
  117.  
    Non DFS Used: 6777208832 (6.31 GB)
  118.  
    DFS Remaining: 11077873664 (10.32 GB)
  119.  
    DFS Used%: 4.75%
  120.  
    DFS Remaining%: 59.09%
  121.  
    Configured Cache Capacity: 0 (0 B)
  122.  
    Cache Used: 0 (0 B)
  123.  
    Cache Remaining: 0 (0 B)
  124.  
    Cache Used%: 100.00%
  125.  
    Cache Remaining%: 0.00%
  126.  
    Xceivers: 1
  127.  
    Last contact: Wed Jan 17 18:13:43 CST 2018

说明：在停止某个数据节点后，刷新节点信息仍然能看到该节点信息，状态noraml 状态，界面上看到last contact 时间是560+s。

在Namenode 的配置文件slaves 中删除该节点，然后重新刷新节点信息，则后台显示：

  1.  
    [hadoop@master bin]$ ./hdfs dfsadmin -refreshNodes
  2.  
    Refresh nodes successful
  3.  
    [hadoop@master bin]$ hadoop dfsadmin -report
  4.  
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
  5.  
    Instead use the hdfs command for it.
  6.  
     
  7.  
    Configured Capacity: 18746441728 (17.46 GB)
  8.  
    Present Capacity: 10945093632 (10.19 GB)
  9.  
    DFS Remaining: 10053734400 (9.36 GB)
  10.  
    DFS Used: 891359232 (850.07 MB)
  11.  
    DFS Used%: 8.14%
  12.  
    Under replicated blocks: 161
  13.  
    Blocks with corrupt replicas: 0
  14.  
    Missing blocks: 0
  15.  
    Missing blocks (with replication factor 1): 0
  16.  
    Pending deletion blocks: 0
  17.  
     
  18.  
    -------------------------------------------------
  19.  
    Live datanodes (1):
  20.  
     
  21.  
    Name: 10.0.1.226:50010 (slave-2)
  22.  
    Hostname: slave-2
  23.  
    Decommission Status : Normal
  24.  
    Configured Capacity: 18746441728 (17.46 GB)
  25.  
    DFS Used: 891359232 (850.07 MB)
  26.  
    Non DFS Used: 7801348096 (7.27 GB)
  27.  
    DFS Remaining: 10053734400 (9.36 GB)
  28.  
    DFS Used%: 4.75%
  29.  
    DFS Remaining%: 53.63%
  30.  
    Configured Cache Capacity: 0 (0 B)
  31.  
    Cache Used: 0 (0 B)
  32.  
    Cache Remaining: 0 (0 B)
  33.  
    Cache Used%: 100.00%
  34.  
    Cache Remaining%: 0.00%
  35.  
    Xceivers: 1
  36.  
    Last contact: Wed Jan 17 18:26:36 CST 2018
  37.  
     
  38.  
     
  39.  
    Dead datanodes (1):
  40.  
     
  41.  
    Name: 10.0.1.227:50010 (slave-1)
  42.  
    Hostname: slave-1
  43.  
    Decommission Status : Normal
  44.  
    Configured Capacity: 0 (0 B)
  45.  
    DFS Used: 0 (0 B)
  46.  
    Non DFS Used: 6777208832 (6.31 GB)
  47.  
    DFS Remaining: 0 (0 B)
  48.  
    DFS Used%: 100.00%
  49.  
    DFS Remaining%: 0.00%
  50.  
    Configured Cache Capacity: 0 (0 B)
  51.  
    Cache Used: 0 (0 B)
  52.  
    Cache Remaining: 0 (0 B)
  53.  
    Cache Used%: 100.00%
  54.  
    Cache Remaining%: 0.00%
  55.  
    Xceivers: 0
  56.  
    Last contact: Wed Jan 17 18:13:43 CST 2018
  57.  
     

4.网络拓扑

  1.  
    [hadoop@master ~]$ hadoop dfsadmin -printTopology
  2.  
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
  3.  
    Instead use the hdfs command for it.
  4.  
     
  5.  
    Rack: /default-rack
  6.  
    10.0.1.226:50010 (slave-2)
  7.  
    10.0.1.227:50010 (slave-1)

总结：

  1.  
    [hadoop@master bin]$ ./hdfs dfsadmin -safemode enter #进入Safemode模式
  2.  
    [hadoop@master bin]$ ./hdfs dfsadmin -safemode get #获取当前运行模式
  3.  
    [hadoop@master bin]$ ./hdfs dfsadmin -safemode leave #退出Safemode模式
  4.  
    [hadoop@master bin]$ hadoop dfsadmin -report #当前hadoop集群状态信息
  5.  
    [hadoop@master bin]$ ./hdfs dfsadmin -refreshNodes #新增删除节点更新集群信息
  6.  
    [hadoop@master sbin]$ ./hadoop-daemon.sh  stop datanode #停止单个数据节点
  7.  
    [hadoop@master ~]$ hadoop dfsadmin -printTopology #打印集群网络拓扑

hadoop 管理命令dfsadmin的更多相关文章

  1. hadoop管理命令

    -help  功能:输出这个命令参数手册 -ls   功能:显示目录信息 示例: hadoop fs -ls hdfs://hadoop-server01:9000/ 备注:这些参数中,所有的hdfs ...

  2. hadoop管理

    hadoop管理工具: 一,dfsadmin工具 dfsadmin 工具用途比较广,既可以查找HDFS状态信息,又可以在HDFS上执行管理操作,以hdfs dfsadmin形式调用,且需要超级用户权限 ...

  3. Hadoop进阶命令使用介绍

    hadoop生产环境的使用是很复杂的,有些公司是CDH或者Ambari等管理工具运维的,有些是原生的集群俗称裸奔,前者优缺点是运维,查看监控等容易,对于底层理解不友好:裸奔集群反之,裸奔集群的很多东西 ...

  4. 【Hadoop篇】--Hadoop常用命令总结

    一.前述 分享一篇hadoop的常用命令的总结,将常用的Hadoop命令总结如下. 二.具体 1.启动hadoop所有进程start-all.sh等价于start-dfs.sh + start-yar ...

  5. Hadoop常用命令总结

    一.前述 分享一篇hadoop的常用命令的总结,将常用的Hadoop命令总结如下. 二.具体 1.启动hadoop所有进程start-all.sh等价于start-dfs.sh + start-yar ...

  6. 17 RAID与mdadm管理命令

    在"14 磁盘及文件系统管理详解"中,我们详细介绍了磁盘的工作原理,但是,有一点我们一定要明白,作为现在存储数据的主要设备,机械磁盘早就是上个世纪的产品,而它的读写速度与内存.CP ...

  7. hadoop基础----hadoop实战(七)-----hadoop管理工具---使用Cloudera Manager安装Hadoop---Cloudera Manager和CDH5.8离线安装

    hadoop基础----hadoop实战(六)-----hadoop管理工具---Cloudera Manager---CDH介绍 简介 我们在上篇文章中已经了解了CDH,为了后续的学习,我们本章就来 ...

  8. 大数据之路week06--day07（Hadoop常用命令）

    一.前述 分享一篇hadoop的常用命令的总结,将常用的Hadoop命令总结如下. 二.具体 1.启动hadoop所有进程start-all.sh等价于start-dfs.sh + start-yar ...

  9. Hadoop Shell命令大全

    hadoop支持命令行操作HDFS文件系统,并且支持shell-like命令与HDFS文件系统交互,对于大多数程序猿/媛来说,shell-like命令行操作都是比较熟悉的,其实这也是Hadoop的极大 ...

随机推荐

  1. centos7下如何隐藏nginx的版本号

    我们在访问nginx的时候会暴露nginx的版本号,如何将这些版本号隐藏呢? 其实隐藏版本号非常简单 在nginx的配置文件中添加一个server——tokens  off:参数就可以了,下面进行操作 ...

  2. UVA11400-Lighting System Design（动态规划基础）

    Problem UVA11400-Lighting System Design Accept: 654  Submit: 4654Time Limit: 3000 mSec Problem Descr ...

  3. 转://Oracle 复合压缩索引场景及性能对比

    摘要:今天为什么提到这个话题,出于一个偶然,一个同事在优化新开发的系统时向我请教如何添加复合压缩索引的问题.我总结了一下,问题有三. 第一:需不需要压缩 第二:对第几列压缩 第三:性能对比,选出最优 ...

  4. P2089 烤鸡（搜索简单题）

    题意:就是x分别是1到3的未知数,求x1+x2+x3.....+x10=n的方案数和输出每种方案.每种方案还必须按字典序输出 思路:就是简单的构建搜索树+约束条件啊,其实数据范围一点都不大,所以,我第 ...

  5. leetcode 74. Search a 2D Matrix 、240. Search a 2D Matrix II

    74. Search a 2D Matrix 整个二维数组是有序排列的,可以把这个想象成一个有序的一维数组,然后用二分找中间值就好了. 这个时候需要将全部的长度转换为相应的坐标,/col获得x坐标,% ...

  6. C学习笔记-一些知识

    memset可以方便的清空一个结构类型的变量或数组. 如: struct sample_struct { ]; int iSeq; int iType; }; 对于变量 struct sample_s ...

  7. 【原创】分布式之redis的三大衍生数据结构

    引言 说起redis的数据结构,大家可能对五大基础数据类型比较熟悉:String,Hash,List,Set,Sorted Set.那么除此之外,还有三大衍生数据结构,大家平时是很少接触的,即:bit ...

  8. 从统计局采集最新的省市区镇数据，用js在浏览器中运行 V2

    本文描述的是对国家统计局于2019-01-31发布的<2018年统计用区划代码和城乡划分代码(截止2018年10月31日)>的采集. 相对于用于和采集2016版.2017版的js代码做了比 ...

  9. RabbitMQ 高可用之镜像队列

    如果RabbitMQ集群只有一个broker节点,那么该节点的失效将导致整个服务临时性的不可用,并且可能会导致message的丢失(尤其是在非持久化message存储于非持久化queue中的时候).可 ...

  10. Jmeter(三十六)_运行过程中改变负载

    顾名思义,jmeter在做性能测试时,可以在不停止脚本的情况下修改负载压力,达到期望的测试效果.我们将通过Constant Throughput Timer(吞吐量计时器)和Beanshell服务器来 ...