关键字：hdfs fsck、block、locations

在HDFS中，提供了fsck命令，用于检查HDFS上文件和目录的健康状态、获取文件的block信息和位置信息等。

fsck命令必须由HDFS超级用户来执行，普通用户无权限。

  1. [hadoop@dev ~]$ hdfs fsck
  2. Usage: DFSck [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]]
  3. start checking from this path
  4. -move move corrupted files to /lost+found
  5. -delete delete corrupted files
  6. -files print out files being checked
  7. -openforwrite print out files opened for write
  8. -includeSnapshots include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it
  9. -list-corruptfileblocks print out list of missing blocks and files they belong to
  10. -blocks print out block report
  11. -locations print out locations for every block
  12. -racks print out network topology for data-node locations

下面介绍每一个选项的含义及用法。

查看文件中损坏的块（-list-corruptfileblocks）

  1. [hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -list-corruptfileblocks
  2. The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/' has 0 CORRUPT files

将损坏的文件移动至/lost+found目录（-move）

  1. [hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 -move
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 at Thu Aug 13 09:36:35 CST 2015
  3. .Status: HEALTHY
  4. Total size: 13497058 B
  5. Total dirs: 0
  6. Total files: 1
  7. Total symlinks: 0
  8. Total blocks (validated): 1 (avg. block size 13497058 B)
  9. Minimally replicated blocks: 1 (100.0 %)
  10. Over-replicated blocks: 0 (0.0 %)
  11. Under-replicated blocks: 0 (0.0 %)
  12. Mis-replicated blocks: 0 (0.0 %)
  13. Default replication factor: 2
  14. Average block replication: 2.0
  15. Corrupt blocks: 0
  16. Missing replicas: 0 (0.0 %)
  17. Number of data-nodes: 15
  18. Number of racks: 1
  19. FSCK ended at Thu Aug 13 09:36:35 CST 2015 in 1 milliseconds
  20. The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168' is HEALTHY

删除损坏的文件（-delete）

  1. [hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 -delete
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 at Thu Aug 13 09:37:58 CST 2015
  3. .Status: HEALTHY
  4. Total size: 13497058 B
  5. Total dirs: 0
  6. Total files: 1
  7. Total symlinks: 0
  8. Total blocks (validated): 1 (avg. block size 13497058 B)
  9. Minimally replicated blocks: 1 (100.0 %)
  10. Over-replicated blocks: 0 (0.0 %)
  11. Under-replicated blocks: 0 (0.0 %)
  12. Mis-replicated blocks: 0 (0.0 %)
  13. Default replication factor: 2
  14. Average block replication: 2.0
  15. Corrupt blocks: 0
  16. Missing replicas: 0 (0.0 %)
  17. Number of data-nodes: 15
  18. Number of racks: 1
  19. FSCK ended at Thu Aug 13 09:37:58 CST 2015 in 1 milliseconds
  20. The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168' is HEALTHY

检查并列出所有文件状态（-files）

  1. [hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -files
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ at Thu Aug 13 09:39:38 CST 2015
  3. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ dir
  4. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/_SUCCESS 0 bytes, 0 block(s): OK
  5. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00000 13583807 bytes, 1 block(s): OK
  6. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00001 13577427 bytes, 1 block(s): OK
  7. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00002 13588601 bytes, 1 block(s): OK
  8. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00003 13479213 bytes, 1 block(s): OK
  9. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00004 13497012 bytes, 1 block(s): OK
  10. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00005 13557451 bytes, 1 block(s): OK
  11. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00006 13580267 bytes, 1 block(s): OK
  12. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00007 13486035 bytes, 1 block(s): OK
  13. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00008 13481498 bytes, 1 block(s): OK
  14. ...

检查并打印正在被打开执行写操作的文件（-openforwrite）

  1. [hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -openforwrite
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ at Thu Aug 13 09:41:28 CST 2015
  3. ....................................................................................................
  4. ....................................................................................................
  5. .Status: HEALTHY
  6. Total size: 2704782548 B
  7. Total dirs: 1
  8. Total files: 201
  9. Total symlinks: 0
  10. Total blocks (validated): 200 (avg. block size 13523912 B)
  11. Minimally replicated blocks: 200 (100.0 %)
  12. Over-replicated blocks: 0 (0.0 %)
  13. Under-replicated blocks: 0 (0.0 %)
  14. Mis-replicated blocks: 0 (0.0 %)
  15. Default replication factor: 2
  16. Average block replication: 2.0
  17. Corrupt blocks: 0
  18. Missing replicas: 0 (0.0 %)
  19. Number of data-nodes: 15
  20. Number of racks: 1
  21. FSCK ended at Thu Aug 13 09:41:28 CST 2015 in 10 milliseconds
  22. The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/' is HEALTHY

打印文件的Block报告（-blocks）

需要和-files一起使用。

  1. [hadoop@dev ~]$ hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015
  3. /logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): OK
  4. 0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2
  5. 1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2
  6. 2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2
  7. 3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2
  8. 4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2
  9. 5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2
  10. 6. BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728 repl=2
  11. 7. BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728 repl=2
  12. 8. BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728 repl=2
  13. 9. BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728 repl=2
  14. 10. BP-1034052771-172.16.212.130-1405595752491:blk_1075893003_2152402 len=134217728 repl=2
  15. 11. BP-1034052771-172.16.212.130-1405595752491:blk_1075893004_2152403 len=134217728 repl=2
  16. 12. BP-1034052771-172.16.212.130-1405595752491:blk_1075893005_2152404 len=134217728 repl=2
  17. 13. BP-1034052771-172.16.212.130-1405595752491:blk_1075893006_2152405 len=134217728 repl=2
  18. 14. BP-1034052771-172.16.212.130-1405595752491:blk_1075893007_2152406 len=134217728 repl=2
  19. ...

其中，/logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): 表示文件的总大小和block数；

0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2
1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2
2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2

前面的0. 1. 2.代表该文件的block索引，56的文件块，就从0-55;

BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381表示block id；

len=134217728 表示该文件块大小；

repl=2 表示该文件块副本数；

打印文件块的位置信息（-locations）

需要和-files -blocks一起使用。

  1. [hadoop@dev ~]$ hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks -locations
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015
  3. /logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): OK
  4. 0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2 [172.16.212.139:50010, 172.16.212.135:50010]
  5. 1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2 [172.16.212.140:50010, 172.16.212.133:50010]
  6. 2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2 [172.16.212.136:50010, 172.16.212.141:50010]
  7. 3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.135:50010]
  8. 4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2 [172.16.212.142:50010, 172.16.212.139:50010]
  9. 5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.139:50010]
  10. 6. BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728 repl=2 [172.16.212.141:50010, 172.16.212.135:50010]
  11. 7. BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728 repl=2 [172.16.212.144:50010, 172.16.212.142:50010]
  12. 8. BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.138:50010]
  13. 9. BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728 repl=2 [172.16.212.140:50010, 172.16.212.134:50010]
  14. ...

和打印出的文件块信息相比，多了一个文件块的位置信息：[172.16.212.139:50010, 172.16.212.135:50010]

打印文件块位置所在的机架信息（-racks）

  1. [hadoop@dev ~]$ hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks -locations -racks
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015
  3. /logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): OK
  4. 0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2 [/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010]
  5. 1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2 [/default-rack/172.16.212.140:50010, /default-rack/172.16.212.133:50010]
  6. 2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2 [/default-rack/172.16.212.136:50010, /default-rack/172.16.212.141:50010]
  7. 3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2 [/default-rack/172.16.212.133:50010, /default-rack/172.16.212.135:50010]
  8. 4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2 [/default-rack/172.16.212.142:50010, /default-rack/172.16.212.139:50010]
  9. 5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2 [/default-rack/172.16.212.133:50010, /default-rack/172.16.212.139:50010]
  10. ...

和前面打印出的信息相比，多了机架信息：[/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010]

hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息(Locations)的更多相关文章

  1. Linux下Qt应用程序的发布（使用LDD命令查看所有依赖的库文件）

    最近一直在学习Qt,用Qt写了一个程序,但是不知道怎么发布,网上说的都是在windows下怎么发布Qt应用程序,但是,在windows下Qt应用程序依赖的库文件与linux下的名字不同.于是,我就想到 ...

  2. 通过命令查看jboss的错误日志文件

    链接jboss: ssh ip -l jboss

  3. fsck获取文件的block信息和位置信息

    原文链接:lxw的大数据田地 » hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息(Locations) 关键字:hdfs fsck.block.locations 在H ...

  4. HDFS的命令行操作

    1.namenode –format:格式化DFS 文件系统 2.secondaryNameNode: 运行DFS的 SecondaryNameNode 进程 hadoop secondaryname ...

  5. 【原创】大数据基础之HDFS（1）HDFS新创建文件如何分配Datanode

    HDFS中的File由Block组成,一个File包含一个或多个Block,当创建File时会创建一个Block,然后根据配置的副本数量(默认是3)申请3个Datanode来存放这个Block: 通过 ...

  6. Mysql: mysqlbinlog命令查看日志文件

    想查看mysql的binlog文件,但是裸的binlog文件是无法直视的,mysqlbinlog这个工具是用来查看binlog文件内容的(使用方式man mysqlbinlog查看),但是使用mysq ...

  7. hadoop命令fsck命令

    在HDFS中,提供了fsck命令,用于检查HDFS上文件和目录的健康状态.获取文件的block块信息和位置信息等. 具体命令介绍: -move: 移动损坏的文件到/lost+found目录下 -del ...

  8. （第3篇）HDFS是什么？HDFS适合做什么？我们应该怎样操作HDFS系统？

    摘要: 这篇文章会详细介绍HDFS是什么,HDFS的作用,适合和不适合的场景,我们该如何操作HDFS?   HDFS文件系统 Hadoop 附带了一个名为 HDFS(Hadoop分布式文件系统)的分布 ...

  9. 用adb pull命令从android系统中读取文件失败的原因及解决办法

    问题:使用adb pull命令从android系统中读取文件失败.显示:Permission denied   原因:是由于文件权限原因引起.       使用ls -l命令查看android系统中的 ...

随机推荐

  1. js中const,var,let区别（转载）

    js中const,var,let区别 来源:https://www.cnblogs.com/zzsdream/p/6372729.html 今天第一次遇到const定义的变量,查阅了相关资料整理了这篇 ...

  2. [PHP]代码执行和生命周期

    PHP代码的执行:1.和大部分程序一样,接收数据,处理数据,输出结果2.编写的代码就是输入的数据,php内核进行处理,返回相应的输出3.php作为业务程序和编译语言的区别就是,php多了一步把用户代码 ...

  3. [angularjs] angularjs系列笔记（三）模型

    ng-model指令 ng-model可以将输入域的值与AngularJs的变量绑定 双向绑定 当修改输入域的值时候,AngularJs属性的值也将修改 <div ng-app="Ho ...

  4. 数据结构——基于java的链表实现（真正理解链表这种数据结构）

    原创不易,如需转载,请注明出处https://www.cnblogs.com/baixianlong/p/10759599.html,否则将追究法律责任!!! 一.链表介绍 1.什么是链表? 链表是一 ...

  5. SpringBoot的Autowierd失败

    通常是以下几种可能: 1.没有加@Service注解,或者是这个bean没有放在标注了@Configuration这个注解的类下. 2.SpringBoot启动类没有开启扫描 @ComponentSc ...

  6. JDK的动态代理-----为接口进行代理

    JDK的动态代理是必须掌握的,动态代理的好处就不用我多说了吧 :) 小弟最近在研究mybatis的源码实现,就开始了解mybatis的Mapper代理机制,为什么接口不用实现类也能代理? 好了,废话不 ...

  7. mysql不能保存中文

    进入mysql文件夹,新建(修改) my.ini 文件, 修改编码 内容如下: [mysql] default-character-set=utf8

  8. CodeMirror 使用

    codemirror是一款在线代码编辑器,官网:https://codemirror.net/doc/manual.html#overview 使用说明 第一步 安装: npm install cod ...

  9. struts2框架-----Action

    控制器Action Action对象是struts2框架的核心,每个URL映射到特定的Action,其提供处理来自用户的请求所需要的处理逻辑.Action有两个重要的功能,即将数据从请求传递到视图和协 ...

  10. MySQL5.7: datetime

    -- 当前日期时间 select now(); select now(3);-- 保留3位毫秒数 SELECT NOW(6); -- 保留6位毫秒数 -- 当前日期和时间 至秒 select curr ...