HDFS重启集群导致数据损坏，使用fsck命令修复过程
HDFS重启集群导致数据损坏，使用fsck命令修复过程
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
我们先看一组输出
[root@flume112 ~]# hdfs fsck /user/kuaikan/report_new////
Connecting to namenode via http://calculation111.aggrx:50070
FSCK started by root (auth:SIMPLE) from /10.1.2.112 for path /user/kuaikan/report_new//// at Sat May :: CST
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................Status: HEALTHY
Total size: B (Total open files size: B)
Total dirs:
Total files:
Total symlinks: (Files currently being written: )
Total blocks (validated): (avg. block size B) (Total open file blocks (not validated): ) #注意，这个26没有进行验证，说明有26个存在问题！
Minimally replicated blocks: (100.0 %)
Over-replicated blocks: (0.0 %)
Under-replicated blocks: (0.0 %)
Mis-replicated blocks: (0.0 %)
Default replication factor:
Average block replication: 3.0
Corrupt blocks:
Missing replicas: (0.0 %)
Number of data-nodes:
Number of racks:
FSCK ended at Sat May :: CST in milliseconds The filesystem under path '/user/kuaikan/report_new/2019/05/23/' is HEALTHY
[root@flume112 ~]#
官方关于fsck命令有对应的解释,我们查看更详细的信息，官方链接：https://hadoop.apache.org/docs/r2.9.1/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#fsck。
[root@flume112 ~]# hdfs fsck /user/kuaikan/report_new//// -openforwrite
Connecting to namenode via http://calculation111.aggrx:50070
FSCK started by root (auth:SIMPLE) from /10.1.2.112 for path /user/kuaikan/report_new//// at Sat May :: CST
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
................................................/user/kuaikan/report_new////report.log.----45.10---160.1558554302405.txt bytes, block(s), OPENFORWRITE: ../user/kuaikan/report_new////report.log.----45.10---248.1558554301696.txt bytes, block(s), OPENFORWRITE: ....../user/kuaikan/report_new////report.log.----45.10---71.1558554301576.txt bytes, block(s), OPENFORWRITE: .........................................
/user/kuaikan/report_new////report.log.----55.10---208.1558554901562.txt bytes, block(s), OPENFORWRITE: .../user/kuaikan/report_new////report.log.----55.10---248.1558554901949.txt bytes, block(s), OPENFORWRITE: ...../user/kuaikan/report_new////report.log.----55.10---233.1558554901647.txt bytes, block(s), OPENFORWRITE: .........................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
................................................................/user/kuaikan/report_new////report.log.----30.10---208.1558575001741.txt bytes, block(s), OPENFORWRITE: ...................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
......................./user/kuaikan/report_new////report.log.----55.10---93.1558587301545.txt bytes, block(s), OPENFORWRITE: ........../user/kuaikan/report_new////report.log.----55.10---88.1558587301803.txt bytes, block(s), OPENFORWRITE: .........................../user/kuaikan/report_new////report.log.----01.10---88.1558587709627.txt bytes, block(s), OPENFORWRITE: .....................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
.........................................................................../user/kuaikan/report_new////report.log.----45.10---35.1558601101874.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---40.1558601102383.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---46.1558601101876.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---58.1558601102255.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---93.1558601102225.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---38.1558601101874.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---208.1558601101557.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---160.1558601101782.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---33.1558601102283.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---150.1558601101668.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---248.1558601101427.txt bytes, block(s), OPENFORWRITE: ./user/kuaikan/report_new////report.log.----45.10---226.1558601101904.txt bytes, block(s), OPENFORWRITE: ./user/kuaikan/report_new////report.log.----45.10---96.1558601101582.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---88.1558601101302.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---233.1558601101531.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---71.1558601101720.txt bytes, block(s), OPENFORWRITE: .......
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
..............................................................Status: HEALTHY
Total size: B
Total dirs:
Total files:
Total symlinks:
Total blocks (validated): (avg. block size B)
Minimally replicated blocks: (99.50153 %)
Over-replicated blocks: (0.0 %)
Under-replicated blocks: (0.0 %)
Mis-replicated blocks: (0.0 %)
Default replication factor:
Average block replication: 2.985046
Corrupt blocks:
Missing replicas: (0.0 %)
Number of data-nodes:
Number of racks:
FSCK ended at Sat May :: CST in milliseconds The filesystem under path '/user/kuaikan/report_new/2019/05/23/' is HEALTHY
[root@flume112 ~]#
[root@flume112 ~]# hdfs fsck /user/kuaikan/report_new/2019/05/23/ -openforwrite
找到损坏的文件后，我们手动进行回复，注意，我们输入的必须是一个文件，并不是一个目录哟！
[root@calculation111 ~]# hdfs debug recoverLease -path /user/kuaikan/report_new////report.log.----45.10---71.1558554301576.txt -retries
recoverLease returned false.
Retrying in ms...
Retry #
recoverLease SUCCEEDED on /user/kuaikan/report_new////report.log.----45.10---71.1558554301576.txt
[root@calculation111 ~]#
HDFS重启集群导致数据损坏，使用fsck命令修复过程的更多相关文章
- Hadoop基础-HDFS集群中大数据开发常用的命令总结
Hadoop基础-HDFS集群中大数据开发常用的命令总结 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本盘博客仅仅列出了我们在实际生成环境中常用的hdfs命令,如果想要了解更多, ...
- Hadoop集群-HDFS集群中大数据运维常用的命令总结
Hadoop集群-HDFS集群中大数据运维常用的命令总结 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客会简单涉及到滚动编辑,融合镜像文件,目录的空间配额等运维操作简介.话 ...
- Hadoop（八）Java程序访问HDFS集群中数据块与查看文件系统
前言 我们知道HDFS集群中,所有的文件都是存放在DN的数据块中的.那我们该怎么去查看数据块的相关属性的呢?这就是我今天分享的内容了 一.HDFS中数据块概述 1.1.HDFS集群中数据块存放位置 我 ...
- 本地日志数据实时接入到hadoop集群的数据接入方案
1. 概述 本手册主要介绍了,一个将传统数据接入到Hadoop集群的数据接入方案和实施方法.供数据接入和集群运维人员参考. 1.1. 整体方案 Flume作为日志收集工具,监控一个文件目录或者一个文 ...
- elasticsearch5.0集群大数据量迁移方法及注意事项
当es集群的数据量较小的情况下elasticdump这个工具比较方便,但是当数据量达到一定级别比如上百G的时候,elasticdump速度就很慢了,此时我们可以使用快照的方法进行备份 elasticd ...
- MongoDB DBA 实践5-----复制集集群的数据同步和故障转移
(1)复制集集群的数据同步 1>主节点数据库test,在其中goods集合中加入一个文档. 2>在副节点中查看 注意:SECONDARY是不允许读写的,要使用rs.slaveOk()获得读 ...
- 软件架构自学笔记----分享“去哪儿 Hadoop 集群 Federation 数据拷贝优化”
去哪儿 Hadoop 集群 Federation 数据拷贝优化 背景 去哪儿 Hadoop 集群随着去哪儿网的发展一直在优化改进,基本保证了业务数据存储量和计算量爆发式增长下的存储服务质量.然而,随着 ...
- Redis 切片集群的数据倾斜分析
Redis 中如何应对数据倾斜 什么是数据倾斜 数据量倾斜 bigkey导致倾斜 Slot分配不均衡导致倾斜 Hash Tag导致倾斜 数据访问倾斜 如何发现 Hot Key Hot Key 如何解决 ...
- KingbbaseES V8R6集群维护案例之---集群之间数据迁移
案例说明: 生产环境是集群环境,测试环境是集群,现需要将生产环境的数据迁移到测试集群中运行,本文档详细介绍了从集群环境迁移数据的操作步骤,可以作为生产环境迁移数据的参考. 适用版本: Kingbase ...
随机推荐
- 【JS】AJAX跨域-JSONP解决方案（一）
AJAX跨域介绍 AJAX 跨域访问是用户访问A网站时所产生的对B网站的跨域访问请求均提交到A网站的指定页面 由于安全方面的原因, 客户端js使用xmlhttprequest只能用来向来源网站发送请求 ...
- Windows10不能进入睡眠
问题 有时候暂时不使用计算机(Windows10笔记本),既想省电,又想回来之后快速启动,一般会选择让计算机进入睡眠状态.睡眠状态下,基本上只有内存通电,保存着之前的工作状态,可以快速恢复. 但是最近 ...
- 【知识总结】Polya 定理
我第一次听说 Polya 原理是 NOIP2017 以前,但我觉得太难想着以后再学: NOIP2018 以前我觉得会考这玩意,下定决心学,后来咕了: WC2019 以前我觉得会考这玩意,下定决心学,后 ...
- springcloud学习的坑
一:启动Euerka作为提供者或者消费者时,启动失败报:Process finished with exit code 0 Unregistering application EUREKA-SERVI ...
- Centos7下nginx的安装与配置
说明:软件安装的基础目录路径:/usr/local 所以下载软件的时候切换到此目录下下载直接解压即可 1.安装gcc gcc-c++依赖包 yum install -y gcc gcc-c++ 2.下 ...
- 解决unzip解压中文乱码问题
使用 unzip XXX.zip 方式解压的时候会出现中文乱码 很多人推荐以下方式: 在windows执行命令,可显示字符集数字一般为936: # chcp // 解压时加上-O cp936,xxx为 ...
- AS3.0 m3u8文件视频播放器
AS3.0 m3u8文件视频播放器(暂无源码): 点击欣赏! http://lxmc.aomaya.com/fengzi/m3u8/m3u8Player.swf
- python --- 字符编码学习小结(二)
距离上一篇的python --- 字符编码学习小结(一)已经过去2年了,2年的时间里,确实也遇到了各种各样的字符编码问题,也能解决,但是每次都是把所有的方法都试一遍,然后终于正常.这种方法显然是不科学 ...
- 7、注解@Mapper、@MapperScan
7.注解@Mapper.@MapperScan 2018年09月20日 11:12:41 飞奔的加瓦 阅读数 3284 版权声明:版权声明:本文为博主原创文章,未经博主允许不得转载. https ...
- Windows中的消息与消息队列
消息 在Windows中,消自由MSG结构体表示 typedef struct tagMSG { HWND hwnd; UINT message; WPARAM wParam; LPARAM lPar ...