HDFS重启集群导致数据损坏，使用fsck命令修复过程
HDFS重启集群导致数据损坏，使用fsck命令修复过程
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
我们先看一组输出
[root@flume112 ~]# hdfs fsck /user/kuaikan/report_new////
Connecting to namenode via http://calculation111.aggrx:50070
FSCK started by root (auth:SIMPLE) from /10.1.2.112 for path /user/kuaikan/report_new//// at Sat May :: CST
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................Status: HEALTHY
Total size: B (Total open files size: B)
Total dirs:
Total files:
Total symlinks: (Files currently being written: )
Total blocks (validated): (avg. block size B) (Total open file blocks (not validated): ) #注意，这个26没有进行验证，说明有26个存在问题！
Minimally replicated blocks: (100.0 %)
Over-replicated blocks: (0.0 %)
Under-replicated blocks: (0.0 %)
Mis-replicated blocks: (0.0 %)
Default replication factor:
Average block replication: 3.0
Corrupt blocks:
Missing replicas: (0.0 %)
Number of data-nodes:
Number of racks:
FSCK ended at Sat May :: CST in milliseconds The filesystem under path '/user/kuaikan/report_new/2019/05/23/' is HEALTHY
[root@flume112 ~]#
官方关于fsck命令有对应的解释,我们查看更详细的信息，官方链接：https://hadoop.apache.org/docs/r2.9.1/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#fsck。
[root@flume112 ~]# hdfs fsck /user/kuaikan/report_new//// -openforwrite
Connecting to namenode via http://calculation111.aggrx:50070
FSCK started by root (auth:SIMPLE) from /10.1.2.112 for path /user/kuaikan/report_new//// at Sat May :: CST
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
................................................/user/kuaikan/report_new////report.log.----45.10---160.1558554302405.txt bytes, block(s), OPENFORWRITE: ../user/kuaikan/report_new////report.log.----45.10---248.1558554301696.txt bytes, block(s), OPENFORWRITE: ....../user/kuaikan/report_new////report.log.----45.10---71.1558554301576.txt bytes, block(s), OPENFORWRITE: .........................................
/user/kuaikan/report_new////report.log.----55.10---208.1558554901562.txt bytes, block(s), OPENFORWRITE: .../user/kuaikan/report_new////report.log.----55.10---248.1558554901949.txt bytes, block(s), OPENFORWRITE: ...../user/kuaikan/report_new////report.log.----55.10---233.1558554901647.txt bytes, block(s), OPENFORWRITE: .........................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
................................................................/user/kuaikan/report_new////report.log.----30.10---208.1558575001741.txt bytes, block(s), OPENFORWRITE: ...................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
......................./user/kuaikan/report_new////report.log.----55.10---93.1558587301545.txt bytes, block(s), OPENFORWRITE: ........../user/kuaikan/report_new////report.log.----55.10---88.1558587301803.txt bytes, block(s), OPENFORWRITE: .........................../user/kuaikan/report_new////report.log.----01.10---88.1558587709627.txt bytes, block(s), OPENFORWRITE: .....................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
.........................................................................../user/kuaikan/report_new////report.log.----45.10---35.1558601101874.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---40.1558601102383.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---46.1558601101876.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---58.1558601102255.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---93.1558601102225.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---38.1558601101874.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---208.1558601101557.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---160.1558601101782.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---33.1558601102283.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---150.1558601101668.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---248.1558601101427.txt bytes, block(s), OPENFORWRITE: ./user/kuaikan/report_new////report.log.----45.10---226.1558601101904.txt bytes, block(s), OPENFORWRITE: ./user/kuaikan/report_new////report.log.----45.10---96.1558601101582.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---88.1558601101302.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---233.1558601101531.txt bytes, block(s), OPENFORWRITE: /user/kuaikan/report_new////report.log.----45.10---71.1558601101720.txt bytes, block(s), OPENFORWRITE: .......
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
..............................................................Status: HEALTHY
Total size: B
Total dirs:
Total files:
Total symlinks:
Total blocks (validated): (avg. block size B)
Minimally replicated blocks: (99.50153 %)
Over-replicated blocks: (0.0 %)
Under-replicated blocks: (0.0 %)
Mis-replicated blocks: (0.0 %)
Default replication factor:
Average block replication: 2.985046
Corrupt blocks:
Missing replicas: (0.0 %)
Number of data-nodes:
Number of racks:
FSCK ended at Sat May :: CST in milliseconds The filesystem under path '/user/kuaikan/report_new/2019/05/23/' is HEALTHY
[root@flume112 ~]#
[root@flume112 ~]# hdfs fsck /user/kuaikan/report_new/2019/05/23/ -openforwrite
找到损坏的文件后，我们手动进行回复，注意，我们输入的必须是一个文件，并不是一个目录哟！
[root@calculation111 ~]# hdfs debug recoverLease -path /user/kuaikan/report_new////report.log.----45.10---71.1558554301576.txt -retries
recoverLease returned false.
Retrying in ms...
Retry #
recoverLease SUCCEEDED on /user/kuaikan/report_new////report.log.----45.10---71.1558554301576.txt
[root@calculation111 ~]#
HDFS重启集群导致数据损坏，使用fsck命令修复过程的更多相关文章
- Hadoop基础-HDFS集群中大数据开发常用的命令总结
Hadoop基础-HDFS集群中大数据开发常用的命令总结 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本盘博客仅仅列出了我们在实际生成环境中常用的hdfs命令,如果想要了解更多, ...
- Hadoop集群-HDFS集群中大数据运维常用的命令总结
Hadoop集群-HDFS集群中大数据运维常用的命令总结 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客会简单涉及到滚动编辑,融合镜像文件,目录的空间配额等运维操作简介.话 ...
- Hadoop（八）Java程序访问HDFS集群中数据块与查看文件系统
前言 我们知道HDFS集群中,所有的文件都是存放在DN的数据块中的.那我们该怎么去查看数据块的相关属性的呢?这就是我今天分享的内容了 一.HDFS中数据块概述 1.1.HDFS集群中数据块存放位置 我 ...
- 本地日志数据实时接入到hadoop集群的数据接入方案
1. 概述 本手册主要介绍了,一个将传统数据接入到Hadoop集群的数据接入方案和实施方法.供数据接入和集群运维人员参考. 1.1. 整体方案 Flume作为日志收集工具,监控一个文件目录或者一个文 ...
- elasticsearch5.0集群大数据量迁移方法及注意事项
当es集群的数据量较小的情况下elasticdump这个工具比较方便,但是当数据量达到一定级别比如上百G的时候,elasticdump速度就很慢了,此时我们可以使用快照的方法进行备份 elasticd ...
- MongoDB DBA 实践5-----复制集集群的数据同步和故障转移
(1)复制集集群的数据同步 1>主节点数据库test,在其中goods集合中加入一个文档. 2>在副节点中查看 注意:SECONDARY是不允许读写的,要使用rs.slaveOk()获得读 ...
- 软件架构自学笔记----分享“去哪儿 Hadoop 集群 Federation 数据拷贝优化”
去哪儿 Hadoop 集群 Federation 数据拷贝优化 背景 去哪儿 Hadoop 集群随着去哪儿网的发展一直在优化改进,基本保证了业务数据存储量和计算量爆发式增长下的存储服务质量.然而,随着 ...
- Redis 切片集群的数据倾斜分析
Redis 中如何应对数据倾斜 什么是数据倾斜 数据量倾斜 bigkey导致倾斜 Slot分配不均衡导致倾斜 Hash Tag导致倾斜 数据访问倾斜 如何发现 Hot Key Hot Key 如何解决 ...
- KingbbaseES V8R6集群维护案例之---集群之间数据迁移
案例说明: 生产环境是集群环境,测试环境是集群,现需要将生产环境的数据迁移到测试集群中运行,本文档详细介绍了从集群环境迁移数据的操作步骤,可以作为生产环境迁移数据的参考. 适用版本: Kingbase ...
随机推荐
- Ubuntu新建用户并指定目录
例如我要新建一个nginx用户,并指定目录,允许使用bash登录 sudo useradd -d "/home/nginx" -m -s "/bin/bash" ...
- GitLab - GitLab的备份与还原
1 - GitLab配置文件 GitLab默认的配置文件路径:/etc/gitlab/ /etc/gitlab/gitlab.rb:主配置文件,包含外部URL.仓库目录.备份目录等 /etc/gitl ...
- C# 恢复工作空间，删除Unversioned文件，两种方式
//通过命令行清理svn的工作空间,删除未add的文件,revert修改的文件 public static bool RevertDelUnversioned(string svnPath) { tr ...
- python爬虫2
学习任务 获取去哪儿网的出发地列表 获取旅游景点列表 获取景点产品列表 存储数据 1 获取出发地站点 (1)访问touch.qunar.com (2)按F12,单击自由行,在自由行页面点击搜索框 (3 ...
- LeetCode 503. 下一个更大元素 II(Next Greater Element II)
503. 下一个更大元素 II 503. Next Greater Element II 题目描述 给定一个循环数组(最后一个元素的下一个元素是数组的第一个元素),输出每个元素的下一个更大元素.数字 ...
- EditPlus配置ftp连接linux
选择文件/FTP下面的设置FTP服务器 1.点击添加 2.填写名称.ftp服务器.用户名.密码信息 3.点击高级设置 4.选择加密方式为sftp,端口22,如果不填端口号,默认也是22,确定 5.确定 ...
- 《算法 - 一致性 (hash) 算法》
图片摘自: 每天进步一点点——五分钟理解一致性哈希算法(consistent hashing) 一:背景 - 一致性哈希算法在1997年由麻省理工学院的Karger等人在解决分布式Cache中提出的. ...
- 06-switch语句
switch语句 switch是一个条件语句,它是可以代替多个if else的常用方式 例子 package main import "fmt" func main() { a:= ...
- leetcode最长回文
给定一个字符串 s,找到 s 中最长的回文子串.你可以假设 s 的最大长度为 1000. 示例 1: 输入: "babad" 输出: "bab" 注意: &qu ...
- proxy_banner