under replicated blocks
解决:
找出没有复制的block:
hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files
然后循环修复:
for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep $hdfsfile; done
输出如下:
Fixing /user/hdfs/.staging/job_1547173493660_0405/job.jar :
Replication set: /user/hdfs/.staging/job_1547173493660_0405/job.jar
Fixing /user/hdfs/.staging/job_1547173493660_0405/job.split :
Replication set: /user/hdfs/.staging/job_1547173493660_0405/job.split
Fixing /user/hdfs/.staging/job_1547173493660_0481/job.jar :
Replication set: /user/hdfs/.staging/job_1547173493660_0481/job.jar
Fixing /user/hdfs/.staging/job_1547173493660_0481/job.split :
Replication set: /user/hdfs/.staging/job_1547173493660_0481/job.split
Fixing /user/hdfs/.staging/job_1547173493660_0483/job.jar :
Replication set: /user/hdfs/.staging/job_1547173493660_0483/job.jar
Fixing /user/hdfs/.staging/job_1547173493660_0483/job.split :
Replication set: /user/hdfs/.staging/job_1547173493660_0483/job.split
Fixing /user/hdfs/.staging/job_1547197402450_0021/job.jar :
Replication set: /user/hdfs/.staging/job_1547197402450_0021/job.jar
Fixing /user/hdfs/.staging/job_1547197402450_0021/job.split :
Replication set: /user/hdfs/.staging/job_1547197402450_0021/job.split

HDFS under replicated blocks的更多相关文章

  1. CDH 报错:under replicated blocks

    1.刚安装好CDH5.16.1集群(2个节点)出现了under replicated blocks错误,如下图 2.原因是CDH默认文件备份3份,而我们只有2节点,所以解决方法如下: ①集群增加一个新 ...

  2. hdp 集群问题解决记录

    2019-04-23 14:16:21,769 WARN namenode.FSImage (EditLogFileInputStream.java:scanEditLog(359)) - Caugh ...

  3. hdfs 如何实现退役节点快速下线(也就是退役节点上的数据块快速迁移)speed up decommission blocks removal

    以下是选择复制源节点的代码 代码总结: A=datanode上要复制block的Queue size与 target datanode没被选出之前待处理复制工作数之和. 1. 优先选择退役中的节点,因 ...

  4. [bigdata] 使用Flume hdfs sink, hdfs文件未关闭的问题

    现象: 执行mapreduce任务时失败 通过hadoop fsck -openforwrite命令查看发现有文件没有关闭. [root@com ~]# hadoop fsck -openforwri ...

  5. HDFS 搭建记录

    1. 三台服务: 172.17.0.62(namenode) 172.17.0.68(datanode) 172.17.0.76(datanode) /etc/hosts包含的内容: 三台都包含的域名 ...

  6. hdfs 常用命令

    (2)bin/hdfs dfs -mkdir -p /home/雨渐渐 (3)scp /media/root/DCE28B65E28B432E/download/第2周/ChinaHadoop第二讲\ ...

  7. [转载] HDFS and Erasure Codes (HDFS-RAID)

    The Hadoop Distributed File System has been great in providing a cloud-type file system. It is robus ...

  8. 【Hadoop】 2.7.3版本 hdfs 命令行使用

    1.查看HDFS下目录结构及文件 dream361@ubuntu:~$ hdfs dfs -ls -R / 2.创建文件目录/tmp dream361@ubuntu:~$ hdfs dfs -mkdi ...

  9. How to fix corrupt HDFS FIles

    1 问题描述 HDFS在机器断电或意外崩溃的情况下,有可能出现正在写的数据(例如保存在DataNode内存的数据等)丢失的问题.再次重启HDFS后,发现hdfs无法启动,查看日志后发现,一直处于安全模 ...

随机推荐

  1. Zabbix-server及zabbix-web安装手册(centos7)

    注:本实验环境在centos7mini下实验,具体相关软件版本最好一致,避免依赖关系缺失. 当然也可以新建用户zabbix(不使用root,避免安全隐患),在zabbix下执行操作,加sudo提升权限 ...

  2. scala对象简单记录

    object Person { private val eyeNum = 2 def getEyeNum = eyeNum def main(args: Array[String]): Unit = ...

  3. The chance for love doesn't come around every day.

    The chance for love doesn't come around every day.爱的机会不是每天都有的.

  4. magento 跳转

    Magento: Redirect functions 原文:http://blog.chapagain.com.np/magento-redirect-functions/ The redirect ...

  5. bzoj1029题解

    [解题思路] 贪心,先按结束时间排序,从左到右扫描过去,如果当前建筑可以修复则入大根堆,否则,若其修复时间比堆顶小则弹出堆顶并入堆,处理完堆后则更新总时间.复杂度O(nlog2n). [参考代码] # ...

  6. NX二次开发-UFUN输入表达式名称,获取它的名称和值UF_MODL_ask_exp

    NX9+VS2012 #include <uf.h> #include <uf_modl.h> UF_initialize(); //创建一个新的表达式,有TAG tag_t ...

  7. hdu多校第八场 1010(hdu6666) Quailty and CCPC 排序/签到

    题意: CCPC前10%能得金牌,给定队伍解题数和罚时,问你有没有一个队伍如果向上取整就金了,四舍五入就银了. 题解: 排序后按题意求解即可. #include<iostream> #in ...

  8. 在vs2010下编译boost

    1. 解压缩后,运行bootstrap.bat批处理文件,得到bjam.exe; 2. 在vs2010下编译boost boost最新版本已经支持vs2010,然而直接下载编译会发现一堆bug: 首先 ...

  9. 记录一次失败的向git提交代码,和解决的方法。(首次创建仓库)

    背景: 向git push代码(创建一个新的仓库) 做法: 在github创建一个新的仓库------>本地新建文件夹------->依次执行了下面的命令 git init git clo ...

  10. class1and2_tkinter之 Label & Button 标签和按钮

    ''' # ------------------------------------------------------------ # # 1. 使用tkinter设置一个标签 # # # 设计tk ...