把以下脚本配置到 crontab 下 ,10分钟一次,进行打点,记录复制块的进度。

想让节点快速退役(下线)的方法可以参考我的 https://www.cnblogs.com/jiangxiaoxian/p/9665588.html 文章。

#!/bin/bash
source /etc/profile
hdfs dfsadmin -metasave 1.meta
date +%Y%m%d-%H:%M:%S >> /opt/hadoop/hadoop-2.6.0/logs/meta_recods.log
cat /opt/hadoop/hadoop-2.6.0/logs/1.meta | grep 'Metasave: Blocks waiting' >> /opt/hadoop/hadoop-2.6.0/logs/meta_recods.log

日志 meta_recods.log 内容

20180925-14:00:18
Metasave: Blocks waiting for replication: 3814058
20180925-14:10:16
Metasave: Blocks waiting for replication: 3801943
20180925-14:20:16
Metasave: Blocks waiting for replication: 3789778
20180925-14:30:18
Metasave: Blocks waiting for replication: 3777659
20180925-14:40:22
Metasave: Blocks waiting for replication: 3765676
20180925-14:50:18
Metasave: Blocks waiting for replication: 3753631
20180925-15:00:17
Metasave: Blocks waiting for replication: 3741611
......

线上参数如下(根据打点日志 1个小时完成了7w左右个块的拷贝。)

线上要退役的节点有400w个块,那理论上一天可完成170w个块(但是hdfs晚上要跑大量的按天任务,比较繁忙),2.5天可以完成。

<!-- speed up decommission  -->
<property>
<name>dfs.namenode.replication.max-streams</name>
<value>10</value>
</property>
<property>
<name>dfs.namenode.replication.max-streams-hard-limit</name>
<value>20</value>
</property>
<property>
<name>dfs.namenode.replication.work.multiplier.per.iteration</name>
<value>5</value>
</property>

metasave 的日志示意

源码中是BlockManager的metaSave(PrintWriter out) 方法

19349608 files and directories, 14515745 blocks = 33865353 total  #共有3.3千万的对象在namenode的内存中
Live Datanodes: 15 #有15个datenode in service
Dead Datanodes: 0 # 死了的dn节点
Metasave: Blocks waiting for replication: 2380757 # 等待要复制的块有238w个
# l: == live(in service):, d: == decommissioned(退役) c: == corrupt(损坏) e: == excess(超额)
# 以下是 文件名 --- 等待复制的block名 --- l:2 d:1 c:0 e:0 --- block所在的节点
/user/hive/warehouse/ods.db/xxx/order_datekey=20170213/000003_1: blk_1286491219_213142207 (replicas: l: 2 d: 1 c: 0 e: 0) xx.x.x.30:50010 : xx.x.x.36:50010 : xx.x.x.34:50010(decommissioned) :
/user/hive/warehouse/analytics.db/xxx/datekey=20180810/part=5/part-0001: blk_1529882603_456688042 (replicas: l: 2 d: 1 c: 0 e: 0) xx.x.x.11:50010 : xx.x.x.34:50010(decommissioned) : xx.x.x.41:50010 :
...
Mis-replicated blocks that have been postponed: # 在做namenode 故障转移时,待复制的block 会延迟复制(新active namenode 并不知道要复制哪些块,只到dn的一次report),所以这个也是空(不发生切换ha的情况)
Metasave: Blocks being replicated: 102 # 当前正在复制的 block 102个
blk_1308167564_234830077 StartTime: 10:39:58 NumReplicaInProgress: 1
blk_1475963863_402729834 StartTime: 10:39:58 NumReplicaInProgress: 1
blk_1090092594_16396468 StartTime: 10:39:58 NumReplicaInProgress: 1
...
Metasave: Blocks 333 waiting deletion from 4 datanodes. # 待被删除的block
xx.x.x.11:50010
LightWeightHashSet(size=81, modification=81, entries.length=128)
xx.x.x.21:50010
LightWeightHashSet(size=90, modification=90, entries.length=128)
xx.x.x.36:50010
LightWeightHashSet(size=75, modification=75, entries.length=128)
xx.x.x.41:50010
LightWeightHashSet(size=87, modification=87, entries.length=128)
Metasave: Number of datanodes: 15 # datanode节点ip:port --- 机架 --- 服务状态(in service;Decommission In Progress;) --- 储存信息(磁盘最大可用,剩余,已用空间)---dump meta 时间 --- 所在节点block 的复制情况及失效情况
xxx.xxx.xxx.xx:50010 default_rack IN 35719153532928(32.49 TB) 19426144912589(17.67 TB) 0.54% 14196798403289(12.91 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018 1 blocks to be replicated;
xxx.xxx.xxx.xx:50010 default_rack IN 54168542494720(49.27 TB) 18217462271284(16.57 TB) 0.34% 35222156404459(32.03 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018 6 blocks to be replicated;
xxx.xxx.xxx.xx:50010 default_rack IN 35719153532928(32.49 TB) 19584613210523(17.81 TB) 0.55% 13989217942458(12.72 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018
xxx.xxx.xxx.34:50010 default_rack DP 43039268741120(39.14 TB) 19133599044243(17.40 TB) 0.44% 16143418817257(14.68 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018 2 blocks to be replicated;
xxx.xxx.xxx.xx:50010 default_rack IN 35719153532928(32.49 TB) 19639941377872(17.86 TB) 0.55% 13921822275178(12.66 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018 64 blocks to be invalidated;
xxx.xxx.xxx.xx:50010 default_rack IN 43044577198080(39.15 TB) 22257842728998(20.24 TB) 0.52% 8827718379586(8.03 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018
xxx.xxx.xxx.xx:50010 default_rack IN 54168542494720(49.27 TB) 19201190599407(17.46 TB) 0.35% 33335553166140(30.32 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018
xxx.xxx.xxx.xx:50010 default_rack IN 54168542494720(49.27 TB) 19669312248147(17.89 TB) 0.36% 32865881808839(29.89 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018
xxx.xxx.xxx.xx:50010 default_rack IN 37762378219520(34.34 TB) 3246159556995(2.95 TB) 0.09% 33678460006824(30.63 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018 63 blocks to be invalidated;
xxx.xxx.xxx.xx:50010 default_rack IN 54168542494720(49.27 TB) 20323539696784(18.48 TB) 0.38% 32271527069120(29.35 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018 7 blocks to be replicated; 73 blocks to be invalidated;
xxx.xxx.xxx.xx:50010 default_rack IN 22880316538880(20.81 TB) 4524463118033(4.11 TB) 0.20% 16963981545595(15.43 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018 80 blocks to be invalidated;
xxx.xxx.xxx.xx:50010 default_rack IN 37723723513860(34.31 TB) 2242584812391(2.04 TB) 0.06% 35079540988600(31.90 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018
xxx.xxx.xxx.xx:50010 default_rack IN 37723723513860(34.31 TB) 864073122598(804.73 GB) 0.02% 36507695599616(33.20 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018
xxx.xxx.xxx.xx:50010 default_rack IN 32044152463360(29.14 TB) 20191502077098(18.36 TB) 0.63% 4276374611465(3.89 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018 5 blocks to be replicated;
xxx.xxx.xxx.xx:50010 default_rack IN 35719153532928(32.49 TB) 19310209465238(17.56 TB) 0.54% 14261995815199(12.97 TB) 0(0 B) 0(0 B) NaN% 0(0 B) Wed Sep 26 10:40:04 CST 2018

监控hdfs 块迁移数量脚本及 metasave 日志内容详解的更多相关文章

  1. gvoory脚本中关于HttpClient使用详解实例

    一.gvoory脚本中关于HttpClient使用详解实例 HttpClient:是一个接口 首先需要先创建一个DefaultHttpClient的实例 HttpClient httpClient=n ...

  2. loadrunner 运行脚本-Run-time Settings-Browser Enmulation设置详解

    运行脚本-Run-time Settings-Browser Enmulation设置详解 by:授客 QQ:1033553122 浏览器模拟 所有Internet Vuser Header包含一个标 ...

  3. loadrunner 脚本优化-关联函数web_reg_save_param()函数详解

    脚本优化-关联函数web_reg_save_param()函数详解   by:授客 QQ:1033553122 Insert->New Step,打开Add Step对话框 选择函数web_re ...

  4. Shell脚本 /dev/null 2>&1详解

    Shell脚本---- /dev/null 2>&1详解   1.可以将/dev/null看作"黑洞". 它非常等价于一个只写文件. 所有写入它的内容都会永远丢失. ...

  5. 一键获取linux内存、cpu、磁盘IO等信息脚本编写,及其原理详解

    更多linux知识,请关注公众号:一口Linux 一.脚本 今天主要分享一个shell脚本,用来获取linux系统CPU.内存.磁盘IO等信息. #!/bin/bash # 获取要监控的本地服务器IP ...

  6. shell 脚本之判断语句 if 详解

    使用 Linux 系统这么长时间,对 shell 脚本也算是比较熟悉.其实不管是搞开发,还是搞运维,shell 脚本都是必备的基本技能.这次抽时间好好总结一下 shell 方面的知识,综合的再学习一下 ...

  7. IdentityServer4迁移至3.x版本注意问题详解

    前言 之前有一位购买我课程的童鞋利用最新的IdentityServer4版本即对应.NET Core 3.x,发布到生产环境在学习,结果出了一些问题,此前我并未过多关注IdentityServer4升 ...

  8. Shell脚本———— /dev/null 2>&1详解

    1.可以将/dev/null看作"黑洞". 它非常等价于一个只写文件. 所有写入它的内容都会永远丢失. 而尝试从它那儿读取内容则什么也读不到. 然而, /dev/null对命令行和 ...

  9. (转) html块级元素和内联元素区别详解

    http://blog.csdn.net/chen_zw/article/details/8713205 块级元素(block)特性: 总是独占一行,表现为另起一行开始,而且其后的元素也必须另起一行显 ...

随机推荐

  1. 未找到路径“/Agent/SissQrTemplate/AddN”的控制器或该控制器未实现 IController。

    未找到路径“/Agent/SissQrTemplate/AddN”的控制器或该控制器未实现 IController. Controller 命名空间错误,应该是Areas.Agent.Controll ...

  2. 为IE内核的WebBrowser控件内存泄漏所烦恼的可以考虑用Cefsharp代替它!

    为IE内核的WebBrowser控件内存泄漏所烦恼的朋友们,可以考虑用Cefsharp代替WebBrowser控件 特意做了一个程序来测试 利用Cefsharp做控件,访问网站.每分钟刷新2次,初始时 ...

  3. 【linux】centos6.9设置etc0网卡开机自动获取ip

    在vm新安装的centos系统中,一般选择NAT来设置和主机共享局域网,通过ifconfig etc0 192.168.xx.xx 这种作法机器重启之后就会失效,所以可以使用更改文件的方式完成设置ce ...

  4. Redis集群事物提交异常Multi-key operations must involve a single slot

    redis做完集群后不同键在同一事物中提交,因为key的hash计算结果不同不能分配到同一个分片上,因此出现此异常. 解决方案:在本次事物的key内添加"{tag}",这时redi ...

  5. Java学习——Applet画8个同心圆

    import java.awt.*; import java.applet.*; public class GUI3 extends Applet{ public void paint(Graphic ...

  6. 学习笔记之Supervised Learning with scikit-learn | DataCamp

    Supervised Learning with scikit-learn | DataCamp https://www.datacamp.com/courses/supervised-learnin ...

  7. 廖雪峰Java2面向对象编程-1面向对象-1面向对象基础

    1.对象的概念 面向对象编程:Object-Oriented Programming 对现实世界建立计算机模型的一种编程方法. 现实世界 计算机模型 Java代码 人 类/class class Pe ...

  8. Vue + TypeScript + ElementUI 封装表头查询组件

    前段时间有朋友私信我 Vue + TypeScript 的问题,然后就打算写一篇 Vue + TypeScript 封装组件的文章 正好公司项目中需要封装一个表头查询组件,就拿出来分享一下~ 组件的整 ...

  9. 第二章:安装zabbix过程

    2.2 安装zabbix过程 2.2.1 安装方式选择 编译安装 (服务较多,环境复杂) yum安装(干净环境) 使用yum 需要镜像yum源 http://www.cnblogs.com/clsn/ ...

  10. HibernateUtil工具类的使用

    为了简化代码的重复性,使用HibernateUtil工具类对Hibernate有关的代码进行整合 主要实现有,getSessionFactory(),getSession(),closeSession ...