[转载] HDFS and Erasure Codes (HDFS-RAID)
The Hadoop Distributed File System has been great in providing a cloud-type file system. It is robust (when administered correctly :-)) and highly scalable. However, one of the main drawbacks of HDFS is that each piece of data is replicated in three places. This is acceptable because disk storage is cheap and is becoming cheaper by the day; this isn't a problem if you have a relatively small to medium size cluster. The price difference (in absolute terms) is not much whether you use 15 disks or whether you use 10 disks. If we consider the cost of $1 per GByte, the price difference between fifteen 1 TB disk and ten 1 TB disk is only $5K. But when the total size of your cluster is 10 PBytes, then the costs savings in storing the data in two places versus three is a huge ten million dollars!
The reason HDFS stores disk blocks in triplicate is because it uses commodity hardware and there is non-negligible probability of a disk failure. It has been observed that a replication factor of 3 and the fact the HDFS aggressively detects failures and immediately replicates failed -block-replicas is sufficient to never lose any data in practice. The challenge now is to achieve an effective replication factor of 3 while keeping the real physical replication factor at close to 2! How best to do it than by usingErasure Codes.
I heard about this idea called DiskReduce from the folks at CMU. The CMU PDL Labs has been a powerhouse of research in file systems and it is no surprise that they proposed a elegant way of implementing erasure codes in HDFS. I borrowed heavily from their idea in my implementation of Erasure Codes in HDFS described in HDFS-503. One of the main motivation of my design is to keep the HDFS Erasure Coding as a software layer above HDFS rather than inter-twining it inside of HDFS code. The HDFS code is complex by itself and it is really nice to not have to make it more complex and heavyweight.
Distributed Raid File System consists of two main software components. The first component is the RaidNode, a daemon that creates parity files from specified HDFS files. The second component "raidfs" is a software that is layered over a HDFS client and it intercepts all calls that an application makes to the HDFS client. If the HDFS client encounters corrupted data while reading a file, the raidfs client detects it; it uses the relevant parity blocks to recover the corrupted data (if possible) and returns the data to the application. The application is completely transparent to the fact that parity data was used to satisfy it's read request. The Distributed Raid File System can be configured in such a way that a set of data blocks of a file are combined together to form one or more parity blocks. This allows one to reduce the replication factor of a HDFS file from 3 to 2 while keeping the failure probabilty relatively same as before.
I have seen that using a stripe size of 10 blocks decreases the physical replication factor of a file to 2.2 while keeping the effective replication factor of a file at 3. This typically results in saving 25% to 30% of storage space in a HDFS cluster.
One of the shortcoming of this implementation is that we need a parity file for every file in HDFS. This potentially increases the number of files in the NameNode. To alleviate this problem, I will enhance this implementation (in future) to use the Hadoop Archive feature to archive all the parity files together in larger containers so that the NameNode does not have to support additional files when the HDFS Erasure Coding is switched on. This works reasonably well because it is a very very rare case that the parity files are ever used to satisfy a read request.
ref: http://hadoopblog.blogspot.com/2009/08/hdfs-and-erasure-codes-hdfs-raid.html
[转载] HDFS and Erasure Codes (HDFS-RAID)的更多相关文章
- HDFS Federation(转HDFS Federation(HDFS 联盟)介绍 CSDN)
转载地址:http://blog.csdn.net/strongerbit/article/details/7013221 HDFS Federation(HDFS 联盟)介绍 1. 当前HDFS架构 ...
- vivo 万台规模 HDFS 集群升级 HDFS 3.x 实践
vivo 互联网大数据团队-Lv Jia Hadoop 3.x的第一个稳定版本在2017年底就已经发布了,有很多重大的改进. 在HDFS方面,支持了Erasure Coding.More than 2 ...
- HDFS shell操作及HDFS Java API编程
HDFS shell操作及HDFS Java API编程 1.熟悉Hadoop文件结构. 2.进行HDFS shell操作. 3.掌握通过Hadoop Java API对HDFS操作. 4.了解Had ...
- (第3篇)HDFS是什么?HDFS适合做什么?我们应该怎样操作HDFS系统?
摘要: 这篇文章会详细介绍HDFS是什么,HDFS的作用,适合和不适合的场景,我们该如何操作HDFS? HDFS文件系统 Hadoop 附带了一个名为 HDFS(Hadoop分布式文件系统)的分布 ...
- Hadoop之HDFS(一)HDFS入门及基本Shell命令操作
1 . HDFS 基本概念 1.1 HDFS 介绍 HDFS 是 Hadoop Distribute File System 的简称,意为:Hadoop 分布式文件系统.是 Hadoop 核心组件之 ...
- HDFS之二:HDFS文件系统JavaAPI接口
HDFS是存取数据的分布式文件系统,HDFS文件操作常有两种方式,一种是命令行方式,即Hadoop提供了一套与Linux文件命令类似的命令行工具.HDFS操作之一:hdfs命令行操作 另一种是Java ...
- hdfs的文件个数 HDFS Quotas Guide
HDFS Quotas Guide Overview HDFS允许管理员为多个每个目录设置使用的命名空间和空间的配额.命名空间配额和空间配额独立操作,但是这两种类型的配额的管理和实现非常类似. Nam ...
- 【转载】Hadoop分布式文件系统HDFS的工作原理详述
转载请注明来自36大数据(36dsj.com):36大数据 » Hadoop分布式文件系统HDFS的工作原理详述 转注:读了这篇文章以后,觉得内容比较易懂,所以分享过来支持一下. Hadoop分布式文 ...
- 【转载】经典漫画讲解HDFS原理
分布式文件系统比较出名的有HDFS 和 GFS,其中HDFS比较简单一点.本文是一篇描述非常简洁易懂的漫画形式讲解HDFS的原理.比一般PPT要通俗易懂很多.不难得的学习资料. 1.三个部分: 客户 ...
随机推荐
- SAE、搜狐云景和百度云之初见
近期有需求将我们的应用部署到公有云的服务平台上,于是找了几家公有云服务做了一下调研, 首先对比一下他们提供的功能: 功能 SAE 搜狐云景 百度云 版本控制工具 svn GIT,和百度云的比起来,用 ...
- 转:Netty系列之Netty高性能之道
1. 背景 1.1. 惊人的性能数据 最近一个圈内朋友通过私信告诉我,通过使用Netty4 + Thrift压缩二进制编解码技术,他们实现了10W TPS(1K的复杂POJO对象)的跨节点远程服务调用 ...
- udp丢包原因分析
1. 发送方没有进行频率控制(令牌桶算法),短时间内大量的包发送到server端,server端是单线程,先epoll wait,再process,就会造程process时丢掉server传过来的包 ...
- 【Hybrid App】关于Hybrid App技术解决方案的选择
[引言]近年来随着移动设备类型的变多,操作系统的变多,用户需求的增加,对于每个项目启动前,大家都会考虑到的成本,团队成员,技术成熟度,时间,项目需求等一堆的因素.因此,开发App的方案已经变得越来越多 ...
- sqlserver如何启动数据库邮件
可以在 msdb 数据库中停止数据库邮件.若要检查数据库邮件的状态,请执行下面的语句: 复制 EXECUTE dbo.sysmail_help_status_sp; 若要在邮件主机数据库中启动数据库邮 ...
- C# 实现文件或目录复制到指定目录
public void CopyFilesToDirKeepSrcDirName(string srcPath, string destDir) { if (Directory.Exists(srcP ...
- SVN 服务器IP地址变更后客户端的修改
找到需要更新修改的代码目录: 在右键菜单TortoiseSVN>Relocate……对话框中可以更改svn服务器地址. so easy!
- Introduction to Web Services
What are Web Services? Web Services are client and server applications that communicate over the Wor ...
- javascript动态改变当前页面中元素的状态行为
function Datea() { var timed = document.getElementById('timed'); var t = setInterval(function TDate( ...
- 让Barebox正确引导Tiny6410的linux内核
本文专属tiny6410吐槽系列~如有得罪,请多包涵~在此也留个联系方式~291222536@qq.com,欢迎大家一起交流吐槽心得~ 上节各位搞得如何~这次怎么修改barebox的一些小细节,并且配 ...