hadoop深入学习之SequenceFile

的1个byte

3.Key和Value的类名

4.压缩相关的信息

5.其他用户定义的元数据

6.同步标记，sync marker

Metadata 在文件创建时就写好了，所以也是不能更改的。条记录存储一个索引映射。当然，记录间隔可人为修改，通过MapFIle.Writer的setIndexInterval()方法，或修改io.map.index.interval属性；
另外，与SequenceFile不同的是，MapFile的KeyClass一定要实现WritableComparable接口,即Key值是可比较的。

FileSystem fs=FileSystem.get(conf);  

Path mapFile=new Path("mapFile.map");  

   

//Writer内部类用于文件的写操作,假设Key和Value都为Text类型  

MapFile.Writer writer=new MapFile.Writer(conf,fs,mapFile.toString(),Text.class,Text.class);  

//通过writer向文档中写入记录  

writer.append(new Text("key"),new Text("value"));  

IOUtils.closeStream(writer);//关闭write流  

   

//Reader内部类用于文件的读取操作  

MapFile.Reader reader=new MapFile.Reader(fs,mapFile.toString(),conf);  

//通过reader从文档中读取记录  

Text key=new Text();  

Text value=new Text();  

while(reader.next(key,value)){  

    System.out.println(key);  

    System.out.println(key);  

}  

IOUtils.closeStream(reader);//关闭read流

1. MapFile       

   一个key-value 对应的查找数据结构，由数据文件/data 和索引文件 /index 组成，数据文件中包含所有需要存储的key-value对，按key的顺序排列。索引文件包含一部分key值，用以指向数据文件的关键位置。
   

   
   

   来源： http://blog.cloudera.com/blog/2011/01/hadoop-io-sequence-map-set-array-bloommap-files/

   The MapFile is a directory that contains two SequenceFile: the data file (“/data”) and the index file (“/index”). The data contains all the key, value records but key N + 1 must be greater then or equal to the key N. This condition is checked during the append() operation, if checkKey fail it throws an IOException “Key out of order”.

   
   

   

   The Index file is populated with the key and a LongWritable that contains the starting byte position of the record. Index does’t contains all the keys but just a fraction of the keys, you can specify the indexInterval calling setIndexInterval() method. The Index is read enteirely into memory, so if you’ve large map you can set a index skip value that allows you to keep in memory just a fraction of the index keys.

   
   

2. SetFile              – 基于 MapFile 实现的，他只有key，value为不可变的数据。SetFile and ArrayFile are based on MapFile, and their implementation are just few lines of code. The SetFile instead of append(key, value) as just the key field append(key) and the value is always the NullWritable instance. 

3. ArrayFile          – 也是基于 MapFile 实现，他就像我们使用的数组一样，key值为序列化的数字。The ArrayFile as just the value field append(value) and the key is a LongWritable that contains the record number, count + 1. 

4. BloomMapFile – 他在 MapFile 的基础上增加了一个 /bloom 文件，包含的是二进制的过滤表，在每一次写操作完成时，会更新这个过滤表The BloomMapFile extends the MapFile adding another file, the bloom file “/bloom”, and this file contains a serialization of the DynamicBloomFilter filled with the added keys. The bloom file is written entirely during the close operation.

If you want to play with SequenceFile, MapFile, SetFile, ArrayFile without using Java, I’ve written a naive implementation in python. You can find it, in my github repository python-hadoop.

 
 

参考url

--------------------------------

1.Hadoop I/O: Sequence, Map, Set, Array, BloomMap Files - Cloudera Engineering Blog

http://blog.cloudera.com/blog/2011/01/hadoop-io-sequence-map-set-array-bloommap-files/

== 以上 2017/7/6 下午5:29:17

--------------------------------

1.[hadoop源码阅读][4]-org.apache.hadoop.io - 阿笨猫 - 博客园

http://www.cnblogs.com/xuxm2007/archive/2012/06/15/2550986.html

== 以上 2017/7/6 下午5:06:26

http://hadoop.apache.org/common/docs/r0.20.2/api/index.html

http://blog.csdn.net/ludi7125/article/details/7605719

http://www.cloudera.com/blog/2011/01/hadoop-io-sequence-map-set-array-bloommap-files/

http://blog.nosqlfan.com/html/1217.html

http://jerrylead.iteye.com/blog/1181716

http://www.itivy.com/arch/archive/2011/12/12/hadoop-writable-interface-introduction.html