write & read a sequence file(基于全新2.2.0API)
write & read a sequence file
write & read a sequence file
import java.io.IOException; import org.apache.hadoop.io.SequenceFile; import org.apache.hadoop.io.SequenceFile.Writer; import org.apache.hadoop.io.SequenceFile.Reader; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.conf.Configuration; public class MySequenceFile { static private final String[] DATA = { "this is the first", "this is the second", "this is the third", "this is the forth" }; public static void main(String[] args) throws IOException { Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(conf); Path path = new Path(args[0]); IntWritable key = new IntWritable(); Text value = new Text(); SequenceFile.Writer writer = null; writer = SequenceFile.createWriter(conf, Writer.file(path), Writer.keyClass(key.getClass()), Writer.valueClass(value.getClass())); for( int i = 0; i < 1000; i++ ) { key.set(i + 1); value.set(DATA[i % DATA.length]); writer.append(key,value); } writer.close(); SequenceFile.Reader reader = new SequenceFile.Reader(conf, Reader.file(path)); while( reader.next(key, value) ) { String syncSeen = reader.syncSeen() ? "*" : "#"; System.err.println(key + "\t" + value + "\t" + reader.getPosition()+ "\t" + syncSeen); } reader.close(); } }
write & read a sequence file(基于全新2.2.0API)的更多相关文章
- MapReduce——计算温度最大值 (基于全新2.2.0API)
MapReduce——计算温度最大值 (基于全新2.2.0API) deprecated: Job类的所有Constructors, 新的API用静态方法getInstance(conf)来去的Job ...
- write & read a MapFile(基于全新2.2.0API)
write & read a MapFile import java.io.IOException; import org.apache.hadoop.io.IntWritable; imp ...
- Configurataion Printer(基于全新2.2.0API)
Configurataion Printer import java.util.Map.Entry; import org.apache.hadoop.conf.Configuration; impo ...
- Combine small files to Sequence file
Combine small files to sequence file or avro files are a good method to feed hadoop. Small files in ...
- Predicting effects of noncoding variants with deep learning–based sequence model | 基于深度学习的序列模型预测非编码区变异的影响
Predicting effects of noncoding variants with deep learning–based sequence model PDF Interpreting no ...
- Flume性能测试报告(翻译Flume官方wiki报告)
因使用flume的时候总是会对其性能有所调研,网上找的要么就是自测的这里找到一份官方wiki的测试报告供大家参考 https://cwiki.apache.org/confluence/display ...
- Hadoop IO基于文件的数据结构详解【列式和行式数据结构的存储策略】
Charles所有关于hadoop的文章参考自hadoop权威指南第四版预览版 大家可以去safari免费阅读其英文预览版.本人也上传了PDF版本在我的资源中可以免费下载,不需要C币,点击这里下载. ...
- 基于docker快速搭建hbase集群
一.概述 HBase是一个分布式的.面向列的开源数据库,该技术来源于 Fay Chang 所撰写的Google论文"Bigtable:一个结构化数据的分布式存储系统".就像Bigt ...
- The Kernel Newbie Corner: Kernel Debugging with proc "Sequence" Files--Part 3
转载:https://www.linux.com/learn/linux-career-center/44184-the-kernel-newbie-corner-kernel-debugging-w ...
随机推荐
- [转]从一行代码里面学点JavaScript
现如今,JavaScript无处不在,因此关于JavaScript的新知识也是层出不穷.JavaScript的特点在于,要学习它的语法入门简简单,但是要精通使用它的方式却是一件不容易的事. 来看看下面 ...
- android 权限总结
1.拨打电话要权限 2.sd目录存东西要权限
- From MSI to WiX, Part 2 - ARP support, by Alex Shevchuk
Following content is directly reprinted from From MSI to WiX, Part 2 - ARP support Author: Alex Shev ...
- HDOJ 1042 N! -- 大数运算
题目地址:http://acm.hdu.edu.cn/showproblem.php?pid=1042 Problem Description Given an integer N(0 ≤ N ≤ 1 ...
- 队列(顺序存储)C++模板实现
队列:一端进行插入,另一端进行删除的线性结构,具有先进先出性.利用数组来实现队列将面临"假溢出"的情况,如下图所示: front:永远指向队首元素,队首在本文中是允许删除元素的一端 ...
- js中的in-for循环
<!doctype html><html lang="en"> <head> <meta charset="UTF-8" ...
- 通过WebApi取出XML数据
Get请求: public static Result<GetExpressCollectionResponseType> GetDataFromWebs(string waybillNu ...
- win7定时任务
最近某app有个小抽奖,每天点击太麻烦,想做个定时任务访问抽奖链接(带着cookie登录信息,即可抽奖成功) 刚开始做了php,最后发现部署在百度bae中没法做定时任务每天执行,后来换成了python ...
- jquery控制左右箭头滚动图片列表
jquery控制左右箭头滚动图片列表的实例. 代码如下: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN&q ...
- SQL技术内幕一
范式:关系模型的规范化规则. Codd提出的三个数据库范式: 1. 第一范式 第一范式要求表中的每一行都是必须是唯一的.因为关系型数据库是基于集合论的,而集合的定义中,要求每一个元素都是唯一的(在关系 ...