org.apache.hadoop.fs-BufferedFSInputStream
封装了FSInputStream
package org.apache.hadoop.fs; import java.io.BufferedInputStream;
import java.io.IOException; /**
* A class optimizes reading from FSInputStream by bufferring
*/
//通过缓存优化FSInputStream读取 public class BufferedFSInputStream extends BufferedInputStream
implements Seekable, PositionedReadable {
//两个接口在前面刚看过了,功能为...
/**
* Creates a <code>BufferedFSInputStream</code>
* with the specified buffer size,
* and saves its argument, the input stream
* <code>in</code>, for later use. An internal
* buffer array of length <code>size</code>
* is created and stored in <code>buf</code>.
*
* @param in the underlying input stream.
* @param size the buffer size.
* @exception IllegalArgumentException if size <= 0.
*/
public BufferedFSInputStream(FSInputStream in, int size) {
super(in, size);
}
//通过跟踪父类代码知道对接了输入流“管道”,初始化了一个大小为size的buffer
public long getPos() throws IOException {
return ((FSInputStream)in).getPos()-(count-pos);
}
//返回现在的偏移量
public long skip(long n) throws IOException {
if (n <= 0) {
return 0;
} seek(getPos()+n);
return n;
}
//跳过n长度后得到现偏移量
public void seek(long pos) throws IOException {
if( pos<0 ) {
return;
}
// optimize: check if the pos is in the buffer
long end = ((FSInputStream)in).getPos();
long start = end - count;
if( pos>=start && pos<end) {
this.pos = (int)(pos-start);
return;
} // invalidate buffer
this.pos = 0;
this.count = 0; ((FSInputStream)in).seek(pos);
}
//实现了Seekable的seek方法
public boolean seekToNewSource(long targetPos) throws IOException {
pos = 0;
count = 0;
return ((FSInputStream)in).seekToNewSource(targetPos);
}
//.....
public int read(long position, byte[] buffer, int offset, int length) throws IOException {
return ((FSInputStream)in).read(position, buffer, offset, length) ;
} public void readFully(long position, byte[] buffer, int offset, int length) throws IOException {
((FSInputStream)in).readFully(position, buffer, offset, length);
} public void readFully(long position, byte[] buffer) throws IOException {
((FSInputStream)in).readFully(position, buffer);
}
}
org.apache.hadoop.fs-BufferedFSInputStream的更多相关文章
- 用java运行Hadoop程序报错:org.apache.hadoop.fs.LocalFileSystem cannot be cast to org.apache.
用java运行Hadoop例程报错:org.apache.hadoop.fs.LocalFileSystem cannot be cast to org.apache.所写代码如下: package ...
- spark-shell报错:Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
环境: openSUSE42.2 hadoop2.6.0-cdh5.10.0 spark1.6.0-cdh5.10.0 按照网上的spark安装教程安装完之后,启动spark-shell,出现如下报错 ...
- 报错:Exception in thread "main" java.lang.NoClassDefFoundError: Lorg/apache/hadoop/fs/FileSystem
报错现象: Exception in thread "main" java.lang.NoClassDefFoundError: Lorg/apache/hadoop/fs/Fil ...
- Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/CanUnbuffer
在执行spark on hive 的时候在 sql.show()处报错 : Exception in thread "main" java.lang.NoClassDefFoun ...
- hive启动时报错 java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at org.apache.hadoop.fs.Path.initialize
错误提示信息如下 错误信息如下 [root@node1 bin]# ./hive Logging initialized -bin/lib/hive-common-.jar!/hive-log4j.p ...
- org.apache.hadoop.fs.FsUrlStreamHandlerFactory 在哪个jar包
org.apache.hadoop.fs.FsUrlStreamHandlerFactory在org.apache.hadoop类中,org.apache.hadoop在hadoop安装目录下.
- sparkOnYarn报错org.apache.hadoop.fs.FSDataInputStream
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInpu ...
- Hadoop 3.1.2报错:xception in thread "main" org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
报错内容如下: Exception in thread "main" org.apache.hadoop.fs.UnsupportedFileSystemException: No ...
- 你遇到了吗?Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException)
我在使用 Structured Streaming 的 ForeachWriter,写 HDFS 文件时,出现了这个异常 这个异常出现的原因是HDFS作为一个分布式文件系统,支持多线程读,但是不支持多 ...
- 错误Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream排查思路
spark1(默认CDH自带版本)不存在这个问题,主要是升级了spark2(CDHparcel升级)版本安装后需要依赖到spark1的旧配置去读取hadoop集群的依赖包. 1./etc/spark2 ...
随机推荐
- ORA-12162: TNS:net service name is incorrectly specified
今天在进行修改oracle_sid环境变量的时候,将相关的环境变量值去掉,从而不能进入sqlplus,报错如下: [oracle@kel ~]$ sqlplus / as sysdba SQL*Plu ...
- 杂谈:HTML 5页面可视性API
译文来源:http://www.ido321.com/1126.html 原文:HTML5 Page Visibility API 译文:HTML 5的页面可视性API 译者:dwqs 在早期,浏览器 ...
- 常用的CSS Hack技术集锦
来源:http://www.ido321.com/938.html 一.什么是CSS Hack? 不同的浏览器对CSS的解析结果是不同的,因此会导致相同的CSS输出的页面效果不同,这就需要CSS Ha ...
- bzoj 1778 [Usaco2010 Hol]Dotp 驱逐猪猡(高斯消元)
[题意] 炸弹从1开始运动,每次有P/Q的概率爆炸,否则等概率沿边移动,问在每个城市爆炸的概率. [思路] 设M表示移动一次后i->j的概率.Mk为移动k次后的概率,则有: Mk=M^k 设S= ...
- php 正则表达
今天看书,看到代码里面出现了一段正则表达式匹配语句preg_match,感觉水很深的感觉,网上搜了一些资料,暂时没时间学习,但是觉得以后学的话有两个网址比较靠谱,如下: php正则表达式手册:php ...
- cocosbuilder学习汇总
目前与cocos2d-x-2.14版本对应的cocosbuilder版本为cocosbuilder-3,目前为alpha-5.稳定版本为cocosbuilder2.1,但与cocos2d-x不匹配(C ...
- InputFormat,OutputFormat,InputSplit,RecordRead(一些常见面试题),使用yum安装64位Mysql
列举出hadoop常用的一些InputFormat InputFormat是用来对我们的输入数据进行格式化的.TextInputFormat是默认的. InputFormat有哪些类型? DBInpu ...
- Uva 208 - Firetruck
[题目链接]http://uva.onlinejudge.org/index.php?option=com_onlinejudge&Itemid=8&category=24&p ...
- [置顶] Kendo UI开发教程: Kendo UI 示例及总结
前面基本介绍完Kendo UI开发的基本概念和开发步骤,Kendo UI的示例网站为http://demos.kendoui.com/ ,包含了三个部分 Web DemoMobile DemoData ...
- 如何在不装ORACLE的情况下使用PLSQL
原来我电脑装了oracle跟plsql,然后使用plsql的.后来因为某些原因,我重装了系统,把装的软件都格调了,需要重新装.当时在装plsql的时候我就想,我一直都是直接用plsql远程连接的服务器 ...