hbase 分页过滤（新老API的差别）

在hbase2.0以前分页过滤必须以上一次的最后一行+空字节数组作为下一次的起始行，

因为scan扫描的时候是包含起始行的，为了既能准确定位起始行，但又不重复把上一次的最末一行加入下一页，

所以，权威指南里才有了加上空字节数组的处理。

hbase2.0以后，新的api是withStartRow(byte[] startRow, boolean inclusive)，可以直接设置是否包含起始行，完美解决问题，但是又保留了对以前api函数的兼容性。

    //分页过滤

    private static void pageFilterData() throws IOException{

        Table table = helper.getConnection().getTable(TableName.valueOf("testtable"));

        final byte[] POSTFIX = new byte[] { 0x00 };

        Filter filter = new PageFilter(10);

        int totalRows = 0;

        byte[] lastRow = null;

        while(true){

            Scan scan = new Scan();

            scan.setFilter(filter);

            if(lastRow!=null){

                  //为了兼容以前的scan.setStartRow()代码

                  //在上一次的最后一行加上一个空的byte数据，在下一个分页上，就会以新的key开始，

                // 但是实际上这个key并不存在，所以还是从真正的下一行开始扫描

                //这么做的原因是scan的扫描会自动包含起始行，如果不加空字节数据，那么定位上就会把上一次的最后一行作为起始行，最后的数据就会多一行。

                //而，新的api是withStartRow(byte[] startRow, boolean inclusive)，可以直接设置是否包含起始行，完美解决问题，但是又保留了对

                //以前api函数的兼容性

//                byte[] startRow = Bytes.add(lastRow,POSTFIX);

//                System.out.println("start row: " + Bytes.toStringBinary(startRow));

//                scan.withStartRow(startRow,true);

                System.out.println("start row: " + Bytes.toStringBinary(lastRow));

                //不包含起始行，所以可以直接使用上一次的最后一行作为起始行

                scan.withStartRow(lastRow,false);

            }

            ResultScanner scanner = table.getScanner(scan);

            int localRows = 0;

            Result result;

            while ((result=scanner.next())!=null){

                System.out.println(localRows++ + ": " + result);

                totalRows++;

                lastRow = result.getRow();

            }

            scanner.close();

            if(localRows==0)break;

        }

        System.out.println("total rows: " + totalRows);

    }

兼容老API输出如下：

0: keyvalues={rowKey0/ex:addr/1555078771906/Put/vlen=7/seqid=0, rowKey0/info:username/1555078771906/Put/vlen=5/seqid=0, rowKey0/memo:detail/1555078771906/Put/vlen=7/seqid=0}

1: keyvalues={rowKey1/ex:addr/1555078771906/Put/vlen=7/seqid=0, rowKey1/info:username/1555078771906/Put/vlen=5/seqid=0, rowKey1/memo:detail/1555078771906/Put/vlen=7/seqid=0}

2: keyvalues={rowKey10/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey10/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey10/memo:detail/1555078771906/Put/vlen=8/seqid=0}

3: keyvalues={rowKey11/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey11/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey11/memo:detail/1555078771906/Put/vlen=8/seqid=0}

4: keyvalues={rowKey12/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey12/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey12/memo:detail/1555078771906/Put/vlen=8/seqid=0}

5: keyvalues={rowKey13/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey13/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey13/memo:detail/1555078771906/Put/vlen=8/seqid=0}

6: keyvalues={rowKey14/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey14/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey14/memo:detail/1555078771906/Put/vlen=8/seqid=0}

7: keyvalues={rowKey15/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey15/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey15/memo:detail/1555078771906/Put/vlen=8/seqid=0}

8: keyvalues={rowKey16/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey16/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey16/memo:detail/1555078771906/Put/vlen=8/seqid=0}

9: keyvalues={rowKey17/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey17/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey17/memo:detail/1555078771906/Put/vlen=8/seqid=0}

start row: rowKey17\x00

0: keyvalues={rowKey18/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey18/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey18/memo:detail/1555078771906/Put/vlen=8/seqid=0}

1: keyvalues={rowKey19/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey19/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey19/memo:detail/1555078771906/Put/vlen=8/seqid=0}

2: keyvalues={rowKey2/ex:addr/1555078771906/Put/vlen=7/seqid=0, rowKey2/info:username/1555078771906/Put/vlen=5/seqid=0, rowKey2/memo:detail/1555078771906/Put/vlen=7/seqid=0}

3: keyvalues={rowKey20/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey20/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey20/memo:detail/1555078771906/Put/vlen=8/seqid=0}

4: keyvalues={rowKey21/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey21/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey21/memo:detail/1555078771906/Put/vlen=8/seqid=0}

5: keyvalues={rowKey22/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey22/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey22/memo:detail/1555078771906/Put/vlen=8/seqid=0}

6: keyvalues={rowKey23/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey23/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey23/memo:detail/1555078771906/Put/vlen=8/seqid=0}

7: keyvalues={rowKey24/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey24/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey24/memo:detail/1555078771906/Put/vlen=8/seqid=0}

8: keyvalues={rowKey25/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey25/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey25/memo:detail/1555078771906/Put/vlen=8/seqid=0}

9: keyvalues={rowKey26/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey26/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey26/memo:detail/1555078771906/Put/vlen=8/seqid=0}

start row: rowKey26\x00

0: keyvalues={rowKey27/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey27/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey27/memo:detail/1555078771906/Put/vlen=8/seqid=0}

1: keyvalues={rowKey28/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey28/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey28/memo:detail/1555078771906/Put/vlen=8/seqid=0}

2: keyvalues={rowKey29/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey29/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey29/memo:detail/1555078771906/Put/vlen=8/seqid=0}

3: keyvalues={rowKey3/ex:addr/1555078771906/Put/vlen=7/seqid=0, rowKey3/info:username/1555078771906/Put/vlen=5/seqid=0, rowKey3/memo:detail/1555078771906/Put/vlen=7/seqid=0}

4: keyvalues={rowKey30/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey30/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey30/memo:detail/1555078771906/Put/vlen=8/seqid=0}

5: keyvalues={rowKey31/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey31/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey31/memo:detail/1555078771906/Put/vlen=8/seqid=0}

6: keyvalues={rowKey32/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey32/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey32/memo:detail/1555078771906/Put/vlen=8/seqid=0}

7: keyvalues={rowKey33/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey33/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey33/memo:detail/1555078771906/Put/vlen=8/seqid=0}

8: keyvalues={rowKey34/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey34/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey34/memo:detail/1555078771906/Put/vlen=8/seqid=0}

9: keyvalues={rowKey35/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey35/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey35/memo:detail/1555078771906/Put/vlen=8/seqid=0}

start row: rowKey35\x00

... ...省略其他数据

起始行加了空字节数据，形成了新的定位行（相当于真正的两页之间插入了一个虚拟行用于定位），解决了下一页首行的问题。

新的api输出：下一页的定位行就是上一页的末行，采用新的API设置首行的时候把inclusive设置成false（即不包含首行），那么输出结果就是正确的。

0: keyvalues={rowKey0/ex:addr/1555078771906/Put/vlen=7/seqid=0, rowKey0/info:username/1555078771906/Put/vlen=5/seqid=0, rowKey0/memo:detail/1555078771906/Put/vlen=7/seqid=0}

1: keyvalues={rowKey1/ex:addr/1555078771906/Put/vlen=7/seqid=0, rowKey1/info:username/1555078771906/Put/vlen=5/seqid=0, rowKey1/memo:detail/1555078771906/Put/vlen=7/seqid=0}

2: keyvalues={rowKey10/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey10/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey10/memo:detail/1555078771906/Put/vlen=8/seqid=0}

3: keyvalues={rowKey11/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey11/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey11/memo:detail/1555078771906/Put/vlen=8/seqid=0}

4: keyvalues={rowKey12/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey12/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey12/memo:detail/1555078771906/Put/vlen=8/seqid=0}

5: keyvalues={rowKey13/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey13/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey13/memo:detail/1555078771906/Put/vlen=8/seqid=0}

6: keyvalues={rowKey14/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey14/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey14/memo:detail/1555078771906/Put/vlen=8/seqid=0}

7: keyvalues={rowKey15/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey15/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey15/memo:detail/1555078771906/Put/vlen=8/seqid=0}

8: keyvalues={rowKey16/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey16/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey16/memo:detail/1555078771906/Put/vlen=8/seqid=0}

9: keyvalues={rowKey17/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey17/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey17/memo:detail/1555078771906/Put/vlen=8/seqid=0}

start row: rowKey17

0: keyvalues={rowKey18/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey18/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey18/memo:detail/1555078771906/Put/vlen=8/seqid=0}

1: keyvalues={rowKey19/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey19/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey19/memo:detail/1555078771906/Put/vlen=8/seqid=0}

2: keyvalues={rowKey2/ex:addr/1555078771906/Put/vlen=7/seqid=0, rowKey2/info:username/1555078771906/Put/vlen=5/seqid=0, rowKey2/memo:detail/1555078771906/Put/vlen=7/seqid=0}

3: keyvalues={rowKey20/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey20/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey20/memo:detail/1555078771906/Put/vlen=8/seqid=0}

4: keyvalues={rowKey21/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey21/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey21/memo:detail/1555078771906/Put/vlen=8/seqid=0}

5: keyvalues={rowKey22/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey22/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey22/memo:detail/1555078771906/Put/vlen=8/seqid=0}

6: keyvalues={rowKey23/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey23/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey23/memo:detail/1555078771906/Put/vlen=8/seqid=0}

7: keyvalues={rowKey24/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey24/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey24/memo:detail/1555078771906/Put/vlen=8/seqid=0}

8: keyvalues={rowKey25/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey25/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey25/memo:detail/1555078771906/Put/vlen=8/seqid=0}

9: keyvalues={rowKey26/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey26/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey26/memo:detail/1555078771906/Put/vlen=8/seqid=0}

start row: rowKey26

0: keyvalues={rowKey27/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey27/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey27/memo:detail/1555078771906/Put/vlen=8/seqid=0}

1: keyvalues={rowKey28/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey28/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey28/memo:detail/1555078771906/Put/vlen=8/seqid=0}

2: keyvalues={rowKey29/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey29/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey29/memo:detail/1555078771906/Put/vlen=8/seqid=0}

3: keyvalues={rowKey3/ex:addr/1555078771906/Put/vlen=7/seqid=0, rowKey3/info:username/1555078771906/Put/vlen=5/seqid=0, rowKey3/memo:detail/1555078771906/Put/vlen=7/seqid=0}

4: keyvalues={rowKey30/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey30/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey30/memo:detail/1555078771906/Put/vlen=8/seqid=0}

5: keyvalues={rowKey31/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey31/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey31/memo:detail/1555078771906/Put/vlen=8/seqid=0}

6: keyvalues={rowKey32/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey32/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey32/memo:detail/1555078771906/Put/vlen=8/seqid=0}

7: keyvalues={rowKey33/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey33/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey33/memo:detail/1555078771906/Put/vlen=8/seqid=0}

8: keyvalues={rowKey34/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey34/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey34/memo:detail/1555078771906/Put/vlen=8/seqid=0}

9: keyvalues={rowKey35/ex:addr/1555078771906/Put/vlen=8/seqid=0, rowKey35/info:username/1555078771906/Put/vlen=6/seqid=0, rowKey35/memo:detail/1555078771906/Put/vlen=8/seqid=0}

start row: rowKey35

... ...其他省略

hbase 分页过滤（新老API的差别）的更多相关文章

Hadoop2.2编程：新旧API的区别
Hadoop最新版本的MapReduce Release 0.20.0的API包括了一个全新的Mapreduce JAVA API,有时候也称为上下文对象. 新的API类型上不兼容以前的API,所以, ...
Android 新老两代 Camera API 大起底
https://blog.csdn.net/Byeweiyang/article/details/80515192 0.背景简介最近有一部分相机相关的需求,专注于对拍摄的照片.视频的噪点.色温.明暗 ...
Java 8的新并行API - 魅力与炫目背后
这是一篇译文,原文链接见这里. 本文同时发表在ImportNew上,转载请注明出处. 我很擅长同时处理多项任务.就算是在写这篇博客的此刻,我仍然在为昨天在聚会上发表了一个让大家都感到诧异的评论而觉得尴 ...
hbase分页查询
为了广大技术爱好者学习netty,在这里帮新浪微博@nettying宣传下他出版的新书 <netty权威指南>@nettying兄在华为NIO实践多年,这本书是他的技术和经验的一个结晶.N ...
Hadoop日记Day15---MapReduce新旧api的比较
我使用hadoop的是hadoop1.1.2,而很多公司也在使用hadoop0.2x版本,因此市面上的hadoop资料版本不一,为了扩充自己的知识面,MapReduce的新旧api进行了比较研究. h ...
hbase分页应用场景及分页思路与代码实现
转自:http://www.aboutyun.com/forum.php?mod=viewthread&tid=7030&extra=page=1 可以带着下面问题来阅读1.hbase ...
Kafka 0.9 新消费者API
kafka诞生之初,它自带一个基于scala的生产者和消费者客户端.但是慢慢的我们认识到这些API有很多限制.比如,消费者有一个“高级”API支持分组和异常控制,但是不支持很多更复杂的应用场景:它也有 ...
MapReduce简述、工作流程及新旧API对照
什么是MapReduce? 你想数出一摞牌中有多少张黑桃.直观方式是一张一张检查而且数出有多少张是黑桃. MapReduce方法则是: 1. 给在座的全部玩家中分配这摞牌. 2. 让每一个玩家数自己手 ...
Hbase 分页设计
hbase 数据获取方式直接根据 rowkey 查找,速度最快 scan,指定 startrowkey.endrowkey 和 limit获取数据,在 rowkey 设计良好的情况下,效率也不错全 ...

随机推荐

Python学习过程笔记整理（二）
程序三大结构 -顺序 -分支 -循环分支 -语法: if 条件表达式: 语句 ... -双向分支 if 条件表达式: 语句 ... else: 语句 ... -多路分支 if 条件表达式: 语句 . ...
《Redis设计与实现》阅读笔记（四）--字典
字典字典,map,是用于保存键值对的抽象数据结构,是hash表实现.字典中的键唯一,通过键来操作值.Redis的数据库使用字典来作为底层实现. 定义 Redis的字典使用哈希表作为底层实现,一个哈希 ...
Hadoop初步简介
Hadoop产生背景: 传统方式,我们使用数据库来对数据进行管理.可是随着数据量的增加,我们要对这个数据库中的海量数据进行处理, 从中提取出有效的信息,这时候面临的问题随之而来: 1.海量数据读取,采 ...
啥是MD5？
啥是MD5加密呢?为啥要使用MD5这种非对称的加密方式呢? 本文将通过漫画的形式来通俗易懂的讲述什么是MD5加密算法(Message Digest Algorithm MD5(中文名为消息摘要算法第五 ...
PHP核心技术——异常和错误处理
PHP只有手动抛出异常后才能捕获异常 $a = null; try { $a = 5/0; echo $a,PHP_EOL; } catch (exception $e) { $e -> get ...
sklearn 中的 Pipeline 机制
转载自:https://blog.csdn.net/lanchunhui/article/details/50521648 from sklearn.pipeline import Pipeline ...
Gaussian Models
Warming Up Before we talk about multivariate Gaussian, let's first review univariate Gaussian, which ...
IOS statusBarStyle 设置
在项目info.plist文件中有 View controller-based status bar appearance 属性. 当设置为NO时通过 [UIApplication sharedAp ...
centos下部署禅道流程
原文摘录:https://www.jianshu.com/p/71e9dab130a5 下面将我在Linux系统下搭建禅道服务的过程分享给大家. 第一步:下载禅道 Linux中可以用以下命令来下载安装 ...
python下graphviz安装
参考链接:https://blog.csdn.net/u013250416/article/details/72790754 1.安装Graphviz 在graphviz的官网(网址:http://w ...

hbase 分页过滤（新老API的差别）

hbase 分页过滤（新老API的差别）的更多相关文章

随机推荐

热门专题