Frame of Reference and Roaring Bitmaps
https://www.elastic.co/cn/blog/frame-of-reference-and-roaring-bitmaps
2015年2月18日Engineering
Frame of Reference and Roaring Bitmaps
Postings lists
While it may surprise you if you are new to search engine internals, one of the most important building blocks of a search engine is the ability to efficiently compress and quickly decode sorted lists of integers. Why is this useful? As you may know, Elasticsearch shards, which are Lucene indices under the hood, split the data that they store into segments which are regularly merged together. Inside each segment, documents are given an identifier between 0 and the number of documents in the segment (up to 231-1). This is conceptually like an index in an array: it is stored nowhere but is enough to identity an item. Segments store data about documents sequentially, and a doc ID is the index of a document in a segment. So the first document in a segment would have a doc ID of 0, the second 1, etc. until the last document, which has a doc ID equal to the total number of documents in the segment minus one.
Why are these doc IDs useful? An inverted index needs to map terms to the list of documents that contain this term, called a postings list, and these doc IDs that we just discussed are a perfect fit since they can be compressed efficiently.
Frame Of Reference
In order to be able to compute intersections and unions efficiently, we require that these postings lists are sorted. A nice side-effect of this decision is that postings lists can be compressed with delta-encoding.
For instance, if your postings list is [73, 300, 302, 332, 343, 372]
, the list of deltas would be [73, 227, 2, 30, 11, 29]
. What is interesting to note here is that all deltas are between 0 and 255, so you only need one byte per value. This is the technique that Lucene is using in order to encode your inverted index on disk: postings lists are split into blocks of 256 doc IDs and then each block is compressed separately using delta-encoding and bit packing: Lucene computes the maximum number of bits required to store deltas in a block, adds this information to the block header, and then encodes all deltas of the block using this number of bits. This encoding technique is known as Frame Of Reference (FOR) in the literature and has been used since Lucene 4.1.
Here is an example with a block size of 3 (instead of 256 in practice):
Frame of Reference and Roaring Bitmaps的更多相关文章
- OD: Register, Stack Frame, Function Reference
几个重要的 Win32 寄存器 EIP 指令寄存器(Extended Instruction Pointer) 存放一个指针,指向下一条等待执行的指令地址 ESP 栈指针寄存器(Extended St ...
- Elasticsearch 通关教程(七): Elasticsearch 的性能优化
硬件选择 Elasticsearch(后文简称 ES)的基础是 Lucene,所有的索引和文档数据是存储在本地的磁盘中,具体的路径可在 ES 的配置文件../config/elasticsearch. ...
- Elasticsearch 技术分析(九):Elasticsearch的使用和原理总结
前言 之前已经分享过Elasticsearch的使用和原理的知识,由于近期在公司内部做了一次内部分享,所以本篇主要是基于之前的博文的一个总结,希望通过这篇文章能让读者大致了解Elasticsearch ...
- 全文搜索引擎Elasticsearch详细介绍
我们生活中的数据总体分为两种:结构化数据 和 非结构化数据. 结构化数据:也称作行数据,是由二维表结构来逻辑表达和实现的数据,严格地遵循数据格式与长度规范,主要通过关系型数据库进行存储和管理.指具有固 ...
- L ==> E · L · K
三剑客:Elastic Stack 在学习ELK前,先对 Lucene作基本了解. 今天才知道关系型数据库的索引是 B-Tree,罪过... 减少磁盘寻道次数 ---> 提高查询性能 Lucen ...
- 带你走进神一样的Elasticsearch索引机制
更多精彩内容请看我的个人博客 前言 相比于大多数人熟悉的MySQL数据库的索引,Elasticsearch的索引机制是完全不同于MySQL的B+Tree结构.索引会被压缩放入内存用于加速搜索过程,这一 ...
- Busting Frame Busting: a Study of Clickjacking Vulnerabilities on Popular Sites
Busting Frame Busting Reference From: http://seclab.stanford.edu/websec/framebusting/framebust.pdf T ...
- Frames of Reference参考框架
Frames of Reference参考框架 When describing the position and orientation of something (for example, your ...
- Elasticsearch索引原理
转载 http://blog.csdn.net/endlu/article/details/51720299 最近在参与一个基于Elasticsearch作为底层数据框架提供大数据量(亿级)的实时统计 ...
随机推荐
- Visual Studio 默认git拉取Github出错 No error could not read Username for 'https://github.com': terminal prompts disabled
发布到远程存储库时遇到错误: Git failed with a fatal error.fatal: HttpRequestException encountered. ��������ʱ��� ...
- [从源码学设计]蚂蚁金服SOFARegistry之Data节点变更
[从源码学设计]蚂蚁金服SOFARegistry之Data节点变更 目录 [从源码学设计]蚂蚁金服SOFARegistry之Data节点变更 0x00 摘要 0x02 引子 0x03 业务范畴 3.1 ...
- [论文阅读笔记] metapath2vec: Scalable Representation Learning for Heterogeneous Networks
[论文阅读笔记] metapath2vec: Scalable Representation Learning for Heterogeneous Networks 本文结构 解决问题 主要贡献 算法 ...
- java中如何实现同一账号不能同时登录
经过两天的研究,下面给两个方法.不个是webwork版本的,一个是修改过后的网上的意见监听器版本的 (一) 首先先上自己的研究成果 1:首先在baseAction 中或者直接在action中写一个方法 ...
- mysql字符串拼接
逗号分隔拼接字符串 SELECT group_concat(USER_ID)from rocky_bankinfo 默认大小2014 2).可以简单一点,执行语句,可以设置作用范围 ...
- Mapreduce实例--求平均值
求平均数是MapReduce比较常见的算法,求平均数的算法也比较简单,一种思路是Map端读取数据,在数据输入到Reduce之前先经过shuffle,将map函数输出的key值相同的所有的value值形 ...
- Java Object to Class
User user = new User(username,password); //添加构成JWT的参数 JwtBuilder builder = Jwts.builder().setHeaderP ...
- Nginx配置请求头
最近发现一个问题: IOS访问后台接口是,总是application/json;charset=utf-8 但是后台接口只支持大写的UTF-8,修改了Nginx的请求头之后正常. proxy_set_ ...
- 图片质量评估论文 | 无监督SER-FIQ | CVPR2020
文章转自:同作者微信公主号[机器学习炼丹术].欢迎交流,共同进步. 论文名称:SER-FIQ: Unsupervised Estimation of Face Image Quality Based ...
- 八:架构,搭建,WAF
WAF防护分析 什么是WAF应用 如何快速识别WAF 识别WAF对于安全测试的意义 CMS识别技术 源码获取技术 架构信息获取 站点搭建分析 搭建习惯-目录型站点 sti.blcu-bbs 目录型站点 ...