public final class Lucene54DocValuesFormat
extends DocValuesFormat
Lucene 5.4 DocValues format.

Encodes the five per-document value types (Numeric,Binary,Sorted,SortedSet,SortedNumeric) with these strategies:

NUMERIC:

  • Delta-compressed: per-document integers written as deltas from the minimum value, compressed with bitpacking. For more information, see DirectWriter.
  • Table-compressed: when the number of unique values is very small (< 256), and when there are unused "gaps" in the range of values used (such as SmallFloat), a lookup table is written instead. Each per-document entry is instead the ordinal to this table, and those ordinals are compressed with bitpacking (DirectWriter).
  • GCD-compressed: when all numbers share a common divisor, such as dates, the greatest common denominator (GCD) is computed, and quotients are stored using Delta-compressed Numerics.
  • Monotonic-compressed: when all numbers are monotonically increasing offsets, they are written as blocks of bitpacked integers, encoding the deviation from the expected delta.
  • Const-compressed: when there is only one possible non-missing value, only the missing bitset is encoded.
  • Sparse-compressed: only documents with a value are stored, and lookups are performed using binary search.

BINARY:

  • Fixed-width Binary: one large concatenated byte[] is written, along with the fixed length. Each document's value can be addressed directly with multiplication (docID * length).
  • Variable-width Binary: one large concatenated byte[] is written, along with end addresses for each document. The addresses are written as Monotonic-compressed numerics.
  • Prefix-compressed Binary: values are written in chunks of 16, with the first value written completely and other values sharing prefixes. chunk addresses are written as Monotonic-compressed numerics. A reverse lookup index is written from a portion of every 1024th term.

SORTED:

  • Sorted: a mapping of ordinals to deduplicated terms is written as Binary, along with the per-document ordinals written using one of the numeric strategies above.

SORTED_SET:

  • Single: if all documents have 0 or 1 value, then data are written like SORTED.
  • SortedSet table: when there are few unique sets of values (< 256) then each set is assigned an id, a lookup table is written and the mapping from document to set id is written using the numeric strategies above.
  • SortedSet: a mapping of ordinals to deduplicated terms is written as Binary, an ordinal list and per-document index into this list are written using the numeric strategies above.

SORTED_NUMERIC:

  • Single: if all documents have 0 or 1 value, then data are written like NUMERIC.
  • SortedSet table: when there are few unique sets of values (< 256) then each set is assigned an id, a lookup table is written and the mapping from document to set id is written using the numeric strategies above.
  • SortedNumeric: a value list and per-document index into this list are written using the numeric strategies above.

Files:

  1. .dvd: DocValues data
  2. .dvm: DocValues metadata

转自:http://lucene.apache.org/core/6_4_2/core/org/apache/lucene/codecs/lucene54/Lucene54DocValuesFormat.html

可以看到占用空间非常小!!!

du -sm elasticsearch/nodes/0/indices/hec_test2/0/index/*
299 elasticsearch/nodes/0/indices/hec_test2/0/index/_e.fdt
1 elasticsearch/nodes/0/indices/hec_test2/0/index/_e.fdx
1 elasticsearch/nodes/0/indices/hec_test2/0/index/_e.fnm
148 elasticsearch/nodes/0/indices/hec_test2/0/index/_e_Lucene50_0.doc
130 elasticsearch/nodes/0/indices/hec_test2/0/index/_e_Lucene50_0.tim
5 elasticsearch/nodes/0/indices/hec_test2/0/index/_e_Lucene50_0.tip
1 elasticsearch/nodes/0/indices/hec_test2/0/index/_e_Lucene54_0.dvd
1 elasticsearch/nodes/0/indices/hec_test2/0/index/_e_Lucene54_0.dvm
1 elasticsearch/nodes/0/indices/hec_test2/0/index/_e.si
1 elasticsearch/nodes/0/indices/hec_test2/0/index/segments_7
0 elasticsearch/nodes/0/indices/hec_test2/0/index/write.lock

Lucene dvd dvm文件便是docvalues文件——就是针对field value的列存储的更多相关文章

  1. 腾讯Hermes设计概要——数据分析用的是列存储,词典文件前缀压缩,倒排文件递增id、变长压缩、依然是跳表-本质是lucene啊

    转自:http://data.qq.com/article?id=817 三.Hermes设计概要 架构描述 系统核心进程均采用分散化设计,根据业务发展需求,可随意扩缩容机器; 周期性数据直接通过td ...

  2. FileShare文件读写锁解决“文件XXX正由另一进程使用,因此该进程无法访问此文件”(转)

    开发过程中,我们往往需要大量与文件交互,读文件,写文件已成家常便饭,本地运行完美,但一上到投产环境,往往会出现很多令人措手不及的意外,或开发中的烦恼,因此,我对普通的C#文件操作做了一次总结,问题大部 ...

  3. .c和.h文件的区别(头文件与之实现文件的的关系~ )

     .c和.h文件的区别 一个简单的问题:.c和.h文件的区别 学了几个月的C语言,反而觉得越来越不懂了.同样是子程序,可以定义在.c文件中,也可以定义在.h文件中,那这两个文件到底在用法上有什么区别呢 ...

  4. [转载]webarchive文件转换成htm文件

    原文地址:webarchive文件转换成htm文件作者:xhbaxf Mac OS X系统带有文件转换功能,可以把webarchive文件变成html文件.方法是:   Step 1: 建立一个文件夹 ...

  5. 怎样将word文件转化为Latex文件:word-to-latex-2.56具体解释

    首先推荐大家读一读这篇博文:http://blog.csdn.net/ibingow/article/details/8613556 --------------------------------- ...

  6. PHP上传文件参考配置大文件上传

    PHP用超级全局变量数组$_FILES来记录文件上传相关信息的. 1.file_uploads=on/off 是否允许通过http方式上传文件 2.max_execution_time=30 允许脚本 ...

  7. R8—批量生成文件夹,批量读取文件夹名称+R文件管理系统操作函数

    一. 批量生成文件夹,批量读取文件夹名称 今日,工作中遇到这样一个问题:boss给我们提供了200多家公司的ID代码(如6007.7920等),需要根据这些ID号去搜索下载新闻,从而将下载到的新闻存到 ...

  8. Python:文件操作总结1——文件基本操作

    一.文件的操作流程 1.打开文件,得到文件句柄并赋值给一个变量 2.通过句柄对文件进行操作 3.关闭文件 二.文件的打开与关闭 A.文件的打开——open函数 语法:open(file[,mode[, ...

  9. linux下压缩成zip文件解压zip文件

    linux  zip命令的基本用法是: zip [参数] [打包后的文件名] [打包的目录路径] linux  zip命令参数列表: -a     将文件转成ASCII模式 -F     尝试修复损坏 ...

随机推荐

  1. 三、Oracle常用内置函数

    1. ASCII  返回与指定的字符对应的十进制数;  SQL> select ascii(A) A,ascii(a) a,ascii(0) zero,ascii( ) space from d ...

  2. python3--算法基础:二分查找/折半查找

    算法基础:二分查找/折半查找 #!/usr/bin/env python # -*- coding:utf-8 -*- # 算法基础:二分查找/折半查找 def binarySearch(dataSo ...

  3. JavaEE JDBC 了解数据库连接池

    了解数据库连接池 @author ixenos 数据库连接是有限的资源,如果用户需要离开应用一段时间,那么他占用的连接就不应该保持开放状态: 另一方面,每次查询都获取连接并在随后关闭它的代价也很高. ...

  4. kubernetes集群新增node

    kubernetes集群要新增node,首先要配置ssh免密登陆 root@ht:/etc/ansible# ssh-copy-id 172.18.196.6 /usr/bin/ssh-copy-id ...

  5. Bzoj 2726 SDOI 任务安排

      Memory Limit: 131072KB   64bit IO Format: %lld & %llu Description 机器上有N个需要处理的任务,它们构成了一个序列.这些任务 ...

  6. 洛谷P3406 海底高铁

    题目背景 大东亚海底隧道连接着厦门.新北.博艾.那霸.鹿儿岛等城市,横穿东海,耗资1000亿博艾元,历时15年,于公元2058年建成.凭借该隧道,从厦门可以乘坐火车直达台湾.博艾和日本,全程只需要4个 ...

  7. C++函数的重载,覆盖和隐藏(——高质量编程第14章)

      函数重载概念 只有C++才有重载的概念,C语言没有. 靠行参列表的不同来区别不同的重载函数, 若是全局函数和成员函数同名时,不算重载,因为它们的作用域不同,所以成员函数会将全局函数给隐藏(遮蔽)了 ...

  8. python学习之 - configparser模块

    configparser模块功能:用于生成和修改常见配置文件.基本常用方法如下: read(filename):直接读取配置文件write(filename):将修改后的配置文件写入文件中.defau ...

  9. Log4j的日志级别分析(转)

    说明:Log4j的日志是有级别的,从低到高顺序为:ALL < DEBUG < INFO < WARN < ERROR < FATAL < OFF,当定义了日志级别为 ...

  10. babel 用法及其 .babelrc 的配置详解,想做前端架构,拒绝一知半解...

    Babel 官方介绍:将 ECMAScript 2015 及其版本以后的 javascript 代码转为旧版本浏览器或者是环境中向后兼容版本的  javascript 代码. 简而言之,就是把不兼容的 ...