PostgreSQL 传统 hash 分区方法和性能
背景
除了传统的基于trigger和rule的分区,PostgreSQL 10开始已经内置了分区功能(目前仅支持list和range),使用pg_pathman则支持hash分区。
从性能角度,目前最好的还是pg_pathman分区。
但是,传统的分区手段,依旧是最灵活的,在其他方法都不奏效时,可以考虑传统方法。
如何创建传统的hash分区
1、创建父表
create table tbl (id int, info text, crt_time timestamp);
2、创建分区表,增加约束
do language plpgsql $$
declare
parts int := 4;
begin
for i in 0..parts-1 loop
execute format('create table tbl%s (like tbl including all) inherits (tbl)', i);
execute format('alter table tbl%s add constraint ck check(mod(id,%s)=%s)', i, parts, i);
end loop;
end;
$$;
3、创建触发器函数,内容为数据路由,路由后返回NULL(即不写本地父表)
create or replace function ins_tbl() returns trigger as $$
declare
begin
case abs(mod(NEW.id,4))
when 0 then
insert into tbl0 values (NEW.*);
when 1 then
insert into tbl1 values (NEW.*);
when 2 then
insert into tbl2 values (NEW.*);
when 3 then
insert into tbl3 values (NEW.*);
else
return NEW; -- 如果是NULL则写本地父表
end case;
return null;
end;
$$ language plpgsql strict;
4、创建before触发器
create trigger tg1 before insert on tbl for each row when (NEW.id is not null) execute procedure ins_tbl();
5、验证
postgres=# insert into tbl values (1);
INSERT 0 0
postgres=# insert into tbl values (null);
INSERT 0 1
postgres=# insert into tbl values (0);
INSERT 0 0
postgres=# insert into tbl values (1);
INSERT 0 0
postgres=# insert into tbl values (2);
INSERT 0 0
postgres=# insert into tbl values (3);
INSERT 0 0
postgres=# insert into tbl values (4);
INSERT 0 0
postgres=# select tableoid::regclass, * from tbl;
tableoid | id | info | crt_time
----------+----+------+----------
tbl | | |
tbl0 | 0 | |
tbl0 | 4 | |
tbl1 | 1 | |
tbl1 | 1 | |
tbl2 | 2 | |
tbl3 | 3 | |
(7 rows)
6、查询时,只要提供了约束条件,会自动过滤到子表,不会扫描不符合约束条件的其他子表。
postgres=# explain select * from tbl where abs(mod(id,4)) = abs(mod(1,4)) and id=1;
QUERY PLAN
--------------------------------------------------------------------------
Append (cost=0.00..979127.84 rows=3 width=45)
-> Seq Scan on tbl (cost=0.00..840377.67 rows=2 width=45)
Filter: ((id = 1) AND (abs(mod(id, 4)) = 1))
-> Seq Scan on tbl1 (cost=0.00..138750.17 rows=1 width=45)
Filter: ((id = 1) AND (abs(mod(id, 4)) = 1))
(5 rows)
这里应该是错误的,因为如果想利用constraint_exclusion来优化sql,where条件应该尽可能简单,尽量和check约束保持一致,不要转换类型,更谈不上使用函数表达式了,上面实测执行计划是走的全表扫描。后面会列出官方文档中提到的有关分区表和constraint_exclusion参数相关的注意事项。
这里我明白德哥的原意了,因为做的hash分区,取模的数值只有4个且均大于等于0,这里加上绝对值是恰当的,但这个abs应该加到check约束里面,不然constraint_exclusion的优化效果还是用不到的。
下面是实测执行计划及修改条件后的执行计划:
db版本:PostgreSQL 10.1,constraint_exclusion:partition
swrd=# explain select * from tbl where abs(mod(id,4)) = abs(mod(1,4)) and id=1;
QUERY PLAN
------------------------------------------------------------
Append (cost=0.00..133.66 rows=5 width=44)
-> Seq Scan on tbl (cost=0.00..3.26 rows=1 width=44)
Filter: ((id = 1) AND (abs(mod(id, 4)) = 1))
-> Seq Scan on tbl0 (cost=0.00..32.60 rows=1 width=44)
Filter: ((id = 1) AND (abs(mod(id, 4)) = 1))
-> Seq Scan on tbl1 (cost=0.00..32.60 rows=1 width=44)
Filter: ((id = 1) AND (abs(mod(id, 4)) = 1))
-> Seq Scan on tbl2 (cost=0.00..32.60 rows=1 width=44)
Filter: ((id = 1) AND (abs(mod(id, 4)) = 1))
-> Seq Scan on tbl3 (cost=0.00..32.60 rows=1 width=44)
Filter: ((id = 1) AND (abs(mod(id, 4)) = 1))
(11 rows)
修改where条件后的执行计划:
swrd=# explain select * from tbl where mod(id,4) = mod(1,4) and id=1;
QUERY PLAN
------------------------------------------------------------
Append (cost=0.00..32.75 rows=2 width=44)
-> Seq Scan on tbl (cost=0.00..2.98 rows=1 width=44)
Filter: ((id = 1) AND (mod(id, 4) = 1))
-> Seq Scan on tbl1 (cost=0.00..29.78 rows=1 width=44)
Filter: ((id = 1) AND (mod(id, 4) = 1))
(5 rows)
传统分区性能 对比 非分区表
传统分区表性能
性能相比没有分区有一定下降。(CPU开销略有提升)
1、创建压测脚本
vi test.sql
\set id random(1,100000)
insert into tbl values (:id);
2、压测
pgbench -M prepared -n -r -P 1 -f ./test.sql -c 56 -j 56 -T 120
transaction type: ./test.sql
scaling factor: 1
query mode: prepared
number of clients: 56
number of threads: 56
duration: 120 s
number of transactions actually processed: 21277635
latency average = 0.316 ms
latency stddev = 0.170 ms
tps = 177290.033472 (including connections establishing)
tps = 177306.915203 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set id random(1,100000)
0.315 insert into tbl values (:id);
3、资源开销
last pid: 36817; load avg: 32.9, 15.7, 7.27; up 15+00:46:36 17:59:17
63 processes: 34 running, 29 sleeping
CPU states: 42.3% user, 0.0% nice, 20.4% system, 37.1% idle, 0.2% iowait
Memory: 192G used, 29G free, 116M buffers, 186G cached
DB activity: 168654 tps, 0 rollbs/s, 928 buffer r/s, 99 hit%, 176 row r/s, 168649 row w/
DB I/O: 0 reads/s, 0 KB/s, 0 writes/s, 0 KB/s
DB disk: 1455.4 GB total, 425.2 GB free (70% used)
Swap:
未分区表性能
postgres=# drop trigger tg1 on tbl ;
1、TPS
transaction type: ./test.sql
scaling factor: 1
query mode: prepared
number of clients: 56
number of threads: 56
duration: 120 s
number of transactions actually processed: 31188395
latency average = 0.215 ms
latency stddev = 0.261 ms
tps = 259884.798007 (including connections establishing)
tps = 259896.495810 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set id random(1,100000)
0.214 insert into tbl values (:id);
2、资源开销
last pid: 36964; load avg: 31.7, 18.7, 8.89; up 15+00:47:41 18:00:22
63 processes: 45 running, 18 sleeping
CPU states: 33.3% user, 0.0% nice, 26.8% system, 39.8% idle, 0.1% iowait
Memory: 194G used, 26G free, 118M buffers, 188G cached
DB activity: 256543 tps, 0 rollbs/s, 1006 buffer r/s, 99 hit%, 176 row r/s, 256538 row w
DB I/O: 0 reads/s, 0 KB/s, 0 writes/s, 0 KB/s
DB disk: 1455.4 GB total, 424.8 GB free (70% used)
Swap:
非整型字段,如何实现哈希分区
1、PostgreSQL内部提供了类型转换的哈希函数,可以将任意类型转换为整型。
List of functions
Schema | Name | Result data type | Argument data types | Type
------------+----------------+------------------+-----------------------------+--------
pg_catalog | hash_aclitem | integer | aclitem | normal
pg_catalog | hash_array | integer | anyarray | normal
pg_catalog | hash_numeric | integer | numeric | normal
pg_catalog | hash_range | integer | anyrange | normal
pg_catalog | hashbpchar | integer | character | normal
pg_catalog | hashchar | integer | "char" | normal
pg_catalog | hashenum | integer | anyenum | normal
pg_catalog | hashfloat4 | integer | real | normal
pg_catalog | hashfloat8 | integer | double precision | normal
pg_catalog | hashinet | integer | inet | normal
pg_catalog | hashint2 | integer | smallint | normal
pg_catalog | hashint4 | integer | integer | normal
pg_catalog | hashint8 | integer | bigint | normal
pg_catalog | hashmacaddr | integer | macaddr | normal
pg_catalog | hashmacaddr8 | integer | macaddr8 | normal
pg_catalog | hashname | integer | name | normal
pg_catalog | hashoid | integer | oid | normal
pg_catalog | hashoidvector | integer | oidvector | normal
pg_catalog | hashtext | integer | text | normal
pg_catalog | hashvarlena | integer | internal | normal
pg_catalog | interval_hash | integer | interval | normal
pg_catalog | jsonb_hash | integer | jsonb | normal
pg_catalog | pg_lsn_hash | integer | pg_lsn | normal
pg_catalog | time_hash | integer | time without time zone | normal
pg_catalog | timestamp_hash | integer | timestamp without time zone | normal
pg_catalog | timetz_hash | integer | time with time zone | normal
pg_catalog | uuid_hash | integer | uuid | normal
2、其他字段类型的哈希表方法如下
如 hashtext
drop table tbl;
create table tbl (id text, info text, crt_time timestamp);
do language plpgsql $$
declare
parts int := 4;
begin
for i in 0..parts-1 loop
execute format('create table tbl%s (like tbl including all) inherits (tbl)', i);
execute format('alter table tbl%s add constraint ck check(abs(mod(hashtext(id),%s))=%s)', i, parts, i);
end loop;
end;
$$;
create or replace function ins_tbl() returns trigger as $$
declare
begin
case abs(mod(hashtext(NEW.id),4))
when 0 then
insert into tbl0 values (NEW.*);
when 1 then
insert into tbl1 values (NEW.*);
when 2 then
insert into tbl2 values (NEW.*);
when 3 then
insert into tbl3 values (NEW.*);
else
return NEW;
end case;
return null;
end;
$$ language plpgsql strict;
create trigger tg1 before insert on tbl for each row when (NEW.id is not null) execute procedure ins_tbl();
性能与整型一样。
传统分区性能 对比 非分区表 - 性能结果
1、性能
模式 | insert N 行/s |
---|---|
基于trigger的hash分区 | 17.7 万 |
未分区 | 26 万 |
2、CPU资源开销
模式 | user | system | idle |
---|---|---|---|
基于trigger的hash分区 | 42.3% | 20.4% | 37.1% |
未分区 | 33.3% | 26.8% | 39.8% |
小结
除了传统的基于trigger和rule的分区,PostgreSQL 10开始已经内置了分区功能(目前仅支持list和range),使用pg_pathman则支持hash分区。
从性能角度,目前最好的还是pg_pathman分区。
《PostgreSQL 10 内置分区 vs pg_pathman perf profiling》
《PostgreSQL 10.0 preview 功能增强 - 内置分区表》
《PostgreSQL 9.5+ 高效分区表实现 - pg_pathman》
但是,传统的分区手段,依旧是最灵活的,在其他方法都不奏效时,可以考虑传统方法。
传统手段中,最懒散的做法(当然是以牺牲性能为前提),例子:
《PostgreSQL general public partition table trigger》
下面则是pg10官方文档中提到的有关分区表和有关参数constraint_exclusion的相关注意事项:
The following caveats apply to constraint exclusion, which is used by both inheritance and partitioned tables:
Constraint exclusion only works when the query's WHERE clause contains constants (or externally supplied parameters). For example, a comparison against a non-immutable function such as CURRENT_TIMESTAMP cannot be optimized, since the planner cannot know which partition the function value might fall into at run time.
Keep the partitioning constraints simple, else the planner may not be able to prove that partitions don't need to be visited. Use simple equality conditions for list partitioning, or simple range tests for range partitioning, as illustrated in the preceding examples. A good rule of thumb is that partitioning constraints should contain only comparisons of the partitioning column(s) to constants using B-tree-indexable operators, which applies even to partitioned tables, because only B-tree-indexable column(s) are allowed in the partition key. (This is not a problem when using declarative partitioning, since the automatically generated constraints are simple enough to be understood by the planner.)
All constraints on all partitions of the master table are examined during constraint exclusion, so large numbers of partitions are likely to increase query planning time considerably. Partitioning using these techniques will work well with up to perhaps a hundred partitions; don't try to use many thousands of partitions.
简单翻译:
- 约束排除只有在查询语句的where部分含有常量时,才有效。比如在做比较时,不可以用non-immutable function,类似CURRENT_TIMESTAMP就不能被优化,因为优化器不能确定这个函数在执行时会落到那个分区。
- 尽量保持分区约束的简单性,不然优化器可能无法确定要访问哪个分区。
- 所有分区表中的约束在优化器进行约束检查时,都会查到,所以只要分区表数量不是成千上万就不会影响太大。
摘自:
https://github.com/digoal/blog/blob/master/201711/20171122_02.md
https://www.postgresql.org/docs/10/static/ddl-partitioning.html#DDL-PARTITIONING-CONSTRAINT-EXCLUSION
PostgreSQL 传统 hash 分区方法和性能的更多相关文章
- PostgreSQL分区表实现——pg_pathman安装、配置
近日由于系统运行时间太长,数据库库表中的数据也是越来越多,为了缩短库表的操作时间,所以对数据库中的部分库表进行分区的操作. 通过研究,决定采用pg_pathman插件对库表进行分区操作.pg_path ...
- 数据库新秀 postgresql vs mongo 性能PK
前几天看了一篇文章<High Performance JSON PostgreSQL vs. MongoDB> 发布在Percona Live Europe 2017 作者是<Dom ...
- PostgreSQL之性能优化(转)
转载自:https://blog.csdn.net/huangwenyi1010/article/details/72853785 解决问题 前言 PostgreSQL的配置参数作为性能调优的一部分, ...
- (转) MySQL分区与传统的分库分表
传统的分库分表 原文:http://blog.csdn.net/kobejayandy/article/details/54799579 传统的分库分表都是通过应用层逻辑实现的,对于数据库层面来说,都 ...
- PostgreSQL系列一:PostgreSQL简介与安装
一.PostgreSQL简介 1.1 PostgreSQL概述 PostgreSQL数据库是目前功能最强大的开源数据库,支持丰富的数据类型(如JSON和JSONB类型. ...
- 德哥PostgreSQL学习资料汇总(转)
文章来自:https://yq.aliyun.com/articles/59251?spm=5176.100239.bloglist.95.5S5P9S 德哥博客新地址:https://billtia ...
- 索引,B+ tree,动态hash表
数据库课索引部分的学习笔记. 教材: Database System: The Complete Book, Chapter 15 Database System Implementation, Ch ...
- PostgreSQL EXPLAIN执行计划学习--多表连接几种Join方式比较
转了一部分.稍后再修改. 三种多表Join的算法: 一. NESTED LOOP: 对于被连接的数据子集较小的情况,嵌套循环连接是个较好的选择.在嵌套循环中,内表被外表驱动,外表返回的每一行都要在内表 ...
- 从NLP任务中文本向量的降维问题,引出LSH(Locality Sensitive Hash 局部敏感哈希)算法及其思想的讨论
1. 引言 - 近似近邻搜索被提出所在的时代背景和挑战 0x1:从NN(Neighbor Search)说起 ANN的前身技术是NN(Neighbor Search),简单地说,最近邻检索就是根据数据 ...
随机推荐
- ADO.Net之SqlConnection、 Sqlcommand的应用
ADO.Net之SqlConnection. Sqlcommand的应用 SqlConnection 的介绍与应用 1.介绍与作用 SqlConnection是ADO.NET中的连接类. 使用sqlc ...
- 华为笔试——C++特定位数比较
题目:特定位数比较 题目介绍:输入两行数据,第一行为 m 个正整数,以空格隔开:第二行为正整数 n ,且 n<= m:要求对第一行的数字的后三位大小进行排序,输出排行 n 的数字,其中,若不满三 ...
- Action Required: Listings Deactivated for Potential Pricing Error
Dear Seller, We are contacting you because we have detected potential pricing errors in your Amazon. ...
- python_MySQL 数据库操作
Python中的mysql操作可以使用MySQLdb模块来完成.它符合Python社区设计的Python Database API SpecificationV2.0标准,所以与其他的数据库操作的AP ...
- Python:模块学习——sys模块
sys模块常见函数和变量 sys.argv:命令行参数,实现从程序外部向程序传递参数 [注]:(1) sys.argv[0] 表示代码本身的文件路径 (2)sys.argv是一个元组,可以用[ ]提取 ...
- 通俗理解Hilbert希尔伯特空间
作者:qang pan 链接:https://www.zhihu.com/question/19967778/answer/28403912 来源:知乎 著作权归作者所有.商业转载请联系作者获得授权, ...
- 201621123037 《Java程序设计》第13周学习总结
作业13-网络 标签(空格分隔): Java 1. 本周学习总结 1.1 以你喜欢的方式(思维导图或其他)归纳总结异常相关内容. 思维导图: 其他: 网络编程:由客户端和服务器组成 - 服务器端 第一 ...
- 如何在Eclipse配置PyDev插件
如何在Eclipse配置PyDev插件 | 浏览:1733 | 更新:2014-04-21 11:36 1 2 3 4 5 分步阅读 Eclipse配置PyDev插件 方法/步骤 从 Eclips ...
- sublime text3 php开发必要的插件
一.安装Sublime Text 3 官网 http://www.sublimetext.com/3 一定要选择ST3,而不是ST2,3比2好用,真的,后面你就知道了. 选择对应的版本安装.完事后,要 ...
- Centos7安装完毕后联网-设置ip地址(VMware虚拟机)
VMware虚拟机中安装了Centos7,为了让Centos能够访问外网及设置固定的ip地址以方便本地通过SSH访问Centos,做以下几步.本文来自osfipin note. 1.确认虚拟机网络链接 ...