KingabseES例程-事实数据与规则的匹配校验
KingabseES例程-事实数据与规则的匹配校验
背景
使用规则，对数据进行校验，比如电商的用户购物订单，是否合法。这就需要订单的多维度，如 用户、地区、物流、支付手段、供应商 等各类信息，进行动态组合条件的关联匹配。
通常的匹配模式
针对某个订单
一种是根据某个订单，与全部规则进行匹配。
select a.*
from rul04_val a,
rul04 b
where 1 = 1
and (a.c1 = b.c1 or b.c1 is null)
and (a.c2 = b.c2 or b.c2 is null)
and (a.c3 = b.c3 or b.c3 is null)
and (a.c4 = b.c4 or b.c4 is null)
and a.id = 1000;
Nested Loop (cost=0.43..151.15 rows=12 width=26)
Join Filter: (((a.c1 = b.c1) OR (b.c1 IS NULL)) AND ((a.c2 = b.c2) OR (b.c2 IS NULL)) AND ((a.c3 = b.c3) OR (b.c3 IS NULL)) AND ((a.c4 = b.c4) OR (b.c4 IS NULL)))
-> Index Scan using rul04_val_pkey on rul04_val a (cost=0.43..8.45 rows=1 width=26)
Index Cond: (id = 1000)
-> Seq Scan on rul04 b (cost=0.00..62.90 rows=3990 width=16)
针对某个规则
一种是，根据某个规则，校验所有订单，
select a.*
from rul04_val a,
rul04 b
where 1 = 1
and (a.c1 = b.c1 or b.c1 is null)
and (a.c2 = b.c2 or b.c2 is null)
and (a.c3 = b.c3 or b.c3 is null)
and (a.c4 = b.c4 or b.c4 is null)
and b.id = 1000;
Nested Loop (cost=0.28..374083.30 rows=30134 width=26)
Join Filter: (((a.c1 = b.c1) OR (b.c1 IS NULL)) AND ((a.c2 = b.c2) OR (b.c2 IS NULL)) AND ((a.c3 = b.c3) OR (b.c3 IS NULL)) AND ((a.c4 = b.c4) OR (b.c4 IS NULL)))
-> Index Scan using rul04_pkey on rul04 b (cost=0.28..8.30 rows=1 width=16)
Index Cond: (id = 1000)
-> Seq Scan on rul04_val a (cost=0.00..174075.00 rows=10000000 width=26)
全部订单和全部规则
如果需要对全部或大量的订单和规则，进行匹配运算，这将是海量的计算。
select a.*
from rul04_val a,
rul04 b
where 1 = 1
and (a.c1 = b.c1 or b.c1 is null)
and (a.c2 = b.c2 or b.c2 is null)
and (a.c3 = b.c3 or b.c3 is null)
and (a.c4 = b.c4 or b.c4 is null) ;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Gather (cost=1000.00..368873734.60 rows=120236596 width=46) (actual time=5.172..879507.928 rows=386411 loops=1)
Workers Planned: 4
Workers Launched: 4
-> Nested Loop (cost=0.00..356849075.00 rows=30059149 width=46) (actual time=12.698..879456.332 rows=77282 loops=5)
Join Filter: (((a.c1 = b.c1) OR (b.c1 IS NULL)) AND ((a.c2 = b.c2) OR (b.c2 IS NULL)) AND ((a.c3 = b.c3) OR (b.c3 IS NULL)) AND ((a.c4 = b.c4) OR (b.c4 IS NULL)))
Rows Removed by Join Filter: 7979922718
-> Parallel Seq Scan on rul04_val a (cost=0.00..99075.00 rows=2500000 width=26) (actual time=0.018..151.852 rows=2000000 loops=5)
-> Seq Scan on rul04 b (cost=0.00..62.90 rows=3990 width=20) (actual time=0.001..0.162 rows=3990 loops=10000000)
Planning Time: 0.289 ms
Execution Time: 879521.346 ms
分解匹配规则
普通模式的查询，不能使用事实表的索引。这是因为关联条件中，AND运算包含OR运算，这就使得关联运算是volatile属性，这里需要进行SQL调优。
此处的SQL，是多列存在匹配需求，或者不需匹配的全集。这里可以利用对bit数据的类比，将匹配和全集的表达式，理解成1与0，反之亦可。从而得知，需要分解成2^4=16个，不含OR运算的查询，然后将16个查询结果合并。
--16个查询，组合。每个查询，都能使用较适合的索引。
select *
from rul04_val a, rul04 b
where b.c1 is null and b.c2 is null and b.c3 is null and b.c4 is null
union all
select *
from rul04_val a, rul04 b
where b.c1 is null and b.c2 is null and b.c3 is null and a.c4 = b.c4
union all
...
union all
from rul04_val a, rul04 b
where a.c1 = b.c1 and a.c2 = b.c2 and a.c3 = b.c3 and a.c4 = b.c4
;
Append (cost=5.24..458866.31 rows=10410804 width=26) (actual time=0.432..1059.117 rows=386411 loops=1)
-> Result (cost=5.24..174080.24 rows=10000000 width=26) (actual time=0.233..0.234 rows=0 loops=1)
One-Time Filter: $0
InitPlan 1 (returns $0)
-> Seq Scan on rul04 b_15 (cost=0.00..62.90 rows=12 width=0) (actual time=0.230..0.230 rows=0 loops=1)
Filter: ((c1 IS NULL) AND (c2 IS NULL) AND (c3 IS NULL) AND (c4 IS NULL))
Rows Removed by Filter: 3990
-> Seq Scan on rul04_val a (cost=5.24..174080.24 rows=10000000 width=26) (never executed)
...
-> Nested Loop (cost=112.08..5417.64 rows=1 width=26) (actual time=9.831..9.832 rows=0 loops=1)
-> HashAggregate (cost=102.80..106.79 rows=399 width=16) (actual time=0.896..1.413 rows=3990 loops=1)
Group Key: b_14.c1, b_14.c2, b_14.c3, b_14.c4
-> Seq Scan on rul04 b_14 (cost=0.00..62.90 rows=3990 width=16) (actual time=0.005..0.158 rows=3990 loops=1)
-> Bitmap Heap Scan on rul04_val a_15 (cost=9.28..13.30 rows=1 width=26) (actual time=0.002..0.002 rows=0 loops=3990)
Recheck Cond: ((c3 = b_14.c3) AND (c4 = b_14.c4) AND (c1 = b_14.c1) AND (c2 = b_14.c2))
-> BitmapAnd (cost=9.28..9.28 rows=1 width=0) (actual time=0.002..0.002 rows=0 loops=3990)
-> Bitmap Index Scan on idx_rul04_val_c34 (cost=0.00..4.51 rows=10 width=0) (actual time=0.001..0.001 rows=5 loops=3990)
Index Cond: ((c3 = b_14.c3) AND (c4 = b_14.c4))
-> Bitmap Index Scan on idx_rul04_val_c12 (cost=0.00..4.51 rows=10 width=0) (actual time=0.001..0.001 rows=4 loops=2240)
Index Cond: ((c1 = b_14.c1) AND (c2 = b_14.c2))
Planning Time: 1.763 ms
Execution Time: 1068.627 ms
由于需要关联的列数是不确定，而且优化后的查询语句，是大量的局部查询语句组合而成，这给开发人员带来巨大的工作量。我们需要一种，通用语句解决这个复杂且不确定的需求。
数据准备
规则表与事实表关联的列，建议列名相同，或使用相同的别名。
规则表
--规则表(4个条件列)
--插入多种（2^4）组合的条件数据
drop table rul04;
create table rul04
(
id int primary key,
c1 int,
c2 int,
c3 int,
c4 int
);
--基于条件列，创建单列索引，这些索引未必使用。
create index idx_rul04_c1 on rul04 (c1);
create index idx_rul04_c2 on rul04 (c2);
create index idx_rul04_c3 on rul04 (c3);
create index idx_rul04_c4 on rul04 (c4);
insert into rul04
select row_number() over (), c1, c2, c3, c4
from (
select null c1, null c2, null c3, null c4
from generate_series(1, 1000) id
union
select null, null, null, (random() * 9)::int
from generate_series(1, 1000)
union
select null, null, (random() * 9)::int, null
from generate_series(1, 1000)
union
select null, (random() * 9)::int, null, null
from generate_series(1, 1000)
union
select (random() * 9)::int, null, null, null
from generate_series(1, 1000)
union
select null, (random() * 9)::int, null, (random() * 9)::int
from generate_series(1, 1000)
union
select null, (random() * 9)::int, (random() * 9)::int, null
from generate_series(1, 1000)
union
select null, null, (random() * 9)::int, (random() * 9)::int
from generate_series(1, 1000)
union
select (random() * 9)::int, null, null, (random() * 9)::int
from generate_series(1, 1000)
union
select (random() * 9)::int, null, (random() * 9)::int, null
from generate_series(1, 1000)
union
select (random() * 9)::int, (random() * 9)::int, null, null
from generate_series(1, 1000)
union
select (random() * 9)::int, null, (random() * 9)::int, (random() * 9)::int
from generate_series(1, 1000)
union
select null, (random() * 9)::int, (random() * 9)::int, (random() * 9)::int
from generate_series(1, 1000)
union
select (random() * 9)::int, (random() * 9)::int, null, (random() * 9)::int
from generate_series(1, 1000)
union
select (random() * 9)::int, (random() * 9)::int, (random() * 9)::int, null
from generate_series(1, 1000)
union
select (random() * 9)::int, (random() * 9)::int, (random() * 9)::int, (random() * 9)::int
from generate_series(1, 1000)
) t
;
事实表
--事实表(4个条件列)
create table rul04_val
(
id int,
c1 int,
c2 int,
c3 int,
c4 int,
val numeric
);
insert into rul04_val
select id,
(random() * 999)::int,
(random() * 999)::int,
(random() * 999)::int,
(random() * 999)::int,
round(random() * 10000, 2)
from generate_series(1, 10000000) id;
alter table rul04_val
add primary key (id);
--单维度列的索引
create index idx_rul04_val_c1 on rul04_val (c1);
create index idx_rul04_val_c2 on rul04_val (c2);
create index idx_rul04_val_c3 on rul04_val (c3);
create index idx_rul04_val_c4 on rul04_val (c4);
--按规则表的数据分布规律，创建2个及以上列的复合索引
create index idx_rul04_val_c12 on rul04_val (c1,c2);
create index idx_rul04_val_c13 on rul04_val (c3,c4);
并行执行动态查询
根据条件的组合关系，动态生成全部16个查询语句，使用dblink的异步查询函数，并行执行这些查询语句，最后结果组合。
创建多个dblink连接
建议数据的socket连接，允许免密。
create extension dblink ;
select connam,
dblink_connect(connam,
format('host=%s port=%s dbname=%s user=%s ',
current_setting('unix_socket_directories', true),
current_setting('port', true),
current_database(),
current_user))::text as reault
from (select 'dblink_' || id as connam
from generate_series(1000, 1000 - 1 + 100) id) t;
事实表与规则表的全量匹配运算
事实表名、规则表名、条件维度列，已经并行度，都可以自定义。
如果并行执行，用时最长的查询，决定整体执行时长。
在CTE中，会话连接（conn）记录数，决定并行度。
为了语句的通用性，可以使用主键列或唯一列，代替CTID列。
--dattab: 事实表名； rultab: 规则表名； colname: 关联的条件维度列名
with val as (select 'rul04_val' dattab, 'rul04' rultab, ('{c1,c2,c3,c4}'::text[]) colname),
sta as
(select id,
format('select a.ctid atid,b.ctid btid from %I a , %I b where %s ', dattab, rultab, whr) sqlsta
from val,
lateral array_length(colname, 1) colnum,
generate_series(0, 2 ^ (array_length(colname, 1)) - 1) id,
lateral ( select string_agg(case
when get_bit(id::bit(10), sn + 9 - colnum) = 1
then format('a.%1$I=b.%1$I', colname[sn])
else format('b.%1$I is null', colname[sn]) end, ' and ') whr
from generate_series(1, colnum) sn)),
conn as
(select connam,
row_number() over () sn
from unnest(dblink_get_connections()) connam
where dblink_is_busy(connam) = 0
limit 4),
qursta as
(select connam, string_agg(sqlsta, ' union all ') as qursta
from conn,
sta,
(select count(*) connum from conn) t
where conn.sn = sta.id % connum + 1
group by connam),
sedqur as
(select array_agg(connam) connams
from qursta
, lateral dblink_send_query(connam, qursta) sq),
res as (
select res.*
from sedqur
, lateral unnest(connams) connam
, lateral dblink_get_result(connam) as res(atid tid, btid tid))
select a.*,b.*
from res
join rul04_val a on a.ctid = atid
join rul04 b on b.ctid = btid;
;
Hash Join (cost=343.84..40906.33 rows=10000 width=46) (actual time=144.807..1444.010 rows=386411 loops=1)
Hash Cond: (res.btid = b.ctid)
CTE conn
-> Limit (cost=0.01..0.17 rows=1 width=40) (actual time=0.046..0.173 rows=20 loops=1)
-> WindowAgg (cost=0.01..0.17 rows=1 width=40) (actual time=0.046..0.171 rows=20 loops=1)
-> Function Scan on unnest connam (cost=0.01..0.15 rows=1 width=32) (actual time=0.042..0.162 rows=20 loops=1)
Filter: (dblink_is_busy(connam) = 0)
Rows Removed by Filter: 2
CTE sedqur
-> Aggregate (cost=30.66..30.67 rows=1 width=32) (actual time=0.447..0.454 rows=1 loops=1)
-> Nested Loop (cost=30.60..30.65 rows=1 width=32) (actual time=0.308..0.447 rows=16 loops=1)
-> GroupAggregate (cost=30.59..30.62 rows=1 width=64) (actual time=0.296..0.318 rows=16 loops=1)
Group Key: conn.connam
-> Sort (cost=30.59..30.60 rows=1 width=64) (actual time=0.293..0.298 rows=16 loops=1)
Sort Key: conn.connam
Sort Method: quicksort Memory: 27kB
-> Nested Loop (cost=30.06..30.58 rows=1 width=64) (actual time=0.220..0.276 rows=16 loops=1)
-> Function Scan on colnum (cost=0.00..0.01 rows=1 width=4) (actual time=0.002..0.003 rows=1 loops=1)
-> Nested Loop (cost=30.06..30.56 rows=1 width=64) (actual time=0.216..0.268 rows=16 loops=1)
-> Hash Join (cost=0.06..0.53 rows=1 width=36) (actual time=0.202..0.211 rows=16 loops=1)
Hash Cond: ((((id.id)::bigint % (count(*))) + 1) = conn.sn)
-> Nested Loop (cost=0.02..0.36 rows=16 width=12) (actual time=0.011..0.015 rows=16 loops=1)
-> Aggregate (cost=0.02..0.03 rows=1 width=8) (actual time=0.006..0.007 rows=1 loops=1)
-> CTE Scan on conn conn_1 (cost=0.00..0.02 rows=1 width=0) (actual time=0.000..0.002 rows=20 loops=1)
-> Function Scan on generate_series id (cost=0.00..0.16 rows=16 width=4) (actual time=0.004..0.005 rows=16 loops=1)
-> Hash (cost=0.02..0.02 rows=1 width=40) (actual time=0.186..0.186 rows=20 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> CTE Scan on conn (cost=0.00..0.02 rows=1 width=40) (actual time=0.047..0.179 rows=20 loops=1)
-> Aggregate (cost=30.00..30.02 rows=1 width=32) (actual time=0.003..0.003 rows=1 loops=16)
-> Function Scan on generate_series sn (cost=0.00..10.00 rows=1000 width=4) (actual time=0.000..0.000 rows=4 loops=16)
-> Function Scan on dblink_send_query sq (cost=0.00..0.01 rows=1 width=0) (actual time=0.008..0.008 rows=1 loops=16)
CTE res
-> Nested Loop (cost=0.01..200.22 rows=10000 width=12) (actual time=143.473..973.506 rows=386411 loops=1)
-> Nested Loop (cost=0.00..0.22 rows=10 width=32) (actual time=0.455..0.470 rows=16 loops=1)
-> CTE Scan on sedqur (cost=0.00..0.02 rows=1 width=32) (actual time=0.449..0.449 rows=1 loops=1)
-> Function Scan on unnest connam_1 (cost=0.00..0.10 rows=10 width=32) (actual time=0.005..0.012 rows=16 loops=1)
-> Function Scan on dblink_get_result res_1 (cost=0.00..10.00 rows=1000 width=12) (actual time=57.747..59.090 rows=24151 loops=16)
-> Nested Loop (cost=0.00..40425.00 rows=10000 width=32) (actual time=143.488..1377.323 rows=386411 loops=1)
-> CTE Scan on res (cost=0.00..200.00 rows=10000 width=12) (actual time=143.475..1029.676 rows=386411 loops=1)
-> Tid Scan on rul04_val a (cost=0.00..4.01 rows=1 width=32) (actual time=0.001..0.001 rows=1 loops=386411)
TID Cond: (ctid = res.atid)
-> Hash (cost=62.90..62.90 rows=3990 width=26) (actual time=1.308..1.308 rows=3990 loops=1)
Buckets: 4096 Batches: 1 Memory Usage: 244kB
-> Seq Scan on rul04 b (cost=0.00..62.90 rows=3990 width=26) (actual time=0.007..0.565 rows=3990 loops=1)
Planning Time: 0.479 ms
Execution Time: 1456.717 ms
断开所有连接
select *
from unnest(dblink_get_connections()) connam
, lateral dblink_disconnect(connam);
总结
全量匹配，可以从几十分钟，减少到几秒。关键要在索引和组合条件之间，搞平衡。本质就是空间与时间的转换。
优化后的查询语句，仰赖事实表的多列索引，与规则数据的适配程度。由于分解重组后的查询语句较多，需要在索引与查询语句之间，进行取舍平衡。
KingabseES例程-事实数据与规则的匹配校验的更多相关文章
- oracle xmltype导入并解析Excel数据 (二)规则说明
规则表字段说明如下: 其中RULE_FUNC_CUSTOMIZE表示,用户自己写函数,去判断数据是否合法,存储的是函数的名字 此函数的参数只有一个,该列的值,字段类型是Varchar2, 校验失败的话 ...
- ElasticSearch(十)Elasticsearch检索出的数据列表按字段匹配的优先顺序及搜索单词拼音一部分搜不到数据
检索出的数据列表按字段匹配的优先顺序 一.举例 比如,发布一篇文章,文章包括基本的字段包括标题.发布时间.点击率.关键字.内容.当在页面中输入“教育”搜索关键词,会检索出指定字段包括“教育”的所有数据 ...
- Spring_数据校验和自定义检验规则和分组校验
@Validated :绑定需要校验的数据. 数据校验规则:为数据绑定校验的规则 private Long booId;@NotNull(message = "不能为空")pri ...
- R合并数据框有重复匹配时只保留第一行
前言 合并数据框有重复匹配时通常会返回所有的匹配,如何只保留匹配的第一行呢?其实这个需求也很常见.如芯片探针ID和基因ID往往多对一,要合并ID对应矩阵和芯片表达矩阵时. 数据例子 data = da ...
- 【SpringMVC】SpringMVC系列12之数据类型转换、格式化、校验
12.数据类型转换.格式化.校验 12.1.数据绑定流程 Spring MVC 主框架将 ServletRequest 对象及目标方法的入参实例传递给 WebDataBinderFacto ...
- python读取两个csv文件数据，进行查找匹配出现次数
现有需求 表1 表2 需要拿表1中的编码去表2中的门票编码列匹配,统计出现的次数,由于表2编码列是区域间,而且列不是固定的,代码如下 #encoding:utf-8 ##导入两个CSV进行比对 imp ...
- Oracle GoldenGate 四、数据过滤和数据项匹配
写在开始前 从两周前我花了大量的业余时间阅读GoldenGate官方文档,并根据文档实践和进一步学习了解GoldenGate,以下便是根据官方文档理解总结的GoldenGate学习内容: Oracle ...
- web.xml中的url-pattern写法规则及匹配过程
servlet和filter在javaEE开发中很常用,因此有必要知道web.xml文件映射的规则 1. 写法 ①完全匹配:以“/”开头,以字母(非“*”)结束 如:<url-patte ...
- 用Excel做数据分析常用函数（数据清理、关联匹配……）
本文总结在使用Excel进行数据分析时,最常用的功能和函数. Excel的功能和函数非常多,用进废退,除了学习基本的函数和功能,最重要的是遇到问题可以快速的搜索并解决. 首先Excel可以处理的数据量 ...
- laravel 数据验证规则
return [ 'accepted' => '必须为yes,on,1,true', 'active_url' => '是否是一个合法的url,基于PHP的checkdnsrr函数,因此也 ...
随机推荐
- 使用 Dockerfile 构建生产环境镜像
传统部署的坑: 1202 年了,如果你连 Docker 都不知道是什么,我建议买一本书看看--或者谷歌一下,博客已经写烂了. 为什么有这篇文章,是因为我在真正做容器化改造的时候,发现公司生产环境存在大 ...
- RAT蓝队自动化测试框架
RAT蓝队自动化测试框架 介绍 RAT 是根据 MITRE ATT&CK 战术矩阵测试蓝队检测能力的脚本框架,由 python2.7 编写,共有 50 多种不同 ATT&CK 技术点和 ...
- 【hack】浅浅说说自己构造hack的一些逻辑~
怎么说呢,相信很多考过竞赛的同学都会在平时的练习/考试中遭遇过100分但没有AC的情况,结果一看评测结果:subtask的数据点没过! 这时候就是遇到hack数据了,如果被这类数据卡住,说明你的代码可 ...
- python:时间模块dateutil
安装 pip install python-dateutil dateutil模块主要有两个函数,parser和rrule. 其中parser是根据字符串解析成datetime,而rrule则是根据定 ...
- ROC 曲线与 PR 曲线
ROC 曲线与 PR 曲线 ROC 曲线与 PR 曲线 ROC 曲线和 PR 曲线是评估机器学习算法性能的两条重要曲线,两者概念比较容易混淆,但是两者的使用场景是不同的.本文主要讲述两种曲线的含义以及 ...
- [kubernetes]集群中部署CoreDNS服务
前言 从k8s 1.11版本开始,k8s集群的dns服务由CoreDNS提供.之前已经使用二进制文件部署了一个三master三node的k8s集群,现在需要在集群内部部署DNS服务. 环境信息 IP ...
- selenium + python自动化环境搭建
Selenium是一个用于Web应用程序测试的工具.Selenium测试直接运行在浏览器中,就像真正的用户在操作一样.支持的浏览器包括IE(7, 8, 9, 10, 11),Mozilla Firef ...
- 文心一言 VS 讯飞星火 VS chatgpt （92）-- 算法导论8.3 5题
五.用go语言,在本节给出的第一个卡片排序算法中,为排序 d 位十进制数,在最坏情况下需要多少轮排序?在最坏情况下,操作员需要记录多少堆卡片? 文心一言: 你提到的第一个卡片排序算法可能是指的基数排序 ...
- 音频格式轻松转 - foobar2000
一.foobar2000简介 foobar2000 是一款免费的专业级别音频解码播放器,支持的诸多音频格式,可加载附加组件扩展更多支持. 除了解码以外,可轻松实现对音频格式的转换,支持几乎所有主流格式 ...
- xmind文件数据解析重构成mindmap可识别数据
[需求背景] 测试平台开发中,需要引入前端mindmap模块,进行在线xmind实时解析并前端展示 [卡点难点] 选取什么库进行xmind解析 如何转换成mindmap可以识别的数据 [xmind解析 ...