union all 优化案例
遇到个子查询嵌套 UNION ALL 的SQL语句很慢，谓词过滤条件不能内推进去，需要优化这段 UNION ALL这块的内容。
UNION ALL 慢SQL：
SELECT * FROM ((SELECT
jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001 AS jfw03001,
jfw03.jfw03002 AS jfw03002,
jfw03.jfw03003 AS jfw03003,
jfw03.jfw03004 AS jfw03004,
jfw03.jfw03005 AS jfw03005,
jfw03.jfw03006 AS jfw03006,
jfw03.jfw03007 AS jfw03007,
jfw03.remark AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM emc jfw03
LEFT JOIN emc old03
ON jfw03.emc_person_id =
old03.emc_person_id
WHERE jfw03.jfw03001 = old03.jfw03001
AND jfw03.handle_mark = '2'
AND old03.handle_mark IN ('1', '3'))
UNION ALL (SELECT jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001,
jfw03.jfw03002,
jfw03.jfw03003,
jfw03.jfw03004,
jfw03.jfw03005,
jfw03.jfw03006,
jfw03.jfw03007,
jfw03.remark,
NULL AS oldId,
NULL AS jfw03001pre,
NULL AS jfw03002pre,
NULL AS jfw03003pre,
NULL AS jfw03004pre,
NULL AS jfw03005pre,
NULL AS jfw03006pre,
NULL AS jfw03007pre,
NULL AS remarkpre
FROM emc jfw03
WHERE jfw03.handle_mark = '2'
AND NOT EXISTS
(SELECT 1
FROM emc old03
WHERE jfw03.emc_person_id =
old03.emc_person_id
AND jfw03.jfw03001 = old03.jfw03001
AND old03.handle_mark IN ('1', '3')))
UNION ALL (SELECT NULL AS newid,
old03.emc_person_id AS bizPersonOid,
NULL AS jfw03001,
NULL AS jfw03002,
NULL AS jfw03003,
NULL AS jfw03004,
NULL AS jfw03005,
NULL AS jfw03006,
NULL AS jfw03007,
NULL AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM emc old03
WHERE old03.handle_mark IN ('1', '3')
AND NOT EXISTS
(SELECT 1
FROM emc jfw03
WHERE jfw03.emc_person_id =
old03.emc_person_id
AND jfw03.jfw03001 = old03.jfw03001
AND jfw03.handle_mark = '2'))) jfw03;
执行计划：
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=0.69..7253737.08 rows=49711 width=274) (actual time=0.252..60906.535 rows=3291407 loops=1)
-> Nested Loop (cost=0.69..2753734.17 rows=49709 width=274) (actual time=0.251..16535.465 rows=49558 loops=1)
-> Seq Scan on emc jfw03 (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.010..5103.042 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 8885720
-> Index Scan using idx_1_2_3_emc on emc old03 (cost=0.69..2.12 rows=1 width=139) (actual time=0.010..0.010 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
-> Nested Loop Anti Join (cost=0.69..1864694.18 rows=1 width=399) (actual time=0.032..16606.236 rows=1065808 loops=1)
-> Seq Scan on emc jfw03_1 (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.013..5171.111 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 8885720
-> Index Only Scan using idx_1_2_3_emc on emc old03_1 (cost=0.69..1.33 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
Heap Fetches: 0
-> Nested Loop Anti Join (cost=0.69..2634563.07 rows=1 width=399) (actual time=0.033..27638.050 rows=2176041 loops=1)
-> Seq Scan on emc old03_2 (cost=0.00..347258.58 rows=2205308 width=139) (actual time=0.014..5510.884 rows=2225077 loops=1)
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 7774923
-> Index Only Scan using idx_1_2_3_emc on emc jfw03_2 (cost=0.69..1.03 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
Heap Fetches: 0
Planning Time: 0.318 ms
Execution Time: 60987.916 ms
(27 行记录)
加个索引看看速度
CREATE INDEX idx_emc_1 ON emc ( handle_mark );
加索引后执行计划
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=21059.67..7011218.65 rows=49969 width=278) (actual time=115.683..46578.675 rows=3291407 loops=1)
-> Nested Loop (cost=21059.67..2655195.15 rows=49967 width=278) (actual time=115.682..11939.294 rows=49558 loops=1)
-> Bitmap Heap Scan on emc jfw03 (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=115.458..417.906 rows=1114280 loops=1)
Recheck Cond: (handle_mark = '2'::bpchar)
Heap Blocks: exact=221151
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..20778.90 rows=1120329 width=0) (actual time=92.887..92.887 rows=1114280 loops=1)
Index Cond: (handle_mark = '2'::bpchar)
-> Index Scan using idx_1_2_3_emc on emc old03 (cost=0.69..2.13 rows=1 width=141) (actual time=0.010..0.010 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
-> Nested Loop Anti Join (cost=21059.67..1766155.15 rows=1 width=401) (actual time=127.721..11690.213 rows=1065808 loops=1)
-> Bitmap Heap Scan on emc jfw03_1 (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=127.640..423.128 rows=1114280 loops=1)
Recheck Cond: (handle_mark = '2'::bpchar)
Heap Blocks: exact=221151
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..20778.90 rows=1120329 width=0) (actual time=105.117..105.117 rows=1114280 loops=1)
Index Cond: (handle_mark = '2'::bpchar)
-> Index Only Scan using idx_1_2_3_emc on emc old03_1 (cost=0.69..1.34 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
Heap Fetches: 0
-> Nested Loop Anti Join (cost=39323.15..2589118.81 rows=1 width=401) (actual time=201.187..22828.908 rows=2176041 loops=1)
-> Bitmap Heap Scan on emc old03_2 (cost=39322.47..289365.69 rows=2222658 width=141) (actual time=201.135..603.170 rows=2225077 loops=1)
Recheck Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
Heap Blocks: exact=222258
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..38766.81 rows=2222658 width=0) (actual time=179.770..179.770 rows=2225077 loops=1)
Index Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
-> Index Only Scan using idx_1_2_3_emc on emc jfw03_2 (cost=0.69..1.02 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
Heap Fetches: 0
Planning Time: 0.373 ms
Execution Time: 46666.296 ms
(33 行记录)
快了14秒，没达到目的效果，无解，只能改写SQL，可以改 FULL JOIN。
等价改写：
SELECT * FROM (
with x as (SELECT * FROM emc where handle_mark IN ('1', '3','2'))
SELECT
jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001 AS jfw03001,
jfw03.jfw03002 AS jfw03002,
jfw03.jfw03003 AS jfw03003,
jfw03.jfw03004 AS jfw03004,
jfw03.jfw03005 AS jfw03005,
jfw03.jfw03006 AS jfw03006,
jfw03.jfw03007 AS jfw03007,
jfw03.remark AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM (SELECT * FROM x WHERE handle_mark IN ('2')) jfw03
full join (SELECT * FROM x WHERE handle_mark IN ('1', '3')) old03
ON jfw03.emc_person_id = old03.emc_person_id AND jfw03.jfw03001 = old03.jfw03001);
等价改写计划：
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------
Merge Full Join (cost=10000479890.42..10000480406.20 rows=33430 width=1844) (actual time=9433.530..10954.254 rows=3291407 loops=1)
Merge Cond: ((x.emc_person_id = x_1.emc_person_id) AND (x.jfw03001 = x_1.jfw03001))
CTE x
-> Bitmap Heap Scan on emc (cost=57545.45..325771.51 rows=3342986 width=145) (actual time=268.766..721.294 rows=3339357 loops=1)
Recheck Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
Heap Blocks: exact=222260
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..56709.71 rows=3342986 width=0) (actual time=246.952..246.953 rows=3339357 loops=1)
Index Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
-> Sort (cost=76389.65..76431.43 rows=16715 width=924) (actual time=3818.370..4021.726 rows=1114280 loops=1)
Sort Key: x.emc_person_id, x.jfw03001
Sort Method: quicksort Memory: 345133kB
-> CTE Scan on x (cost=0.00..75217.18 rows=16715 width=924) (actual time=268.780..1391.594 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 2225077
-> Sort (cost=77729.26..77812.83 rows=33430 width=924) (actual time=5615.144..5949.562 rows=2225599 loops=1)
Sort Key: x_1.emc_person_id, x_1.jfw03001
Sort Method: quicksort Memory: 676056kB
-> CTE Scan on x x_1 (cost=0.00..75217.18 rows=33430 width=924) (actual time=0.007..360.353 rows=2225077 loops=1)
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1114280
Planning Time: 0.138 ms
Execution Time: 11101.829 ms
(22 行记录)
改写后11s就能跑出结果，符合性能优化目标。
这条改写SQL留个坑，不完全等价，完全等价的没放出来，有兴趣的同学可以思考下，自己做做实验。
union all 优化案例的更多相关文章
- Hive优化案例
1.Hadoop计算框架的特点 数据量大不是问题,数据倾斜是个问题. jobs数比较多的作业效率相对比较低,比如即使有几百万的表,如果多次关联多次汇总,产生十几个jobs,耗时很长.原因是map re ...
- 数据库优化案例——————某市中心医院HIS系统
记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也开始分享自己做的优化案例. 最近一直很忙,博客产出也少的 ...
- mysql优化案例
MySQL优化案例 Mysql5.1大表分区效率测试 Mysql5.1大表分区效率测试MySQL | add at 2009-03-27 12:29:31 by PConline | view:60, ...
- SQL 优化案例 1
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
- 老李案例分享：Weblogic性能优化案例
老李案例分享:Weblogic性能优化案例 POPTEST的测试技术交流qq群:450192312 网站应用首页大小在130K左右,在之前的测试过程中,其百用户并发的平均响应能力在6.5秒,性能优化后 ...
- 数据库优化案例——————某知名零售企业ERP系统
写在前面 记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也分享自己做的优化案例. 之前分享过OA系统.H ...
- Spark集群之yarn提交作业优化案例
Spark集群之yarn提交作业优化案例 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.启动Hadoop集群 1>.自定义批量管理脚本 [yinzhengjie@s101 ...
- MySQL参数优化案例
环境介绍 优化层级与指导思想 优化过程 最小化安装情况下的性能表现 优化innodb_buffer_pool_size 优化innodb_log_files_in_group&innodb_l ...
- MySQL的索引单表优化案例分析
建表 建立本次优化案例中所需的数据库及数据表 CREATE DATABASE db0206; USE db0206; CREATE TABLE `db0206`.`article`( `id` INT ...
- SQL 优化案例
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
随机推荐
- 【译】发布 .NET Aspire 预览版 2（二）
原文 | Damian Edwards 翻译 | 郑子铭 组件更新 组件包现在有单独的图标 大多数 Aspire 组件的 NuGet 包现在都具有代表性图标,以便在 NuGet 包管理器对话框中更轻松 ...
- ARM中PC和LR寄存器的关系
我们常常听说的PC,LR到底是什么关系,我这次终于弄明白了.我们都知道,LR是指向PC下一次要执行的地址,但是ARM不同的工作模式,他们有不同的关系.ARM有如下几种工作模式:用户模式,FIQ模式,I ...
- Libata Error Message 解析
Libata error messages Contents [hide] 1 Overview 2 Prefix 3 Exception line 4 Input taskfile 5 O ...
- OGP协议的使用
OGP协议是一套Metatags的规格,用来标注页面,告诉我们你的网页快照.帮助社交app高效并准确的获取网页中的核心链接.标题.主图.正文摘要等信息,使得该网页在社交分享中有更好的展现体验. 如果网 ...
- OCR 01: EasyOCR
Catalog OCR 01: EasyOCR OCR 02: Tesseract-OCR OCR 03: PaddleOCR Related Links Official site with onl ...
- 【Android】使用MediaExtractor、MediaMuxer去掉视频文件中的音频数据
1 简介 本文以 mp4 文件为例,讲解去音频操作.mp4 是一种视频封装的容器,里面包含音频(audio)和视频(video)数据,对应的数据编码格式分别为 aac 和 h264.在去音频过程中 ...
- c2工具sliver的python客户端无法修改grpc超时时间的解决办法
业务需要,调用了很多implants来执行对应系统上的命令, 但是无论怎么指定interactive.py中execute方法参数, 命令执行超时时间总是30. 后面通过扩展execute方法增加一个 ...
- nmap top N端口获取
使用nmap 扫描时可能会扫描tcp top100 top1000 端口, 有时需要去配置文件提取,配置文件路径/usr/share/nmap/nmap-services, 具体根据实际安装情况调整: ...
- 【Android 抓包对抗】代理检查绕过
1. 安装apk,点进去发现一点就挂 2. apk 拖入到jadx中观察,发现多出检查,一旦满足条件就会退出 .... if (((ConnectivityManager) getSystemServ ...
- Xray安全评估工具使用
xray 是一款功能强大的安全评估工具,主要特性有: 检测速度快.发包速度快; 漏洞检测算法高效. 支持范围广.大至 OWASP Top 10 通用漏洞检测,小至各种 CMS 框架 POC,均可以支持 ...