union all 优化案例
遇到个子查询嵌套 UNION ALL 的SQL语句很慢，谓词过滤条件不能内推进去，需要优化这段 UNION ALL这块的内容。
UNION ALL 慢SQL：
SELECT * FROM ((SELECT
jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001 AS jfw03001,
jfw03.jfw03002 AS jfw03002,
jfw03.jfw03003 AS jfw03003,
jfw03.jfw03004 AS jfw03004,
jfw03.jfw03005 AS jfw03005,
jfw03.jfw03006 AS jfw03006,
jfw03.jfw03007 AS jfw03007,
jfw03.remark AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM emc jfw03
LEFT JOIN emc old03
ON jfw03.emc_person_id =
old03.emc_person_id
WHERE jfw03.jfw03001 = old03.jfw03001
AND jfw03.handle_mark = '2'
AND old03.handle_mark IN ('1', '3'))
UNION ALL (SELECT jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001,
jfw03.jfw03002,
jfw03.jfw03003,
jfw03.jfw03004,
jfw03.jfw03005,
jfw03.jfw03006,
jfw03.jfw03007,
jfw03.remark,
NULL AS oldId,
NULL AS jfw03001pre,
NULL AS jfw03002pre,
NULL AS jfw03003pre,
NULL AS jfw03004pre,
NULL AS jfw03005pre,
NULL AS jfw03006pre,
NULL AS jfw03007pre,
NULL AS remarkpre
FROM emc jfw03
WHERE jfw03.handle_mark = '2'
AND NOT EXISTS
(SELECT 1
FROM emc old03
WHERE jfw03.emc_person_id =
old03.emc_person_id
AND jfw03.jfw03001 = old03.jfw03001
AND old03.handle_mark IN ('1', '3')))
UNION ALL (SELECT NULL AS newid,
old03.emc_person_id AS bizPersonOid,
NULL AS jfw03001,
NULL AS jfw03002,
NULL AS jfw03003,
NULL AS jfw03004,
NULL AS jfw03005,
NULL AS jfw03006,
NULL AS jfw03007,
NULL AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM emc old03
WHERE old03.handle_mark IN ('1', '3')
AND NOT EXISTS
(SELECT 1
FROM emc jfw03
WHERE jfw03.emc_person_id =
old03.emc_person_id
AND jfw03.jfw03001 = old03.jfw03001
AND jfw03.handle_mark = '2'))) jfw03;
执行计划：
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=0.69..7253737.08 rows=49711 width=274) (actual time=0.252..60906.535 rows=3291407 loops=1)
-> Nested Loop (cost=0.69..2753734.17 rows=49709 width=274) (actual time=0.251..16535.465 rows=49558 loops=1)
-> Seq Scan on emc jfw03 (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.010..5103.042 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 8885720
-> Index Scan using idx_1_2_3_emc on emc old03 (cost=0.69..2.12 rows=1 width=139) (actual time=0.010..0.010 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
-> Nested Loop Anti Join (cost=0.69..1864694.18 rows=1 width=399) (actual time=0.032..16606.236 rows=1065808 loops=1)
-> Seq Scan on emc jfw03_1 (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.013..5171.111 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 8885720
-> Index Only Scan using idx_1_2_3_emc on emc old03_1 (cost=0.69..1.33 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
Heap Fetches: 0
-> Nested Loop Anti Join (cost=0.69..2634563.07 rows=1 width=399) (actual time=0.033..27638.050 rows=2176041 loops=1)
-> Seq Scan on emc old03_2 (cost=0.00..347258.58 rows=2205308 width=139) (actual time=0.014..5510.884 rows=2225077 loops=1)
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 7774923
-> Index Only Scan using idx_1_2_3_emc on emc jfw03_2 (cost=0.69..1.03 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
Heap Fetches: 0
Planning Time: 0.318 ms
Execution Time: 60987.916 ms
(27 行记录)
加个索引看看速度
CREATE INDEX idx_emc_1 ON emc ( handle_mark );
加索引后执行计划
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=21059.67..7011218.65 rows=49969 width=278) (actual time=115.683..46578.675 rows=3291407 loops=1)
-> Nested Loop (cost=21059.67..2655195.15 rows=49967 width=278) (actual time=115.682..11939.294 rows=49558 loops=1)
-> Bitmap Heap Scan on emc jfw03 (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=115.458..417.906 rows=1114280 loops=1)
Recheck Cond: (handle_mark = '2'::bpchar)
Heap Blocks: exact=221151
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..20778.90 rows=1120329 width=0) (actual time=92.887..92.887 rows=1114280 loops=1)
Index Cond: (handle_mark = '2'::bpchar)
-> Index Scan using idx_1_2_3_emc on emc old03 (cost=0.69..2.13 rows=1 width=141) (actual time=0.010..0.010 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
-> Nested Loop Anti Join (cost=21059.67..1766155.15 rows=1 width=401) (actual time=127.721..11690.213 rows=1065808 loops=1)
-> Bitmap Heap Scan on emc jfw03_1 (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=127.640..423.128 rows=1114280 loops=1)
Recheck Cond: (handle_mark = '2'::bpchar)
Heap Blocks: exact=221151
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..20778.90 rows=1120329 width=0) (actual time=105.117..105.117 rows=1114280 loops=1)
Index Cond: (handle_mark = '2'::bpchar)
-> Index Only Scan using idx_1_2_3_emc on emc old03_1 (cost=0.69..1.34 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
Heap Fetches: 0
-> Nested Loop Anti Join (cost=39323.15..2589118.81 rows=1 width=401) (actual time=201.187..22828.908 rows=2176041 loops=1)
-> Bitmap Heap Scan on emc old03_2 (cost=39322.47..289365.69 rows=2222658 width=141) (actual time=201.135..603.170 rows=2225077 loops=1)
Recheck Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
Heap Blocks: exact=222258
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..38766.81 rows=2222658 width=0) (actual time=179.770..179.770 rows=2225077 loops=1)
Index Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
-> Index Only Scan using idx_1_2_3_emc on emc jfw03_2 (cost=0.69..1.02 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
Heap Fetches: 0
Planning Time: 0.373 ms
Execution Time: 46666.296 ms
(33 行记录)
快了14秒，没达到目的效果，无解，只能改写SQL，可以改 FULL JOIN。
等价改写：
SELECT * FROM (
with x as (SELECT * FROM emc where handle_mark IN ('1', '3','2'))
SELECT
jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001 AS jfw03001,
jfw03.jfw03002 AS jfw03002,
jfw03.jfw03003 AS jfw03003,
jfw03.jfw03004 AS jfw03004,
jfw03.jfw03005 AS jfw03005,
jfw03.jfw03006 AS jfw03006,
jfw03.jfw03007 AS jfw03007,
jfw03.remark AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM (SELECT * FROM x WHERE handle_mark IN ('2')) jfw03
full join (SELECT * FROM x WHERE handle_mark IN ('1', '3')) old03
ON jfw03.emc_person_id = old03.emc_person_id AND jfw03.jfw03001 = old03.jfw03001);
等价改写计划：
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------
Merge Full Join (cost=10000479890.42..10000480406.20 rows=33430 width=1844) (actual time=9433.530..10954.254 rows=3291407 loops=1)
Merge Cond: ((x.emc_person_id = x_1.emc_person_id) AND (x.jfw03001 = x_1.jfw03001))
CTE x
-> Bitmap Heap Scan on emc (cost=57545.45..325771.51 rows=3342986 width=145) (actual time=268.766..721.294 rows=3339357 loops=1)
Recheck Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
Heap Blocks: exact=222260
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..56709.71 rows=3342986 width=0) (actual time=246.952..246.953 rows=3339357 loops=1)
Index Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
-> Sort (cost=76389.65..76431.43 rows=16715 width=924) (actual time=3818.370..4021.726 rows=1114280 loops=1)
Sort Key: x.emc_person_id, x.jfw03001
Sort Method: quicksort Memory: 345133kB
-> CTE Scan on x (cost=0.00..75217.18 rows=16715 width=924) (actual time=268.780..1391.594 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 2225077
-> Sort (cost=77729.26..77812.83 rows=33430 width=924) (actual time=5615.144..5949.562 rows=2225599 loops=1)
Sort Key: x_1.emc_person_id, x_1.jfw03001
Sort Method: quicksort Memory: 676056kB
-> CTE Scan on x x_1 (cost=0.00..75217.18 rows=33430 width=924) (actual time=0.007..360.353 rows=2225077 loops=1)
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1114280
Planning Time: 0.138 ms
Execution Time: 11101.829 ms
(22 行记录)
改写后11s就能跑出结果，符合性能优化目标。
这条改写SQL留个坑，不完全等价，完全等价的没放出来，有兴趣的同学可以思考下，自己做做实验。
union all 优化案例的更多相关文章
- Hive优化案例
1.Hadoop计算框架的特点 数据量大不是问题,数据倾斜是个问题. jobs数比较多的作业效率相对比较低,比如即使有几百万的表,如果多次关联多次汇总,产生十几个jobs,耗时很长.原因是map re ...
- 数据库优化案例——————某市中心医院HIS系统
记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也开始分享自己做的优化案例. 最近一直很忙,博客产出也少的 ...
- mysql优化案例
MySQL优化案例 Mysql5.1大表分区效率测试 Mysql5.1大表分区效率测试MySQL | add at 2009-03-27 12:29:31 by PConline | view:60, ...
- SQL 优化案例 1
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
- 老李案例分享：Weblogic性能优化案例
老李案例分享:Weblogic性能优化案例 POPTEST的测试技术交流qq群:450192312 网站应用首页大小在130K左右,在之前的测试过程中,其百用户并发的平均响应能力在6.5秒,性能优化后 ...
- 数据库优化案例——————某知名零售企业ERP系统
写在前面 记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也分享自己做的优化案例. 之前分享过OA系统.H ...
- Spark集群之yarn提交作业优化案例
Spark集群之yarn提交作业优化案例 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.启动Hadoop集群 1>.自定义批量管理脚本 [yinzhengjie@s101 ...
- MySQL参数优化案例
环境介绍 优化层级与指导思想 优化过程 最小化安装情况下的性能表现 优化innodb_buffer_pool_size 优化innodb_log_files_in_group&innodb_l ...
- MySQL的索引单表优化案例分析
建表 建立本次优化案例中所需的数据库及数据表 CREATE DATABASE db0206; USE db0206; CREATE TABLE `db0206`.`article`( `id` INT ...
- SQL 优化案例
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
随机推荐
- 深入浅出Java多线程(九)：synchronized与锁
引言 大家好,我是你们的老伙计秀才!今天带来的是[深入浅出Java多线程]系列的第九篇内容:synchronized与锁.大家觉得有用请点赞,喜欢请关注!秀才在此谢过大家了!!! 在现代软件开发中,多 ...
- Linux--top命令解释
top命令解释 1.1 系统运行时间和平均负载: top命令的顶部显示与uptime命令相似的输出 这些字段显示: 当前时间 系统已运行的时间 当前登录用户的数量 相应最近5.10和15分钟内的平均负 ...
- Python-字符串format方法指定参数
一.字符串的format方法有几种指定参数的方式:(1)按照位置传参(默认方式),传入的参数与{}一一对应(2)关键字传参,关键字(keyword)传递是根据每个参数的名字传递参数.关键字并不用遵守位 ...
- MySQL-报错提示：ERROR 2002 (HY000): Can't connect to local MySQL
场景:通过mysql -h localhost -u root -p 连接MySQL数据库时报错:ERROR 2002 (HY000): Can't connect to local MySQL ...
- Linux dmesg命令使用方法详解
一.命令简介 dmesg(display message)命令用于显示开机信息.kernel 会将开机信息存储在 ring buffer 中.您若是开机时来不及查看信息,可利用 dmesg 来查看. ...
- Spring事务使用注意事项
Spring提供的事务使用起来很方便,一个@Transactional注解就搞定全部,但是如果不注意,也会踩坑 提到事务就应该想到至少以下几点: 1.在事务方法中加锁,可能会导致锁失效 无论是Java ...
- java 注解结合 spring aop 自动输出日志新增拦截器与过滤器
auto-log auto-log 是一款为 java 设计的自动日志监控框架. 前面已经写过了两篇: java 注解结合 spring aop 实现自动输出日志 java 注解结合 spring a ...
- OCR 03: PaddleOCR
Catalog OCR 01: EasyOCR OCR 02: Tesseract-OCR OCR 03: PaddleOCR About Git repository https://github. ...
- elasticsearch should实现or功能，设置minimum_should_match
elasticsearch实现传统数据库中的or功能,需要使用bool下面的should关键字,对于A or B的情况,应该至少返回A和B中的一个,但是如下语句,不仅返回A和B中的至少一个,也返回了没 ...
- 使用JS实现博客搜索关键字高亮
说明 最近博客添加了搜索功能,有个需求是要针对搜索结果中搜索关键字需要高亮显示. 以便用户可以更快速的挑选自己中意的文章. 原理就是在渲染列表数据中给含有关键字的文本标签添加自定义class,渲染完毕 ...