union all 优化案例
遇到个子查询嵌套 UNION ALL 的SQL语句很慢，谓词过滤条件不能内推进去，需要优化这段 UNION ALL这块的内容。
UNION ALL 慢SQL：
SELECT * FROM ((SELECT
jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001 AS jfw03001,
jfw03.jfw03002 AS jfw03002,
jfw03.jfw03003 AS jfw03003,
jfw03.jfw03004 AS jfw03004,
jfw03.jfw03005 AS jfw03005,
jfw03.jfw03006 AS jfw03006,
jfw03.jfw03007 AS jfw03007,
jfw03.remark AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM emc jfw03
LEFT JOIN emc old03
ON jfw03.emc_person_id =
old03.emc_person_id
WHERE jfw03.jfw03001 = old03.jfw03001
AND jfw03.handle_mark = '2'
AND old03.handle_mark IN ('1', '3'))
UNION ALL (SELECT jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001,
jfw03.jfw03002,
jfw03.jfw03003,
jfw03.jfw03004,
jfw03.jfw03005,
jfw03.jfw03006,
jfw03.jfw03007,
jfw03.remark,
NULL AS oldId,
NULL AS jfw03001pre,
NULL AS jfw03002pre,
NULL AS jfw03003pre,
NULL AS jfw03004pre,
NULL AS jfw03005pre,
NULL AS jfw03006pre,
NULL AS jfw03007pre,
NULL AS remarkpre
FROM emc jfw03
WHERE jfw03.handle_mark = '2'
AND NOT EXISTS
(SELECT 1
FROM emc old03
WHERE jfw03.emc_person_id =
old03.emc_person_id
AND jfw03.jfw03001 = old03.jfw03001
AND old03.handle_mark IN ('1', '3')))
UNION ALL (SELECT NULL AS newid,
old03.emc_person_id AS bizPersonOid,
NULL AS jfw03001,
NULL AS jfw03002,
NULL AS jfw03003,
NULL AS jfw03004,
NULL AS jfw03005,
NULL AS jfw03006,
NULL AS jfw03007,
NULL AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM emc old03
WHERE old03.handle_mark IN ('1', '3')
AND NOT EXISTS
(SELECT 1
FROM emc jfw03
WHERE jfw03.emc_person_id =
old03.emc_person_id
AND jfw03.jfw03001 = old03.jfw03001
AND jfw03.handle_mark = '2'))) jfw03;
执行计划：
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=0.69..7253737.08 rows=49711 width=274) (actual time=0.252..60906.535 rows=3291407 loops=1)
-> Nested Loop (cost=0.69..2753734.17 rows=49709 width=274) (actual time=0.251..16535.465 rows=49558 loops=1)
-> Seq Scan on emc jfw03 (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.010..5103.042 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 8885720
-> Index Scan using idx_1_2_3_emc on emc old03 (cost=0.69..2.12 rows=1 width=139) (actual time=0.010..0.010 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
-> Nested Loop Anti Join (cost=0.69..1864694.18 rows=1 width=399) (actual time=0.032..16606.236 rows=1065808 loops=1)
-> Seq Scan on emc jfw03_1 (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.013..5171.111 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 8885720
-> Index Only Scan using idx_1_2_3_emc on emc old03_1 (cost=0.69..1.33 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
Heap Fetches: 0
-> Nested Loop Anti Join (cost=0.69..2634563.07 rows=1 width=399) (actual time=0.033..27638.050 rows=2176041 loops=1)
-> Seq Scan on emc old03_2 (cost=0.00..347258.58 rows=2205308 width=139) (actual time=0.014..5510.884 rows=2225077 loops=1)
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 7774923
-> Index Only Scan using idx_1_2_3_emc on emc jfw03_2 (cost=0.69..1.03 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
Heap Fetches: 0
Planning Time: 0.318 ms
Execution Time: 60987.916 ms
(27 行记录)
加个索引看看速度
CREATE INDEX idx_emc_1 ON emc ( handle_mark );
加索引后执行计划
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=21059.67..7011218.65 rows=49969 width=278) (actual time=115.683..46578.675 rows=3291407 loops=1)
-> Nested Loop (cost=21059.67..2655195.15 rows=49967 width=278) (actual time=115.682..11939.294 rows=49558 loops=1)
-> Bitmap Heap Scan on emc jfw03 (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=115.458..417.906 rows=1114280 loops=1)
Recheck Cond: (handle_mark = '2'::bpchar)
Heap Blocks: exact=221151
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..20778.90 rows=1120329 width=0) (actual time=92.887..92.887 rows=1114280 loops=1)
Index Cond: (handle_mark = '2'::bpchar)
-> Index Scan using idx_1_2_3_emc on emc old03 (cost=0.69..2.13 rows=1 width=141) (actual time=0.010..0.010 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
-> Nested Loop Anti Join (cost=21059.67..1766155.15 rows=1 width=401) (actual time=127.721..11690.213 rows=1065808 loops=1)
-> Bitmap Heap Scan on emc jfw03_1 (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=127.640..423.128 rows=1114280 loops=1)
Recheck Cond: (handle_mark = '2'::bpchar)
Heap Blocks: exact=221151
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..20778.90 rows=1120329 width=0) (actual time=105.117..105.117 rows=1114280 loops=1)
Index Cond: (handle_mark = '2'::bpchar)
-> Index Only Scan using idx_1_2_3_emc on emc old03_1 (cost=0.69..1.34 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
Heap Fetches: 0
-> Nested Loop Anti Join (cost=39323.15..2589118.81 rows=1 width=401) (actual time=201.187..22828.908 rows=2176041 loops=1)
-> Bitmap Heap Scan on emc old03_2 (cost=39322.47..289365.69 rows=2222658 width=141) (actual time=201.135..603.170 rows=2225077 loops=1)
Recheck Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
Heap Blocks: exact=222258
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..38766.81 rows=2222658 width=0) (actual time=179.770..179.770 rows=2225077 loops=1)
Index Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
-> Index Only Scan using idx_1_2_3_emc on emc jfw03_2 (cost=0.69..1.02 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
Heap Fetches: 0
Planning Time: 0.373 ms
Execution Time: 46666.296 ms
(33 行记录)
快了14秒，没达到目的效果，无解，只能改写SQL，可以改 FULL JOIN。
等价改写：
SELECT * FROM (
with x as (SELECT * FROM emc where handle_mark IN ('1', '3','2'))
SELECT
jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001 AS jfw03001,
jfw03.jfw03002 AS jfw03002,
jfw03.jfw03003 AS jfw03003,
jfw03.jfw03004 AS jfw03004,
jfw03.jfw03005 AS jfw03005,
jfw03.jfw03006 AS jfw03006,
jfw03.jfw03007 AS jfw03007,
jfw03.remark AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM (SELECT * FROM x WHERE handle_mark IN ('2')) jfw03
full join (SELECT * FROM x WHERE handle_mark IN ('1', '3')) old03
ON jfw03.emc_person_id = old03.emc_person_id AND jfw03.jfw03001 = old03.jfw03001);
等价改写计划：
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------
Merge Full Join (cost=10000479890.42..10000480406.20 rows=33430 width=1844) (actual time=9433.530..10954.254 rows=3291407 loops=1)
Merge Cond: ((x.emc_person_id = x_1.emc_person_id) AND (x.jfw03001 = x_1.jfw03001))
CTE x
-> Bitmap Heap Scan on emc (cost=57545.45..325771.51 rows=3342986 width=145) (actual time=268.766..721.294 rows=3339357 loops=1)
Recheck Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
Heap Blocks: exact=222260
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..56709.71 rows=3342986 width=0) (actual time=246.952..246.953 rows=3339357 loops=1)
Index Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
-> Sort (cost=76389.65..76431.43 rows=16715 width=924) (actual time=3818.370..4021.726 rows=1114280 loops=1)
Sort Key: x.emc_person_id, x.jfw03001
Sort Method: quicksort Memory: 345133kB
-> CTE Scan on x (cost=0.00..75217.18 rows=16715 width=924) (actual time=268.780..1391.594 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 2225077
-> Sort (cost=77729.26..77812.83 rows=33430 width=924) (actual time=5615.144..5949.562 rows=2225599 loops=1)
Sort Key: x_1.emc_person_id, x_1.jfw03001
Sort Method: quicksort Memory: 676056kB
-> CTE Scan on x x_1 (cost=0.00..75217.18 rows=33430 width=924) (actual time=0.007..360.353 rows=2225077 loops=1)
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1114280
Planning Time: 0.138 ms
Execution Time: 11101.829 ms
(22 行记录)
改写后11s就能跑出结果，符合性能优化目标。
这条改写SQL留个坑，不完全等价，完全等价的没放出来，有兴趣的同学可以思考下，自己做做实验。
union all 优化案例的更多相关文章
- Hive优化案例
1.Hadoop计算框架的特点 数据量大不是问题,数据倾斜是个问题. jobs数比较多的作业效率相对比较低,比如即使有几百万的表,如果多次关联多次汇总,产生十几个jobs,耗时很长.原因是map re ...
- 数据库优化案例——————某市中心医院HIS系统
记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也开始分享自己做的优化案例. 最近一直很忙,博客产出也少的 ...
- mysql优化案例
MySQL优化案例 Mysql5.1大表分区效率测试 Mysql5.1大表分区效率测试MySQL | add at 2009-03-27 12:29:31 by PConline | view:60, ...
- SQL 优化案例 1
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
- 老李案例分享：Weblogic性能优化案例
老李案例分享:Weblogic性能优化案例 POPTEST的测试技术交流qq群:450192312 网站应用首页大小在130K左右,在之前的测试过程中,其百用户并发的平均响应能力在6.5秒,性能优化后 ...
- 数据库优化案例——————某知名零售企业ERP系统
写在前面 记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也分享自己做的优化案例. 之前分享过OA系统.H ...
- Spark集群之yarn提交作业优化案例
Spark集群之yarn提交作业优化案例 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.启动Hadoop集群 1>.自定义批量管理脚本 [yinzhengjie@s101 ...
- MySQL参数优化案例
环境介绍 优化层级与指导思想 优化过程 最小化安装情况下的性能表现 优化innodb_buffer_pool_size 优化innodb_log_files_in_group&innodb_l ...
- MySQL的索引单表优化案例分析
建表 建立本次优化案例中所需的数据库及数据表 CREATE DATABASE db0206; USE db0206; CREATE TABLE `db0206`.`article`( `id` INT ...
- SQL 优化案例
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
随机推荐
- ElasticSearch7.3学习(十七)----搜索结果字段解析及time_out字段解析
1.搜索结果字段解析 首先插入一条测试数据 PUT /my_index/_doc/1 { "title": "2019-09-10" } 然后无条件搜索所有 G ...
- 零基础入门Vue之皇帝的新衣——样式绑定
回顾 大致掌握了上一节的 插值语法 我已经可以把想要的数据显示到页面上,并且仅需要修改变量,页面就会跟着实时改变 但如果对于已经熟悉前端的人来说,单单有数据还是不太行,还需要css对数据进行样式的修饰 ...
- 小知识：统计Oracle的日归档量
首先这对于Oracle DBA来说是个初级问题,即使不熟悉的初级DBA也可以快速在网上搜索到现成的SQL语句. 网上搜到的查询SQL基本类似这样的逻辑: select trunc(completion ...
- 好书推荐之《码出高效》、《阿里巴巴JAVA开发手册》
好评如潮 <阿里巴巴Java开发手册> 简介 <阿里巴巴Java开发手册>的愿景是码出高效,码出质量.它结合作者的开发经验和架构历程,提炼阿里巴巴集团技术团队的集体编程经验和软 ...
- google recaptcha 谷歌人机身份验证超详细使用教程，前端/后端集成说明
壹 ❀ 引 在日常页面交互中,验证码使用是极为频繁的,登录注册验证,非机器人操作验证等等,它遍布于每一个网站.说到验证码实现,Goole Recaptcha是一个非常不错的选择,那么希望通过本文的使用 ...
- STM32F103C8T6与W5500的运行示例
模块说明 W5500的厂商是韩国WIZnet, 特性如下 全硬件TCP/IP协议栈: TCP,UDP,ICMP,IPv4,ARP,IGMP,PPPoE -- 注意只有IPv4 支持SPI模式0,3, ...
- Java中各种比较对象方式对比
1.介绍 比较对象是面向对象编程语言的一个基本特征.在本教程中,我们将介绍Java语言的一些特性,这些特性允许我们比较对象.此外,我们还将研究外部库中的这些特性. 2.==和!=操作符 让我们从==和 ...
- Java Socket设置timeout几种常用方式总结
原文链接:https://my.oschina.net/shipley/blog/715196 最近碰到读取第三方提供socket接口需要设置超时时间问题,特此记录一下.原文中有几处拼写错误顺便改掉了 ...
- 《深入理解JAVA虚拟机》(一) JVM 结构 + 栈帧 详解
1.程序计数器(Program Counter Register) 线程独有,每个线程都有自己的计数器:由于CPU的任意时刻只能执行所有线程中的一条,所以需要使用程序计数器来支持J ...
- Google C++ 风格指南记录
最近在看谷歌的 C++ 风格指南发现了一些有意思的知识点,遂记录下 1. 第六章第二小节介绍了右值引用 只在定义移动构造函数与移动赋值操作时使用右值引用. 不要使用 std::forward. 定义: ...