union all 优化案例
遇到个子查询嵌套 UNION ALL 的SQL语句很慢，谓词过滤条件不能内推进去，需要优化这段 UNION ALL这块的内容。
UNION ALL 慢SQL：
SELECT * FROM ((SELECT
jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001 AS jfw03001,
jfw03.jfw03002 AS jfw03002,
jfw03.jfw03003 AS jfw03003,
jfw03.jfw03004 AS jfw03004,
jfw03.jfw03005 AS jfw03005,
jfw03.jfw03006 AS jfw03006,
jfw03.jfw03007 AS jfw03007,
jfw03.remark AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM emc jfw03
LEFT JOIN emc old03
ON jfw03.emc_person_id =
old03.emc_person_id
WHERE jfw03.jfw03001 = old03.jfw03001
AND jfw03.handle_mark = '2'
AND old03.handle_mark IN ('1', '3'))
UNION ALL (SELECT jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001,
jfw03.jfw03002,
jfw03.jfw03003,
jfw03.jfw03004,
jfw03.jfw03005,
jfw03.jfw03006,
jfw03.jfw03007,
jfw03.remark,
NULL AS oldId,
NULL AS jfw03001pre,
NULL AS jfw03002pre,
NULL AS jfw03003pre,
NULL AS jfw03004pre,
NULL AS jfw03005pre,
NULL AS jfw03006pre,
NULL AS jfw03007pre,
NULL AS remarkpre
FROM emc jfw03
WHERE jfw03.handle_mark = '2'
AND NOT EXISTS
(SELECT 1
FROM emc old03
WHERE jfw03.emc_person_id =
old03.emc_person_id
AND jfw03.jfw03001 = old03.jfw03001
AND old03.handle_mark IN ('1', '3')))
UNION ALL (SELECT NULL AS newid,
old03.emc_person_id AS bizPersonOid,
NULL AS jfw03001,
NULL AS jfw03002,
NULL AS jfw03003,
NULL AS jfw03004,
NULL AS jfw03005,
NULL AS jfw03006,
NULL AS jfw03007,
NULL AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM emc old03
WHERE old03.handle_mark IN ('1', '3')
AND NOT EXISTS
(SELECT 1
FROM emc jfw03
WHERE jfw03.emc_person_id =
old03.emc_person_id
AND jfw03.jfw03001 = old03.jfw03001
AND jfw03.handle_mark = '2'))) jfw03;
执行计划：
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=0.69..7253737.08 rows=49711 width=274) (actual time=0.252..60906.535 rows=3291407 loops=1)
-> Nested Loop (cost=0.69..2753734.17 rows=49709 width=274) (actual time=0.251..16535.465 rows=49558 loops=1)
-> Seq Scan on emc jfw03 (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.010..5103.042 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 8885720
-> Index Scan using idx_1_2_3_emc on emc old03 (cost=0.69..2.12 rows=1 width=139) (actual time=0.010..0.010 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
-> Nested Loop Anti Join (cost=0.69..1864694.18 rows=1 width=399) (actual time=0.032..16606.236 rows=1065808 loops=1)
-> Seq Scan on emc jfw03_1 (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.013..5171.111 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 8885720
-> Index Only Scan using idx_1_2_3_emc on emc old03_1 (cost=0.69..1.33 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
Heap Fetches: 0
-> Nested Loop Anti Join (cost=0.69..2634563.07 rows=1 width=399) (actual time=0.033..27638.050 rows=2176041 loops=1)
-> Seq Scan on emc old03_2 (cost=0.00..347258.58 rows=2205308 width=139) (actual time=0.014..5510.884 rows=2225077 loops=1)
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 7774923
-> Index Only Scan using idx_1_2_3_emc on emc jfw03_2 (cost=0.69..1.03 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
Heap Fetches: 0
Planning Time: 0.318 ms
Execution Time: 60987.916 ms
(27 行记录)
加个索引看看速度
CREATE INDEX idx_emc_1 ON emc ( handle_mark );
加索引后执行计划
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=21059.67..7011218.65 rows=49969 width=278) (actual time=115.683..46578.675 rows=3291407 loops=1)
-> Nested Loop (cost=21059.67..2655195.15 rows=49967 width=278) (actual time=115.682..11939.294 rows=49558 loops=1)
-> Bitmap Heap Scan on emc jfw03 (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=115.458..417.906 rows=1114280 loops=1)
Recheck Cond: (handle_mark = '2'::bpchar)
Heap Blocks: exact=221151
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..20778.90 rows=1120329 width=0) (actual time=92.887..92.887 rows=1114280 loops=1)
Index Cond: (handle_mark = '2'::bpchar)
-> Index Scan using idx_1_2_3_emc on emc old03 (cost=0.69..2.13 rows=1 width=141) (actual time=0.010..0.010 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
-> Nested Loop Anti Join (cost=21059.67..1766155.15 rows=1 width=401) (actual time=127.721..11690.213 rows=1065808 loops=1)
-> Bitmap Heap Scan on emc jfw03_1 (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=127.640..423.128 rows=1114280 loops=1)
Recheck Cond: (handle_mark = '2'::bpchar)
Heap Blocks: exact=221151
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..20778.90 rows=1120329 width=0) (actual time=105.117..105.117 rows=1114280 loops=1)
Index Cond: (handle_mark = '2'::bpchar)
-> Index Only Scan using idx_1_2_3_emc on emc old03_1 (cost=0.69..1.34 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1
Heap Fetches: 0
-> Nested Loop Anti Join (cost=39323.15..2589118.81 rows=1 width=401) (actual time=201.187..22828.908 rows=2176041 loops=1)
-> Bitmap Heap Scan on emc old03_2 (cost=39322.47..289365.69 rows=2222658 width=141) (actual time=201.135..603.170 rows=2225077 loops=1)
Recheck Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
Heap Blocks: exact=222258
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..38766.81 rows=2222658 width=0) (actual time=179.770..179.770 rows=2225077 loops=1)
Index Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
-> Index Only Scan using idx_1_2_3_emc on emc jfw03_2 (cost=0.69..1.02 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
Heap Fetches: 0
Planning Time: 0.373 ms
Execution Time: 46666.296 ms
(33 行记录)
快了14秒，没达到目的效果，无解，只能改写SQL，可以改 FULL JOIN。
等价改写：
SELECT * FROM (
with x as (SELECT * FROM emc where handle_mark IN ('1', '3','2'))
SELECT
jfw03.id AS newid,
jfw03.emc_person_id AS bizPersonOid,
jfw03.jfw03001 AS jfw03001,
jfw03.jfw03002 AS jfw03002,
jfw03.jfw03003 AS jfw03003,
jfw03.jfw03004 AS jfw03004,
jfw03.jfw03005 AS jfw03005,
jfw03.jfw03006 AS jfw03006,
jfw03.jfw03007 AS jfw03007,
jfw03.remark AS remark,
old03.id AS oldId,
old03.jfw03001 AS jfw03001pre,
old03.jfw03002 AS jfw03002pre,
old03.jfw03003 AS jfw03003pre,
old03.jfw03004 AS jfw03004pre,
old03.jfw03005 AS jfw03005pre,
old03.jfw03006 AS jfw03006pre,
old03.jfw03007 AS jfw03007pre,
old03.remark AS remarkpre
FROM (SELECT * FROM x WHERE handle_mark IN ('2')) jfw03
full join (SELECT * FROM x WHERE handle_mark IN ('1', '3')) old03
ON jfw03.emc_person_id = old03.emc_person_id AND jfw03.jfw03001 = old03.jfw03001);
等价改写计划：
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------
Merge Full Join (cost=10000479890.42..10000480406.20 rows=33430 width=1844) (actual time=9433.530..10954.254 rows=3291407 loops=1)
Merge Cond: ((x.emc_person_id = x_1.emc_person_id) AND (x.jfw03001 = x_1.jfw03001))
CTE x
-> Bitmap Heap Scan on emc (cost=57545.45..325771.51 rows=3342986 width=145) (actual time=268.766..721.294 rows=3339357 loops=1)
Recheck Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
Heap Blocks: exact=222260
-> Bitmap Index Scan on idx_emc_1 (cost=0.00..56709.71 rows=3342986 width=0) (actual time=246.952..246.953 rows=3339357 loops=1)
Index Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
-> Sort (cost=76389.65..76431.43 rows=16715 width=924) (actual time=3818.370..4021.726 rows=1114280 loops=1)
Sort Key: x.emc_person_id, x.jfw03001
Sort Method: quicksort Memory: 345133kB
-> CTE Scan on x (cost=0.00..75217.18 rows=16715 width=924) (actual time=268.780..1391.594 rows=1114280 loops=1)
Filter: (handle_mark = '2'::bpchar)
Rows Removed by Filter: 2225077
-> Sort (cost=77729.26..77812.83 rows=33430 width=924) (actual time=5615.144..5949.562 rows=2225599 loops=1)
Sort Key: x_1.emc_person_id, x_1.jfw03001
Sort Method: quicksort Memory: 676056kB
-> CTE Scan on x x_1 (cost=0.00..75217.18 rows=33430 width=924) (actual time=0.007..360.353 rows=2225077 loops=1)
Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
Rows Removed by Filter: 1114280
Planning Time: 0.138 ms
Execution Time: 11101.829 ms
(22 行记录)
改写后11s就能跑出结果，符合性能优化目标。
这条改写SQL留个坑，不完全等价，完全等价的没放出来，有兴趣的同学可以思考下，自己做做实验。
union all 优化案例的更多相关文章
- Hive优化案例
1.Hadoop计算框架的特点 数据量大不是问题,数据倾斜是个问题. jobs数比较多的作业效率相对比较低,比如即使有几百万的表,如果多次关联多次汇总,产生十几个jobs,耗时很长.原因是map re ...
- 数据库优化案例——————某市中心医院HIS系统
记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也开始分享自己做的优化案例. 最近一直很忙,博客产出也少的 ...
- mysql优化案例
MySQL优化案例 Mysql5.1大表分区效率测试 Mysql5.1大表分区效率测试MySQL | add at 2009-03-27 12:29:31 by PConline | view:60, ...
- SQL 优化案例 1
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
- 老李案例分享：Weblogic性能优化案例
老李案例分享:Weblogic性能优化案例 POPTEST的测试技术交流qq群:450192312 网站应用首页大小在130K左右,在之前的测试过程中,其百用户并发的平均响应能力在6.5秒,性能优化后 ...
- 数据库优化案例——————某知名零售企业ERP系统
写在前面 记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也分享自己做的优化案例. 之前分享过OA系统.H ...
- Spark集群之yarn提交作业优化案例
Spark集群之yarn提交作业优化案例 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.启动Hadoop集群 1>.自定义批量管理脚本 [yinzhengjie@s101 ...
- MySQL参数优化案例
环境介绍 优化层级与指导思想 优化过程 最小化安装情况下的性能表现 优化innodb_buffer_pool_size 优化innodb_log_files_in_group&innodb_l ...
- MySQL的索引单表优化案例分析
建表 建立本次优化案例中所需的数据库及数据表 CREATE DATABASE db0206; USE db0206; CREATE TABLE `db0206`.`article`( `id` INT ...
- SQL 优化案例
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
随机推荐
- [JVM]逃逸分析
逃逸分析 JVM的内存分配策略 首先回顾一下JVM的内存分配策略. JVM的内存包括方法区.堆.虚拟机栈.本地方法栈.程序计数器.一般情况下JVM运行时的数据都是存在栈和堆上的.栈用来存放一些基本变量 ...
- 使用lvs实现ftp的负载均衡
操作系统:CentOS6.5_x64 问题描述 使用lvs实现ftp的负载均衡 为了使模型足够简单,这里只实现了loadblance,HA并未实现,可以借助keepalived实现. 具体实现 hos ...
- 【分布式】load balance 04-java 从零手写实现负载均衡
负载均衡系列专题 01-负载均衡基础知识 02-一致性 hash 原理 03-一致性哈希算法 java 实现 04-负载均衡算法 java 实现 本节我们来看一下如何实现一负载均衡框架. 源码 核心接 ...
- STC系列8位MCU在Windows下的开发
STC系列 MCS-51 8位MCU 简介 STC的8位MCU有89/90/10/11/12/15/8(A/F/G/H)这几个大系列, 都是8051衍生的8位单片机, 每个系列的特点如下 STC89系 ...
- Mybatis的缓存过期机制和RedisCache
MyBatis的缓存过期机制, flushInterval参数 在实际测试中, 发现Redis中的缓存数据TTL为-1, 在Hash中的key也无过期时间信息, 怀疑RedisCache的实现是否能正 ...
- python中矩阵合并、拼接、组合
1 numpy数组 1.1 append() import numpy as np a=np.array([[1,3],[5,7]]) b=np.array([[2,4],[6,8]]) c=np.a ...
- linux 快速安装LAMP教程
最近学习linux,安装lamp遇到一些问题,记录下来分享一下: ------------------------------------------------------------------- ...
- win10下开启IIS部署网站
今天测试一个有关IIS的安全漏洞用了下win10下的IIS,记录一下,供网友参考. 1.启用IIS 依次点开: 控制面板-----程序----启用或关闭Windows功能 以下图片中打钩的地方全勾上. ...
- Java集合框架学习（五） ArrayList详解
Arraylist介绍 Arraylist类实现了List接口. public class ArrayList<E> extends AbstractList<E> imple ...
- OpenCV开发笔记（六十一）：红胖子8分钟带你深入了解Shi-Tomasi角点检测（图文并茂+浅显易懂+程序源码）
若该文为原创文章,未经允许不得转载原博主博客地址:https://blog.csdn.net/qq21497936原博主博客导航:https://blog.csdn.net/qq21497936/ar ...