记录一次 postgresql 优化案例（嵌套循环改HASH JOIN ）

今天同事给我一条5秒的SQL看看能不能优化。

表数据量：

select count(1) from AAAA

union all

select count(1) from XXXXX;

  count

---------

 1000001

  998000

(2 rows)

原始SQL：

SELECT A1.PK_DEPT, A1.ENABLESTATE

FROM AAAA A1

         JOIN AAAA A2 ON A1.PK_DEPT = A2.PK_DEPT

WHERE ((A1.PK_GROUP = 'Group9' AND A1.PK_ORG IN ('Org9')))

  AND (A1.PK_DEPT IN (SELECT T1.ORGID

                      FROM XXXXX T1

                               INNER JOIN (SELECT (CASE WHEN ORGID3 IS NULL THEN ORGID2 ELSE ORGID3 END) ORGID

                                           FROM XXXXX

                                           WHERE ORGID = 'Org108') T2

                                          ON (T1.ORGID2 = T2.ORGID OR T1.ORGID3 = T2.ORGID)))

  AND (A1.ENABLESTATE IN (2))

ORDER BY A1.PK_DEPT, A1.ENABLESTATE;

执行计划：

                                                                                                                  QUERY PLAN                                                           

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------

 Sort  (cost=16098.39..16098.40 rows=1 width=13) (actual time=5435.964..5454.953 rows=1000000 loops=1)

   Sort Key: a1.pk_dept

   Sort Method: quicksort  Memory: 79264kB

   ->  Nested Loop Semi Join  (cost=1039.46..16098.38 rows=1 width=13) (actual time=0.389..5338.781 rows=1000000 loops=1)

         Join Filter: ((a1.pk_dept)::text = (t1.orgid)::text)

         ->  Gather  (cost=1038.61..16089.43 rows=1 width=22) (actual time=0.368..55.998 rows=1000000 loops=1)

               Workers Planned: 2

               Workers Launched: 2

               ->  Hash Join  (cost=38.61..15089.33 rows=1 width=22) (actual time=0.246..49.481 rows=333333 loops=3)

                     Hash Cond: ((a2.pk_dept)::text = (a1.pk_dept)::text)

                     ->  Parallel Seq Scan on aaaa a2  (cost=0.00..13491.33 rows=415833 width=9) (actual time=0.009..14.206 rows=332667 loops=3)

                     ->  Hash  (cost=38.60..38.60 rows=1 width=13) (actual time=0.193..0.195 rows=1000 loops=3)

                           Buckets: 1024  Batches: 1  Memory Usage: 51kB

                           ->  Bitmap Heap Scan on aaaa a1  (cost=34.58..38.60 rows=1 width=13) (actual time=0.068..0.142 rows=1000 loops=3)

                                 Recheck Cond: (((pk_org)::text = 'Org9'::text) AND ((pk_group)::text = 'Group9'::text))

                                 Filter: (enablestate = 2)

                                 Heap Blocks: exact=9

                                 ->  BitmapAnd  (cost=34.58..34.58 rows=1 width=0) (actual time=0.062..0.063 rows=0 loops=3)

                                       ->  Bitmap Index Scan on idx_aaaa_pkorg  (cost=0.00..17.17 rows=632 width=0) (actual time=0.031..0.031 rows=1000 loops=3)

                                             Index Cond: ((pk_org)::text = 'Org9'::text)

                                       ->  Bitmap Index Scan on idx_aaaa_pkgroup  (cost=0.00..17.17 rows=632 width=0) (actual time=0.030..0.030 rows=1000 loops=3)

                                             Index Cond: ((pk_group)::text = 'Group9'::text)

         ->  Nested Loop  (cost=0.85..8.94 rows=1 width=9) (actual time=0.005..0.005 rows=1 loops=1000000)

               Join Filter: (((t1.orgid2)::text = (CASE WHEN (xxxxx.orgid3 IS NULL) THEN xxxxx.orgid2 ELSE xxxxx.orgid3 END)::text) OR ((t1.orgid3)::text = (CASE WHEN (xxxxx.orgid3 IS

 NULL) THEN xxxxx.orgid2 ELSE xxxxx.orgid3 END)::text))

               ->  Index Scan using idx_xxxxx_orgid on xxxxx t1  (cost=0.42..0.49 rows=1 width=27) (actual time=0.003..0.003 rows=1 loops=1000000)

                     Index Cond: ((orgid)::text = (a2.pk_dept)::text)

               ->  Index Scan using idx_3_4 on xxxxx  (cost=0.42..8.44 rows=1 width=18) (actual time=0.002..0.002 rows=1 loops=1000000)

                     Index Cond: ((orgid)::text = 'Org108'::text)

 Planning Time: 0.326 ms

 Execution Time: 5478.431 ms

(30 rows)

如果经常做优化的同学对于简单的SQL，相信可以使用瞪眼大法基本定位到语句慢的位置 
AAAA、XXXXX 两张表都不算是小表，数据量在百万级别，在执行计划中，谓词都是有索引进行过滤的，
但是两张表关联以后却走了嵌套循环（Nested Loop），导致t1表和t2表关联后的内联视图作为被驱动表被干了1000000次，很明显这个执行计划是错误的。
最主要原因就是关联条件是or的逻辑条件。

可以通过等价改写来搞一下这条SQL，让 Nested Loop 改变成 hash join 
等价改写SQL：

SELECT A1.PK_DEPT, A1.ENABLESTATE

FROM AAAA A1

         JOIN AAAA A2 ON A1.PK_DEPT = A2.PK_DEPT

         JOIN (SELECT T1.ORGID

               FROM XXXXX T1

                        INNER JOIN (SELECT COALESCE(ORGID3, ORGID2) ORGID FROM XXXXX WHERE ORGID = 'Org108') T2

                                   ON T1.ORGID2 = T2.ORGID

               UNION

               SELECT T1.ORGID

               FROM XXXXX T1

                        INNER JOIN (SELECT COALESCE(ORGID3, ORGID2) ORGID FROM XXXXX WHERE ORGID = 'Org108') T2

                                   ON T1.ORGID3 = T2.ORGID) X ON A1.PK_DEPT = X.ORGID

WHERE ((A1.PK_GROUP = 'Group9' AND A1.PK_ORG IN ('Org9')))

  AND (A1.ENABLESTATE IN (2))

ORDER BY A1.PK_DEPT, A1.ENABLESTATE;

改写后执行计划：

                                                                            QUERY PLAN

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

 Nested Loop  (cost=1072.44..16123.32 rows=1 width=13) (actual time=190.512..312.537 rows=1000000 loops=1)

   Join Filter: ((a1.pk_dept)::text = (t1.orgid)::text)

   Rows Removed by Join Filter: 3000000

   ->  Unique  (cost=33.83..33.84 rows=2 width=516) (actual time=0.073..0.086 rows=4 loops=1)

         ->  Sort  (cost=33.83..33.84 rows=2 width=516) (actual time=0.072..0.079 rows=5 loops=1)

               Sort Key: t1.orgid

               Sort Method: quicksort  Memory: 25kB

               ->  Append  (cost=0.85..33.82 rows=2 width=516) (actual time=0.037..0.068 rows=5 loops=1)

                     ->  Nested Loop  (cost=0.85..16.90 rows=1 width=9) (actual time=0.037..0.045 rows=2 loops=1)

                           ->  Index Scan using idx_3_4 on xxxxx  (cost=0.42..8.44 rows=1 width=18) (actual time=0.022..0.023 rows=2 loops=1)

                                 Index Cond: ((orgid)::text = 'Org108'::text)

                           ->  Index Scan using idx_xxxxx_orgid2 on xxxxx t1  (cost=0.42..8.44 rows=1 width=18) (actual time=0.009..0.009 rows=1 loops=2)

                                 Index Cond: ((orgid2)::text = (COALESCE(xxxxx.orgid3, xxxxx.orgid2))::text)

                     ->  Nested Loop  (cost=0.85..16.90 rows=1 width=9) (actual time=0.014..0.021 rows=3 loops=1)

                           ->  Index Scan using idx_3_4 on xxxxx xxxxx_1  (cost=0.42..8.44 rows=1 width=18) (actual time=0.003..0.003 rows=2 loops=1)

                                 Index Cond: ((orgid)::text = 'Org108'::text)

                           ->  Index Scan using idx_xxxxx_orgid3 on xxxxx t1_1  (cost=0.42..8.44 rows=1 width=18) (actual time=0.008..0.008 rows=2 loops=2)

                                 Index Cond: ((orgid3)::text = (COALESCE(xxxxx_1.orgid3, xxxxx_1.orgid2))::text)

   ->  Materialize  (cost=1038.61..16089.43 rows=1 width=22) (actual time=0.096..43.254 rows=1000000 loops=4)

         ->  Gather  (cost=1038.61..16089.43 rows=1 width=22) (actual time=0.384..44.877 rows=1000000 loops=1)

               Workers Planned: 2

               Workers Launched: 2

               ->  Hash Join  (cost=38.61..15089.33 rows=1 width=22) (actual time=0.257..48.484 rows=333333 loops=3)

                     Hash Cond: ((a2.pk_dept)::text = (a1.pk_dept)::text)

                     ->  Parallel Seq Scan on aaaa a2  (cost=0.00..13491.33 rows=415833 width=9) (actual time=0.009..14.053 rows=332667 loops=3)

                     ->  Hash  (cost=38.60..38.60 rows=1 width=13) (actual time=0.217..0.219 rows=1000 loops=3)

                           Buckets: 1024  Batches: 1  Memory Usage: 51kB

                           ->  Bitmap Heap Scan on aaaa a1  (cost=34.58..38.60 rows=1 width=13) (actual time=0.085..0.160 rows=1000 loops=3)

                                 Recheck Cond: (((pk_org)::text = 'Org9'::text) AND ((pk_group)::text = 'Group9'::text))

                                 Filter: (enablestate = 2)

                                 Heap Blocks: exact=9

                                 ->  BitmapAnd  (cost=34.58..34.58 rows=1 width=0) (actual time=0.077..0.078 rows=0 loops=3)

                                       ->  Bitmap Index Scan on idx_aaaa_pkorg  (cost=0.00..17.17 rows=632 width=0) (actual time=0.039..0.039 rows=1000 loops=3)

                                             Index Cond: ((pk_org)::text = 'Org9'::text)

                                       ->  Bitmap Index Scan on idx_aaaa_pkgroup  (cost=0.00..17.17 rows=632 width=0) (actual time=0.035..0.036 rows=1000 loops=3)

                                             Index Cond: ((pk_group)::text = 'Group9'::text)

 Planning Time: 0.236 ms

 Execution Time: 337.656 ms

(38 rows)

差集比对

SELECT A1.PK_DEPT, A1.ENABLESTATE

FROM AAAA A1

         JOIN AAAA A2 ON A1.PK_DEPT = A2.PK_DEPT

         JOIN (SELECT T1.ORGID

               FROM XXXXX T1

                        INNER JOIN (SELECT COALESCE(ORGID3, ORGID2) ORGID FROM XXXXX WHERE ORGID = 'Org108') T2

                                   ON T1.ORGID2 = T2.ORGID

               UNION

               SELECT T1.ORGID

               FROM XXXXX T1

                        INNER JOIN (SELECT COALESCE(ORGID3, ORGID2) ORGID FROM XXXXX WHERE ORGID = 'Org108') T2

                                   ON T1.ORGID3 = T2.ORGID) X ON A1.PK_DEPT = X.ORGID

WHERE ((A1.PK_GROUP = 'Group9' AND A1.PK_ORG IN ('Org9')))

  AND (A1.ENABLESTATE IN (2))

EXCEPT

SELECT A1.PK_DEPT, A1.ENABLESTATE

FROM AAAA A1

         JOIN AAAA A2 ON A1.PK_DEPT = A2.PK_DEPT

WHERE ((A1.PK_GROUP = 'Group9' AND A1.PK_ORG IN ('Org9')))

  AND (A1.PK_DEPT IN (SELECT T1.ORGID

                      FROM XXXXX T1

                               INNER JOIN (SELECT (CASE WHEN ORGID3 IS NULL THEN ORGID2 ELSE ORGID3 END) ORGID

                                           FROM XXXXX

                                           WHERE ORGID = 'Org108') T2

                                          ON (T1.ORGID2 = T2.ORGID OR T1.ORGID3 = T2.ORGID))

    )

  AND (A1.ENABLESTATE IN (2));

 pk_dept | enablestate

---------+-------------

(0 rows)

Time: 5740.419 ms (00:05.740)

可以看到改写完以后，A1和A2表已经被物化，t1 内联视图作为一个整体和A1和A2进行关联，SQL执行时间也从5S降到337ms就能出结果。

通过差集比对，两条SQL是等价的，本次案例的SQL优化已完成

记录一次 postgresql 优化案例（嵌套循环改HASH JOIN ）的更多相关文章

PostgreSQL 涉及复杂视图查询的优化案例
一.前言对于含有union , group by 等的视图,我们称之为复杂视图. 这类的视图会影响优化器对于视图的提升,也就是视图无法与父查询进行合并,从而影响访问路径.连接方法.连接顺序等.本文通 ...
mysql优化案例
MySQL优化案例 Mysql5.1大表分区效率测试 Mysql5.1大表分区效率测试MySQL | add at 2009-03-27 12:29:31 by PConline | view:60, ...
Hive优化案例
1.Hadoop计算框架的特点数据量大不是问题,数据倾斜是个问题. jobs数比较多的作业效率相对比较低,比如即使有几百万的表,如果多次关联多次汇总,产生十几个jobs,耗时很长.原因是map re ...
记一次mysql多表查询(left jion)优化案例
一次mysql多表查询(left jion)优化案例在新上线的供需模块中,发现某一个查询按钮点击后,出不来结果,找到该按钮对应sql手动执行,发现需要20-30秒才能出结果,所以服务端程序判断超时, ...
数据库优化案例——————某市中心医院HIS系统
记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也开始分享自己做的优化案例. 最近一直很忙,博客产出也少的 ...
SQL 优化案例 1
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
数据库优化案例——————某知名零售企业ERP系统
写在前面记得在自己学习数据库知识的时候特别喜欢看案例,因为优化的手段是容易掌握的,但是整体的优化思想是很难学会的.这也是为什么自己特别喜欢看案例,今天也分享自己做的优化案例. 之前分享过OA系统.H ...
MySQL参数优化案例
环境介绍优化层级与指导思想优化过程最小化安装情况下的性能表现优化innodb_buffer_pool_size 优化innodb_log_files_in_group&innodb_l ...
SQL 优化案例
create or replace procedure SP_GET_NEWEST_CAPTCHA( v_ACCOUNT_ID in VARCHAR2, --接收短信的手机号 v_Tail_num i ...
老李案例分享：Weblogic性能优化案例
老李案例分享:Weblogic性能优化案例 POPTEST的测试技术交流qq群:450192312 网站应用首页大小在130K左右,在之前的测试过程中,其百用户并发的平均响应能力在6.5秒,性能优化后 ...

随机推荐

QPushButton按钮的使用
1 import sys 2 from PyQt5.QtCore import * 3 from PyQt5.QtGui import * 4 from PyQt5.QtWidgets import ...
自治系统/自治域和自治系统编号（ASN）
定义: 自治系统或自治域(英文:Autonomous system, AS)是指在互联网中,一个或多个实体管辖下的所有IP网络和路由器的组合,它们对互联网执行共同的路由策略.参看RFC 1930中更新 ...
JavaScript 基础(1) - 笔记
1 JavaScript基础 1.1 JavaScript 是什么 1.JavaScript(是什么?) 是一种运行在客户端(浏览器)的编程语言,实现人机交互效果. 2.作用(做什么?) 网页特效(监 ...
10、Mybatis之缓存
10.1.环境搭建 10.1.1.创建新module 创建名为mybatis_cache的新module,过程参考5.1节 10.1.2.创建Mapper接口和映射文件 package org.rai ...
实在智能TARS-RPA-Agent，业界首发的产品级大模型Agent有何非凡之处？
融合LLM的RPA进化到什么程度? AIGC如何借AI Agent落地? 像生成文本一样生成流程的ChatRPA,能够提升RPA新体验? 边探索边创建的ChatRPA,能否破解RPA与LLM融合难题? ...
HTML视频背景(动态背景)
网页动态背景一般是用视频实现的,能增添网页的感染力,我觉得很好看,也不难,不妨学一下. 先加入下面一串代码: 1 <style> 2 video{ 3 height: 100%; 4 wi ...
The Missing Semester - 第五讲学习笔记（二）
第五讲(二) SSH入门介绍完命令行环境后,这半节主要介绍的是ssh的有关入门知识.SSH是Secure Shell的简称. 课程视频地址:https://www.bilibili.com/vide ...
Solution -「洛谷 P3267」「JLOI 2016」「SHOI 2016」侦察守卫
Description Link. 给你一棵树,放置守卫在某个点上面需要一定代价和一定的有效范围.让你覆盖若干指定点,求最小代价 Solution 算法标签: $\ \ \ \ \ \ \ \ \ $ ...
vue2中使用antv/G6节点内容可滚动的ER图
先举一个栗子: 效果链接:https://code.juejin.cn/pen/7226264955824930816 如果不会请移步到官网的栗子,请点击查看狠人话不多,直接给大家上代码: 整体代码 ...
Oracle字符串函数-Translate()总结
Oracle的Translate(expr,from_string,to_string)是字符串操作函数,实现from_string,to_string字符的一一替换 1)典型示例: select ...

记录一次 postgresql 优化案例（ 嵌套循环改HASH JOIN ）

记录一次 postgresql 优化案例（ 嵌套循环改HASH JOIN ）的更多相关文章

随机推荐

热门专题

记录一次 postgresql 优化案例（嵌套循环改HASH JOIN ）

记录一次 postgresql 优化案例（嵌套循环改HASH JOIN ）的更多相关文章