https://dev.mysql.com/doc/refman/5.6/en/subquery-optimization.html

Semi-join in MySQL 5.6

 
MySQL 5.6.5 Development Milestone Release has a whole new set of algorithms for processing subqueries. It is based on transforming a subquery into a semi-join operation, and then treating semi-join like another join operation throughout the optimizer.

A subquery can be transformed to a semi-join if it matches these criteria:

  • The subquery is part of an IN or =ANY predicate. It cannot be e.g. NOT IN.
  • The subquery consists of a single query block (it must not contain UNION).
  • The subquery does not contain GROUP BY or HAVING.
  • The subquery is not implicitly grouped (it contains no aggregate functions).
  • The subquery predicate is part of a WHERE clause.
  • The subquery predicate must not be part of a disjunctive nor a negated search condition.
  • Neither query block contains the STRAIGHT_JOIN qualifier.
  • The statement must be a SELECT or INSERT statement. Semi-joins are not allowed with UPDATE or DELETE statements.
上面这段话的中文翻译
semi join的定义点wiki, MySQL需要满足如下条件,才会转换成Semi-join
#子查询是IN或者=ANY的,不可以是NOT IN
#子查询只能包含一个Query block,不可以有UNION等操作
#子查询不能包含GROUP BY或者HAVING
#子查询不能包含聚合函数
#子查询谓语不可以是外接查询条件或者否定查询条件
#不可以包含STRAIGHT_JOIN限定词
#Semi-join只能用于SELECT或者INSERT,不可用于UPDATE和DELETE
 
Example subquery that can be transformed to a semi-join:
 
  SELECT * FROM nation
  WHERE n_regionkey IN (SELECT r_regionkey FROM region
                        WHERE r_name='AFRICA');

What is a semi-join operation

A semi-join operation is similar to a regular join operation: It takes two (sets of) tables and combines them using a join condition.
 
Like an outer join, a semi-join is a noncommutative join operator. We denote the tables of the containing query block the outer tables and the tables of the subquery the inner tables.
 
Two factors distinguish a semi-join from a regular join:
  • In a semi-join, the inner tables do not cause duplicates in the result.
  • No columns from the inner tables are added to the result of the operation.
This means that the result of the semi-join is a subset of the rows from the outer tables. It also means that much of the special handling of semi-join is about efficiently eliminating duplicates from the inner tables.
 
We can represent the above query using the artificial SEMI JOIN operator as follows:
 
  SELECT nation.*
  FROM nation SEMI JOIN region
       ON nation.n_regionkey = region.r_regionkey AND
          region.r_name='AFRICA';

Semi-join optimization

If possible, a subquery is transformed into a semi-join during the query resolution stage. The query blocks containing the outer tables and the inner tables are then combined into a larger query block.
 
The next step of the optimization is to determine inner tables that are candidate for table pullout. If the inner table of a semi-join has an eq-ref relationship to the outer part of the query, there can be no duplicates from the inner side, and the semi-join can be converted to a regular join. We call that a table pullout, because the table is "pulled out" from the semi-join and joined with the outer tables.
 
The above subquery can be transformed using table pullout, because there is a unique index on the r_regionkey column. The explain result of the query is as follows:
 
| id | select_type | table  | type | possible_keys | key           | key_len | ref                     | rows | Extra       |
+----+-------------+--------+------+---------------+---------------+---------+-------------------------+------+-------------+
| 1 | PRIMARY | region | ALL | PRIMARY | NULL | NULL | NULL | 5 | Using where |
| 1 | PRIMARY | nation | ref | i_n_regionkey | i_n_regionkey | 5 | dbt3.region.r_regionkey | 2 | |
 
 
As you can see, there is no difference between this explain result and the explain result of a regular join between these two tables.
 
The cost-based optimization procedure is then applied to the tables of the query block. Tables taking part in semi-joins are combined in various ways, with the appropriate cost added to the total plan cost. The end result is an optimal table order and an optimal set of semi-join execution strategies.

Semi-join execution strategies

As said before, semi-join is a regular join operation, combined with removal of possible duplicates from the semi-join inner tables. MySQL implements four different semi-join execution strategies, which have different ways of removing duplicates:
  1. FirstMatch
  2. DuplicateWeedout
  3. Materialization
  4. LooseScan
I will handle the FirstMatch strategy in more detail below. The other semi-join strategies will be handled in later blogs. How semi-join strategies can be combined with join buffering will also be handled.

Controlling semi-join strategies

Use of semi-join is controlled with the optimizer_switch flag semijoin:
 
  set optimizer_switch='semijoin=on'
 
This command enables transformation of subqueries into semi-join operations. This flag is on by default in MySQL 5.6.5.
 
In addition, individual semi-join strategies can be turned on and off with the following optimizer_switch flags:
 
  firstmatch, materialization, loosescan
 
These flags enable or disable the use of the respective semi-join strategies. Notice that there is no way to disable the DuplicateWeedout strategy, as this is the "last resort" strategy selected by the cost-based optimizer. There is also no way to disable the TablePullout strategy, when semi-join is enabled.
 
The default for all semi-join related optimizer_switch flags are:
 
  'semijoin=on,firstmatch=on,materialization=on,loosescan=on'
 
Notice also that the optimizer_switch flag materialization has a second meaning: It controls the use of the subquery materialization feature (which should not be confused with the semi-join materialization strategy).

Semi-join FirstMatch strategy

The semi-join FirstMatch strategy executes a subquery very similar to how the IN-TO-EXISTS strategy familiar from earlier versions of MySQL works: for each matching row in the outer table, check for a match in the inner table. When a match is found, return the row from the outer table, otherwise continue scanning the inner table until reaching the end.
 
Here is a query that is processed using FirstMatch strategy:
 
  SELECT * FROM nation
  WHERE n_nationkey IN (SELECT c_nationkey FROM customer);
 
Notice the FirstMatch(nation) indication in explain output, which means that after a match has been found in table customer, the query executor goes back to scan more rows in table nation:
 
| id | select_type | table    | type | possible_keys | key           | key_len | ref                     | rows | Extra                           |
+----+-------------+----------+------+---------------+---------------+---------+-------------------------+------+---------------------------------+
| 1 | PRIMARY | nation | ALL | PRIMARY | NULL | NULL | NULL | 25 | |
| 1 | PRIMARY | customer | ref | i_c_nationkey | i_c_nationkey | 5 | dbt3.nation.n_nationkey | 1115 | Using index; FirstMatch(nation) |
 
Here is the result from IN-TO-EXISTS transformation, which can still be enabled by setting optimizer_switch flags:
 
set optimizer_switch='semijoin=off,materialization=off' :
 
| id | select_type        | table    | type           | possible_keys | key           | key_len | ref  | rows | Extra       |
+----+--------------------+----------+----------------+---------------+---------------+---------+------+------+-------------+
| 1 | PRIMARY | nation | ALL | NULL | NULL | NULL | NULL | 25 | Using where |
| 2 | DEPENDENT SUBQUERY | customer | index_subquery | i_c_nationkey | i_c_nationkey | 5 | func | 1115 | Using index |
 
As this is basically the same strategy that MySQL 5.5 would choose, there is usually no speedup to gain when FirstMatch is chosen. The advantage may however come when FirstMatch is not the most efficient strategy, or when the cost-based optimizer chooses a more efficient table order. The optimizer can determine a better table order, because it will estimate a realistic cost for a semi-join operation. This contrasts MySQL 5.5 where the placement of a subquery in the query plan was strictly rule-based.

Multiple subqueries

It is possible to combine multiple subqueries with AND and still have them transformed to a semi-join:
 
  SELECT * FROM nation
  WHERE n_regionkey IN (SELECT r_regionkey FROM region
                        WHERE r_name='AFRICA') AND
        n_nationkey IN (SELECT c_nationkey FROM customer);
 
The explain output shows one FirstMatch strategy:
 
| id | select_type | table    | type | possible_keys         | key           | key_len | ref                     | rows | Extra                           |
+----+-------------+----------+------+-----------------------+---------------+---------+-------------------------+------+---------------------------------+
| 1 | PRIMARY | region | ALL | PRIMARY | NULL | NULL | NULL | 5 | Using where |
| 1 | PRIMARY | nation | ref | PRIMARY,i_n_regionkey | i_n_regionkey | 5 | dbt3.region.r_regionkey | 2 | |
| 1 | PRIMARY | customer | ref | i_c_nationkey | i_c_nationkey | 5 | dbt3.nation.n_nationkey | 1115 | Using index; FirstMatch(nation) |

Nested subqueries

MySQL can process nested subqueries and transform them into one semi-join operation. Here is a slightly artificial example:
 
  SELECT * FROM nation
  WHERE n_nationkey IN
     (SELECT c_nationkey FROM customer
      WHERE c_acctbal IN
         (SELECT o_totalprice FROM orders
          WHERE o_orderstatus = 'P'));
 
From the explain output, we see a FirstMatch strategy applied to tables customer and order, jumping back to table nation when a match is found:
 
| id | select_type | table    | type | possible_keys | key           | key_len | ref                     | rows    | Extra                           |
+----+-------------+----------+------+---------------+---------------+---------+-------------------------+---------+---------------------------------+
| 1 | PRIMARY | nation | ALL | PRIMARY | NULL | NULL | NULL | 25 | |
| 1 | PRIMARY | customer | ref | i_c_nationkey | i_c_nationkey | 5 | dbt3.nation.n_nationkey | 1115 | |
| 1 | PRIMARY | orders | ALL | NULL | NULL | NULL | NULL | 1486350 | Using where; FirstMatch(nation) |

mysql5.6子查询的优化的更多相关文章

  1. MySQL子查询的优化

    本文基于MySQL5.7.19测试 创建四张表,pt1.pt2表加上主键 mysql> create table t1 (a1 int, b1 int); mysql> create ta ...

  2. MySQL 子查询(四)子查询的优化、将子查询重写为连接

    MySQL 5.7 ref ——13.2.10.10优化子查询 十.子查询的优化 开发正在进行中,因此从长远来看,没有什么优化建议是可靠的.以下列表提供了一些您可能想要使用的有趣技巧.See also ...

  3. 优化系列 | DELETE子查询改写优化

    0.导读 有个采用子查询的DELETE执行得非常慢,改写成SELECT后执行却很快,最后把这个子查询DELETE改写成JOIN优化过程 1.问题描述 朋友遇到一个怪事,一个用子查询的DELETE,执行 ...

  4. Mysql查询优化器之关于子查询的优化

    下面这些sql都含有子查询: mysql> select * from t1 where a in (select a from t2); mysql> select * from (se ...

  5. mysql关联、子查询索引优化

    1.驱动表:加索引不起作用,因为全表扫描.表1 left join 表2 ,此时表1是驱动表 被驱动表:给这个加索引.  关联查询  子查询时 尽量不使用not in 或者not exists 而是用 ...

  6. oracle查询优化之子查询条件优化

    环境:oracle 11g 现有a表与b表通过a01字段关联,要查询出a表的数据在b表没有数据的数据:sql如下 ) ) 因为flag是虚拟字段没有走不了索引导致这条sql执行起来特别慢 310W条数 ...

  7. 深入理解MySql子查询IN的执行和优化

    IN为什么慢? 在应用程序中使用子查询后,SQL语句的查询性能变得非常糟糕.例如: SELECT driver_id FROM driver where driver_id in (SELECT dr ...

  8. MySQL的一次优化记录 (IN子查询和索引优化)

    这两天实习项目遇到一个网页加载巨慢的问题(10多秒),然后定位到是一个MySQL查询特别慢的语句引起的: SELECT * FROM ( SELECT DISTINCT t.vc_date, t.c_ ...

  9. 聊聊MySQL的子查询

    1. 背景 在之前介绍MySQL执行计划的博文中已经谈及了一些关于子查询相关的执行计划与优化.本文将重点介绍MySQL中与子查询相关的内容,设计子查询优化策略,包含半连接子查询的优化与非半连接子查询的 ...

随机推荐

  1. java poi导出EXCEL xls文件代码

    String _currentPage = request.getParameter("currentPage"); Integer currentPage = 0; if(_cu ...

  2. .net RESX资源文件

    RESX资源文件最大的优势就是: 支持多语言 快速创建资源 管理方便 RESX可以支持多语言,Visual Studio编译后会出现附属程序集(satellite assembly),事实上是连接器( ...

  3. lintcode 中等题:Singleton number II 落单的数 II

    题目 落单的数 II 给出3*n + 1 个的数字,除其中一个数字之外其他每个数字均出现三次,找到这个数字. 样例 给出 [1,1,2,3,3,3,2,2,4,1] ,返回 4 挑战 一次遍历,常数级 ...

  4. java io异步

    1.一般来说,可以通过多线程的方式来实现异步 2.同步和异步着重点在于多个任务的执行过程中,一个任务的执行是否会导致整个流程的暂时等待: 3.而阻塞和非阻塞着重点在于发出一个请求操作时,如果进行操作的 ...

  5. 电容值E系列标称方法

    本节首先介绍常用的E系列标称方法,然后介绍电阻.电容器.电感器.二极管的分类.性能和识别方法,以及简单的实用电路. 一.E系列标称方法 厂家生产的电阻器,并不是包含任何阻值,就像人民币,只有1.2.5 ...

  6. 企业用户2T(含秒传),普通用户20G

    周鸿祎一定要看的建议(要求置顶):可以解决本次云盘事件的建议!!! 2016-10-23 20:23 | 复制链接 | 淘帖 461334 本帖最后由 cqthxin 于 2016-10-23 20: ...

  7. Android:查看应用创建的数据库

    每个Android应用程序都可以使用SQLite数据库.它创建的位置在data/data/<项目文件夹>/databases/ 运行后打开,window->show view-> ...

  8. Hibernate 多表关联

    hibernate中可以一次对多个表进行数据插入,这种插入类似 Hibernate的关联映射关系有:多对一 ---- many-to-one一对多 ---- one-to-many一对一 ---- o ...

  9. 我的第一个jquery插件:下拉多选框

    <!DOCTYPE HTML> <html> <head> <title> New Document </title> <meta n ...

  10. WCf的理解

    从 .NET 3.5 开始 WCF 已经支持用 WebHttpBinding 构建 RESTful Web 服务,基于 WCF 框架的 RESTful Web 服务还是建立在 WCF Message ...