背景介绍

记录共128W条！
 
SELECT cpe_id, COUNT(*) restarts
FROM business_log
WHERE operate_time>='2012-12-05 00:00:00' AND operate_time<'2018-01-05 00:00:00' AND operate_type=3 AND result=0
GROUP BY cpe_id
 
尝试对原SQL语句进行优化后发现，统计速度依旧没有获得满意的提升。单独运行条件查询语句（不包含GROUP BY和COUNT函数）后发现，查询的结果数据量只有6655条，耗时0.825s；加上统计语句后，时间飙升至3s。

原理

mysql 解释：
MySQL说明文档中关于优化GROUP BY的部分指出：The most general way to satisfy a GROUP BY clause is to scan the whole table and create a new temporary table where all rows from each group are consecutive, and then use this temporary table to discover groups and apply aggregate functions (if any)。即，GROUP BY语句会扫描全表并新建一个临时表用来分组存放数据，然后根据临时表中的分组对数据执行聚合函数。现在的问题聚焦在：如果GROUP BY和WHERE在同一个语句中，这个“全表”指的是物理表还是WHERE过滤后的数据集合？
 
SELECT cpe_id, COUNT(*) restarts
FROM (
SELECT cpe_id
FROM business_log
WHERE operate_time>='2012-12-05 00:00:00' AND operate_time<'2018-01-05 00:00:00' AND operate_type=3 AND result=0
) t
GROUP BY cpe_id
---------------------
如上述语句所示，在查询语句外包了一个统计语句。执行结果：0.851s。时间消耗大幅减少！

结论

利用GROUP BY统计大数据时，应当将查询与统计分离，优化查询语句。
-
 
 

实际业务中

原查询过程
SELECT SUM(a.reportcount)reportcount,a.OrganizationId,a.organizationcode,a.organizationname,org.UpperComCode,org.organizationlevel FROM (SELECT COUNT(1) ReportCount, o.OrganizationCode ,o.organizationname, o.id OrganizationId,o.organizationlevel FROM report r JOIN organization o ON r.OrganizationCode = o.OrganizationCode WHERE o.OrganizationCode in ('44010000','44040000','44006600','44008800','44020000','44050000','44060000','44070000','44080000','44090000','44120000','44150000','44160000','44170000','44180000','44190000','44510000','44520000','44530000','44710000','44940000','44950000') and r.ReportTime>'2019/4/25 0:00:00' and r.ReportReasonSubmitCode in ('A10006','A10007') and r.ReportTime<'2019/5/20 0:00:00' and r.LossTime>'2019/4/11 0:00:00' and r.LossTime<'2019/5/6 0:00:00' and r.InsuranceType in (5,6,7,8) AND r.CatasCollectionId=361 GROUP BY o.OrganizationCode,o.organizationname, o.id,o.organizationlevel union all SELECT COUNT(1) ReportCount, o.OrganizationCode,o.organizationname, o.id OrganizationId,o.organizationlevel FROM report r JOIN organization o ON r.OrganizationCode = o.OrganizationCode WHERE o.UpperComCode IN('44010000','44040000','44006600','44008800','44020000','44050000','44060000','44070000','44080000','44090000','44120000','44150000','44160000','44170000','44180000','44190000','44510000','44520000','44530000','44710000','44940000','44950000')and r.ReportTime>'2019/4/25 0:00:00' and r.ReportReasonSubmitCode in ('A10006','A10007') and r.ReportTime<'2019/5/20 0:00:00' and r.LossTime>'2019/4/11 0:00:00' and r.LossTime<'2019/5/6 0:00:00' and r.InsuranceType in (5,6,7,8) AND r.CatasCollectionId=361 GROUP BY o.OrganizationCode,o.organizationname, o.id ,o.organizationlevel union all SELECT COUNT(1) ReportCount , o.UpperComCode OrganizationCode,o.ParentOrgName OrganizationName, o.ParentOrgId OrganizationId,o.organizationlevel FROM report r JOIN organization o ON r.OrganizationCode = o.OrganizationCode WHERE o.UpperComCode IN(SELECT OrganizationCode FROM organization WHERE UpperComCode IN ('44010000','44040000','44006600','44008800','44020000','44050000','44060000','44070000','44080000','44090000','44120000','44150000','44160000','44170000','44180000','44190000','44510000','44520000','44530000','44710000','44940000','44950000' )and r.ReportTime>'2019/4/25 0:00:00' and r.ReportReasonSubmitCode in ('A10006','A10007') and r.ReportTime<'2019/5/20 0:00:00' and r.LossTime>'2019/4/11 0:00:00' and r.LossTime<'2019/5/6 0:00:00' and r.InsuranceType in (5,6,7,8) AND r.CatasCollectionId=361) GROUP BY o.UpperComCode ,o.ParentOrgName, o.ParentOrgId,o.organizationlevel union all SELECT SUM(reportcount) reportcount , organization.UpperComCode OrganizationCode, organization.ParentOrgName OrganizationName,organization.ParentOrgId OrganizationId,organization.OrganizationLevel FROM ( SELECT COUNT(1) reportcount , o.UpperComCode newcode, o.organizationlevel FROM report r JOIN organization o ON r.OrganizationCode = o.OrganizationCode WHERE o.UpperComCode IN( SELECT OrganizationCode FROM organization WHERE UpperComCode IN(SELECT OrganizationCode FROM organization WHERE UpperComCode IN ('44010000','44040000','44006600','44008800','44020000','44050000','44060000','44070000','44080000','44090000','44120000','44150000','44160000','44170000','44180000','44190000','44510000','44520000','44530000','44710000','44940000','44950000' )and r.ReportTime>'2019/4/25 0:00:00' and r.ReportReasonSubmitCode in ('A10006','A10007') and r.ReportTime<'2019/5/20 0:00:00' and r.LossTime>'2019/4/11 0:00:00' and r.LossTime<'2019/5/6 0:00:00' and r.InsuranceType in (5,6,7,8) AND r.CatasCollectionId=361 ) ) GROUP BY o.UpperComCode, o.organizationlevel ) a JOIN organization ON a.newcode = organization.OrganizationCode GROUP BY organization.UpperComCode , organization.ParentOrgName ,organization.ParentOrgId ,organization.OrganizationLevel ) a JOIN organization org ON a.organizationcode=org.organizationcode GROUP BY a.organizationcode,a.organizationname,org.UpperComCode,org.organizationlevel,a.OrganizationId ORDER BY OrganizationLevel
优化结果
SELECT COUNT(1) AS reportcount,a.OrganizationId,a.organizationcode,a.organizationname,a.UpperComCode,a.organizationlevel FROM (
 
 
SELECT r.*,r.id AS reportid, o.OrganizationCode ,o.organizationname, o.id OrganizationId,o.organizationlevel ,o.UpperComCode
FROM report r JOIN organization o ON r.OrganizationCode = o.OrganizationCode
WHERE o.OrganizationCode IN ('44010000','44040000','44006600','44008800','44020000','44050000','44060000','44070000','44080000','44090000','44120000','44150000','44160000','44170000','44180000','44190000','44510000','44520000','44530000','44710000','44940000','44950000') AND r.ReportTime>'2010/4/25 0:00:00' AND r.ReportReasonSubmitCode IN ('A10006','A10007') AND r.ReportTime<'2019/5/20 0:00:00' AND r.LossTime>'2010/4/11 0:00:00' AND r.LossTime<'2019/5/6 0:00:00' AND r.InsuranceType IN (5,6,7,8) AND r.CatasCollectionId=0
 
UNION ALL
SELECT r.id AS reportid , o.OrganizationCode,o.organizationname, o.id OrganizationId,o.organizationlevel ,o.UpperComCode
FROM report r JOIN organization o ON r.OrganizationCode = o.OrganizationCode
WHERE o.UpperComCode IN('44010000','44040000','44006600','44008800','44020000','44050000','44060000','44070000','44080000','44090000','44120000','44150000','44160000','44170000','44180000','44190000','44510000','44520000','44530000','44710000','44940000','44950000')AND r.ReportTime>'2010/4/25 0:00:00' AND r.ReportReasonSubmitCode IN ('A10006','A10007') AND r.ReportTime<'2019/5/20 0:00:00' AND r.LossTime>'2010/4/11 0:00:00' AND r.LossTime<'2019/5/6 0:00:00' AND r.InsuranceType IN (5,6,7,8) AND r.CatasCollectionId=0
 
UNION ALL
SELECT r.id AS reportid , o.UpperComCode OrganizationCode,o.ParentOrgName OrganizationName, o.ParentOrgId OrganizationId,ou.organizationlevel ,ou.UpperComCode
FROM report r JOIN organization o ON r.OrganizationCode = o.OrganizationCode
JOIN organization ou ON o.UpperComCode=ou.OrganizationCode
WHERE o.UpperComCode IN
(SELECT OrganizationCode FROM organization WHERE UpperComCode IN
('44010000','44040000','44006600','44008800','44020000','44050000','44060000','44070000','44080000','44090000','44120000','44150000','44160000','44170000','44180000','44190000','44510000','44520000','44530000','44710000','44940000','44950000' )AND r.ReportTime>'2010/4/25 0:00:00' AND r.ReportReasonSubmitCode IN ('A10006','A10007') AND r.ReportTime<'2019/5/20 0:00:00' AND r.LossTime>'2010/4/11 0:00:00' AND r.LossTime<'2019/5/6 0:00:00' AND r.InsuranceType IN (5,6,7,8) AND r.CatasCollectionId=0)
 
) a
GROUP BY a.OrganizationCode,a.organizationname, a.OrganizationId,a.organizationlevel,a.UpperComCode
ORDER BY a.organizationlevel
 

mysql 之 group by 性能优化 查询与统计分离的更多相关文章

  1. Web 性能优化： 使用 Webpack 分离数据的正确方法

    摘要: Webpack骚操作. 原文:Web 性能优化: 使用 Webpack 分离数据的正确方法 作者:前端小智 Fundebug经授权转载,版权归原作者所有. 制定向用户提供文件的最佳方式可能是一 ...

  2. MySQL查询语句执行过程及性能优化-查询过程及优化方法（JOIN/ORDER BY）

    在上一篇文章MySQL查询语句执行过程及性能优化-基本概念和EXPLAIN语句简介中介绍了EXPLAIN语句,并举了一个慢查询例子:

  3. MySQL有关Group By的优化

    昨天我写了有关MySQL的loose index scan的相关博文(http://www.cnblogs.com/wingsless/p/5037625.html),后来我发现上次提到的那个优化方法 ...

  4. Mysql Join语法以及性能优化

    引言 内外联结的区别是内联结将去除所有不符合条件的记录,而外联结则保留其中部分.外左联结与外右联结的区别在于如果用A左联结B则A中所有记录都会保留在结果中,此时B中只有符合联结条件的记录,而右联结相反 ...

  5. mysql问题排查与性能优化

     MySQL 问题排查都有哪些手段? 使用 show processlist 命令查看当前所有连接信息. 使用 explain 命令查询 SQL 语句执行计划. 开启慢查询日志,查看慢查询的 SQL. ...

  6. mysql之数据库添加索引优化查询效率

    项目中如果表中的数据过多的话,会影响查询的效率,那么我们需要想办法优化查询,通常添加索引就是我们的选择之一: 1.添加PRIMARY KEY(主键索引) mysql>ALTER TABLE `t ...

  7. mysql use index （） 优化查询的例子

    USE INDEX在你查询语句中表名的后面,添加 USE INDEX 来提供你希望 MySQ 去参考的索引列表,就可以让 MySQL 不再考虑其他可用的索引.Eg:SELECT * FROM myta ...

  8. mysql explain的使用（优化查询）

    explain显示了mysql如何使用索引来处理select语句以及连接表.可以帮助选择更好的索引和写出更优化的查询语句. 1.创建数据库 创建的sql语句如下: /* Navicat MySQL D ...

  9. SqlServer性能优化 查询和索引优化（十二）

    查询优化的过程: 查询优化: 功能:分析语句后最终生成执行计划 分析:获取操作语句参数 索引选择 Join算法选择 创建测试的表: select * into EmployeeOp from Adve ...

随机推荐

  1. 如何去掉drwxr-xr-x@中的@符号Linux文件扩展信息

    如何去掉drwxr-xr-x@中的@符号Linux文件扩展信息ls -lart drwxrwxrwx@ 10 rlanffy staff 340B 3 6 2015 files-rwxrwxrwx@ ...

  2. linux大小写转换

    [root@ob2 mytmp]# cat aa2.txt|sed 's#[a-z]#\u&#g'ETH0      LINK ENCAP:ETHERNET  HWADDR 00:0C:29: ...

  3. C# 字典 Dictionary 遍历

    using System; using System.Collections.Generic; public class Example { public static void Main() { / ...

  4. DataGridView使用技巧七：列顺序的调整、操作行头列头的标题

    一.列顺序的调整 设定DataGridView的AllowUserToOrderColumns为True的时候,用户可以自由调整列的顺序. 当用户改变列的顺序的时候,其本身的Index不好改变,但是D ...

  5. Java-ThreadLocal，Java中特殊的线程绑定机制

    在DRP项目中,我们使用了ThreadLocal来创建Connection连接,避免了一直以参数的形式将Connection向下传递(传递connection的目的是由于jdbc事务要求确保使用同一个 ...

  6. java-过滤器Filter_多个Filter的执行顺序

    http://www.cnblogs.com/HigginCui/p/5772514.html [Filter链] *在一个web应用中,可以开发编写多个Filter,这些Filter组合起来称为一个 ...

  7. 格林威治时间格式(GMT)与普通时间格式的互相转换

    GMT --> 普通时间格式: 方法: function GMTToStr(time){ var date = new Date(time) var Str=date.getFullYear() ...

  8. r语言 函数

    R语言实际上是函数的集合,用户可以使用base,stats等包中的基本函数,也可以自己编写函数完成一定的功能.但是初学者往往认为编写R函数十分困难,或者难以理解.这里对如何编写R函数进行简要的介绍. ...

  9. 基于Dedup的数据打包技术

    基于Dedup的数据打包技术 0.引言    Tar, winrar, winzip是最为常见的数据打包工具软件,它们把文件集体封装成一个单独的数据包,从而方便数据的分布.传输.归档以及持久保存等目的 ...

  10. Mysql经常使用基本命令汇总及默认账户权限与改动

    一直仅仅是在浅显利用数据库存储数据.也被windows惯坏了.非常多命令使用的时候记不起来.so,换LINUX系统!不再使用GUI管理数据库!也想深入学习下Mysql.从权限管理開始.也就诞生了这篇学 ...