转自：http://lxw1234.com/archives/2015/04/176.htm，Hive分析窗口函数(一) SUM,AVG,MIN,MAX

之前看到大数据田地有关于max()over(partition by)的用法，今天恰好工作中用到了它，但是使用中遇到了一个问题：在max(rsrp)over(partition by buildingid,height) as max_rsrp返回的结果不是分组中的最大值。最中找到了问题的原因：max_rsrp数据类型为string而不是double类型，导致的一个bug问题。

再处理的过程中也再次把大数据田地的中关于sum,avg,max,min的函数用法做了demo，因此有了该参考后的文章。

数据准备：

echo ''>data_file.txt

vim data_file.txt

cookie1,2015-04-10,1

cookie1,2015-04-11,5

cookie1,2015-04-12,7

cookie1,2015-04-13,3

cookie1,2015-04-14,2

cookie1,2015-04-15,4

cookie1,2015-04-16,4

cookie2,2015-04-10,6

cookie2,2015-04-11,5

cookie2,2015-04-12,7

cookie2,2015-04-13,4

cookie2,2015-04-14,3

cookie2,2015-04-15,5

cookie2,2015-04-16,5

hadoop fs -rm -r /user/jrf/test_data

hadoop fs -mkdir /user/jrf/test_data

hadoop fs -copyFromLocal data_file.txt /user/jrf/test_data/
drop table if exists test_data;

create EXTERNAL TABLE test_data (

cookieid string,

createtime string,   --day

pv INT

) ROW FORMAT DELIMITED

FIELDS TERMINATED BY ','

stored as textfile location '/user/jrf/test_data/';

select * from test_data;

+---------------------+-----------------------+---------------+--+

| test_data.cookieid  | test_data.createtime  | test_data.pv  |

+---------------------+-----------------------+---------------+--+

| cookie1             | 2015-04-10            | 1             |

| cookie1             | 2015-04-11            | 5             |

| cookie1             | 2015-04-12            | 7             |

| cookie1             | 2015-04-13            | 3             |

| cookie1             | 2015-04-14            | 2             |

| cookie1             | 2015-04-15            | 4             |

| cookie1             | 2015-04-16            | 4             |

| cookie2             | 2015-04-10            | 6             |

| cookie2             | 2015-04-11            | 5             |

| cookie2             | 2015-04-12            | 7             |

| cookie2             | 2015-04-13            | 4             |

| cookie2             | 2015-04-14            | 3             |

| cookie2             | 2015-04-15            | 5             |

| cookie2             | 2015-04-16            | 5             |

+---------------------+-----------------------+---------------+--+

SUM — 注意,结果和ORDER BY相关,默认为升序

SELECT cookieid,createtime,pv,

SUM(pv) OVER(PARTITION BY cookieid ORDER BY createtime) AS pv1, -- 默认为从起点到当前行

SUM(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS pv2, --从起点到当前行，结果同pv1

SUM(pv) OVER(PARTITION BY cookieid) AS pv3,--分组内所有行

SUM(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PRECEDING AND CURRENT ROW) AS pv4,--当前行+往前3行

SUM(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) AS pv5,--当前行+往前3行+往后1行

SUM(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS pv6 --当前行+往后所有行

FROM test_data order by cookieid,createtime;

+-----------+-------------+-----+------+------+------+------+------+------+--+

| cookieid  | createtime  | pv  | pv1  | pv2  | pv3  | pv4  | pv5  | pv6  |

+-----------+-------------+-----+------+------+------+------+------+------+--+

| cookie1   | 2015-04-10  | 1   | 1    | 1    | 26   | 1    | 6    | 26   |

| cookie1   | 2015-04-11  | 5   | 6    | 6    | 26   | 6    | 13   | 25   |

| cookie1   | 2015-04-12  | 7   | 13   | 13   | 26   | 13   | 16   | 20   |

| cookie1   | 2015-04-13  | 3   | 16   | 16   | 26   | 16   | 18   | 13   |

| cookie1   | 2015-04-14  | 2   | 18   | 18   | 26   | 17   | 21   | 10   |

| cookie1   | 2015-04-15  | 4   | 22   | 22   | 26   | 16   | 20   | 8    |

| cookie1   | 2015-04-16  | 4   | 26   | 26   | 26   | 13   | 13   | 4    |

| cookie2   | 2015-04-10  | 6   | 6    | 6    | 35   | 6    | 11   | 35   |

| cookie2   | 2015-04-11  | 5   | 11   | 11   | 35   | 11   | 18   | 29   |

| cookie2   | 2015-04-12  | 7   | 18   | 18   | 35   | 18   | 22   | 24   |

| cookie2   | 2015-04-13  | 4   | 22   | 22   | 35   | 22   | 25   | 17   |

| cookie2   | 2015-04-14  | 3   | 25   | 25   | 35   | 19   | 24   | 13   |

| cookie2   | 2015-04-15  | 5   | 30   | 30   | 35   | 19   | 24   | 10   |

| cookie2   | 2015-04-16  | 5   | 35   | 35   | 35   | 17   | 17   | 5    |

+-----------+-------------+-----+------+------+------+------+------+------+--+

pv1: 分组内从起点到当前行的pv累积，如，11号的pv1=10号的pv+11号的pv, 12号=10号+11号+12号

pv2: 同pv1

pv3: 分组内(cookie1)所有的pv累加

pv4: 分组内当前行+往前3行，如，11号=10号+11号， 12号=10号+11号+12号， 13号=10号+11号+12号+13号， 14号=11号+12号+13号+14号

pv5: 分组内当前行+往前3行+往后1行，如，14号=11号+12号+13号+14号+15号=5+7+3+2+4=21

pv6: 分组内当前行+往后所有行，如，13号=13号+14号+15号+16号=3+2+4+4=13，14号=14号+15号+16号=2+4+4=10

如果不指定ROWS BETWEEN,默认为从起点到当前行;
如果不指定ORDER BY，则将分组内所有值累加;
关键是理解ROWS BETWEEN含义,也叫做WINDOW子句：
PRECEDING：往前
FOLLOWING：往后
CURRENT ROW：当前行
UNBOUNDED：起点，UNBOUNDED PRECEDING 表示从前面的起点， UNBOUNDED FOLLOWING：表示到后面的终点

–其他AVG，MIN，MAX，和SUM用法一样。

--AVG

SELECT cookieid,createtime,pv,

AVG(pv) OVER(PARTITION BY cookieid ORDER BY createtime) AS pv1, -- 默认为从起点到当前行

AVG(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS pv2, --从起点到当前行，结果同pv1

AVG(pv) OVER(PARTITION BY cookieid) AS pv3,--分组内所有行

AVG(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PRECEDING AND CURRENT ROW) AS pv4,--当前行+往前3行

AVG(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) AS pv5,--当前行+往前3行+往后1行

AVG(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS pv6 --当前行+往后所有行

FROM test_data  order by cookieid,createtime;

+-----------+-------------+-----+---------------------+---------------------+---------------------+--------------------+--------------------+---------------------+--+

| cookieid  | createtime  | pv  |         pv1         |         pv2         |         pv3         |        pv4         |        pv5         |         pv6         |

+-----------+-------------+-----+---------------------+---------------------+---------------------+--------------------+--------------------+---------------------+--+

| cookie1   | 2015-04-10  | 1   | 1.0                 | 1.0                 | 3.7142857142857144  | 1.0                | 3.0                | 3.7142857142857144  |

| cookie1   | 2015-04-11  | 5   | 3.0                 | 3.0                 | 3.7142857142857144  | 3.0                | 4.333333333333333  | 4.166666666666667   |

| cookie1   | 2015-04-12  | 7   | 4.333333333333333   | 4.333333333333333   | 3.7142857142857144  | 4.333333333333333  | 4.0                | 4.0                 |

| cookie1   | 2015-04-13  | 3   | 4.0                 | 4.0                 | 3.7142857142857144  | 4.0                | 3.6                | 3.25                |

| cookie1   | 2015-04-14  | 2   | 3.6                 | 3.6                 | 3.7142857142857144  | 4.25               | 4.2                | 3.3333333333333335  |

| cookie1   | 2015-04-15  | 4   | 3.6666666666666665  | 3.6666666666666665  | 3.7142857142857144  | 4.0                | 4.0                | 4.0                 |

| cookie1   | 2015-04-16  | 4   | 3.7142857142857144  | 3.7142857142857144  | 3.7142857142857144  | 3.25               | 3.25               | 4.0                 |

| cookie2   | 2015-04-10  | 6   | 6.0                 | 6.0                 | 5.0                 | 6.0                | 5.5                | 5.0                 |

| cookie2   | 2015-04-11  | 5   | 5.5                 | 5.5                 | 5.0                 | 5.5                | 6.0                | 4.833333333333333   |

| cookie2   | 2015-04-12  | 7   | 6.0                 | 6.0                 | 5.0                 | 6.0                | 5.5                | 4.8                 |

| cookie2   | 2015-04-13  | 4   | 5.5                 | 5.5                 | 5.0                 | 5.5                | 5.0                | 4.25                |

| cookie2   | 2015-04-14  | 3   | 5.0                 | 5.0                 | 5.0                 | 4.75               | 4.8                | 4.333333333333333   |

| cookie2   | 2015-04-15  | 5   | 5.0                 | 5.0                 | 5.0                 | 4.75               | 4.8                | 5.0                 |

| cookie2   | 2015-04-16  | 5   | 5.0                 | 5.0                 | 5.0                 | 4.25               | 4.25               | 5.0                 |

+-----------+-------------+-----+---------------------+---------------------+---------------------+--------------------+--------------------+---------------------+--+

--MIN

SELECT cookieid,createtime,pv,

MIN(pv) OVER(PARTITION BY cookieid ORDER BY createtime) AS pv1, -- 默认为从起点到当前行

MIN(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS pv2,--从起点到当前行，结果同pv1

MIN(pv) OVER(PARTITION BY cookieid) AS pv3,--分组内所有行

MIN(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PRECEDING AND CURRENT ROW) AS pv4,--当前行+往前3行

MIN(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) AS pv5,--当前行+往前3行+往后1行

MIN(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS pv6 --当前行+往后所有行

FROM test_data order by cookieid,createtime;

+-----------+-------------+-----+------+------+------+------+------+------+--+

| cookieid  | createtime  | pv  | pv1  | pv2  | pv3  | pv4  | pv5  | pv6  |

+-----------+-------------+-----+------+------+------+------+------+------+--+

| cookie1   | 2015-04-10  | 1   | 1    | 1    | 1    | 1    | 1    | 1    |

| cookie1   | 2015-04-11  | 5   | 1    | 1    | 1    | 1    | 1    | 2    |

| cookie1   | 2015-04-12  | 7   | 1    | 1    | 1    | 1    | 1    | 2    |

| cookie1   | 2015-04-13  | 3   | 1    | 1    | 1    | 1    | 1    | 2    |

| cookie1   | 2015-04-14  | 2   | 1    | 1    | 1    | 2    | 2    | 2    |

| cookie1   | 2015-04-15  | 4   | 1    | 1    | 1    | 2    | 2    | 4    |

| cookie1   | 2015-04-16  | 4   | 1    | 1    | 1    | 2    | 2    | 4    |

| cookie2   | 2015-04-10  | 6   | 6    | 6    | 3    | 6    | 5    | 3    |

| cookie2   | 2015-04-11  | 5   | 5    | 5    | 3    | 5    | 5    | 3    |

| cookie2   | 2015-04-12  | 7   | 5    | 5    | 3    | 5    | 4    | 3    |

| cookie2   | 2015-04-13  | 4   | 4    | 4    | 3    | 4    | 3    | 3    |

| cookie2   | 2015-04-14  | 3   | 3    | 3    | 3    | 3    | 3    | 3    |

| cookie2   | 2015-04-15  | 5   | 3    | 3    | 3    | 3    | 3    | 5    |

| cookie2   | 2015-04-16  | 5   | 3    | 3    | 3    | 3    | 3    | 5    |

+-----------+-------------+-----+------+------+------+------+------+------+--+

--MAX

SELECT cookieid,createtime,pv,

MAX(pv) OVER(PARTITION BY cookieid ORDER BY createtime) AS pv1, -- 默认为从起点到当前行

MAX(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS pv2, --从起点到当前行，结果同pv1

MAX(pv) OVER(PARTITION BY cookieid) AS pv3, --分组内所有行

MAX(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PRECEDING AND CURRENT ROW) AS pv4, --当前行+往前3行

MAX(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) AS pv5, --当前行+往前3行+往后1行

MAX(pv) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS pv6 --当前行+往后所有行

FROM test_data order by cookieid,createtime;

+-----------+-------------+-----+------+------+------+------+------+------+--+

| cookieid  | createtime  | pv  | pv1  | pv2  | pv3  | pv4  | pv5  | pv6  |

+-----------+-------------+-----+------+------+------+------+------+------+--+

| cookie1   | 2015-04-10  | 1   | 1    | 1    | 7    | 1    | 5    | 7    |

| cookie1   | 2015-04-11  | 5   | 5    | 5    | 7    | 5    | 7    | 7    |

| cookie1   | 2015-04-12  | 7   | 7    | 7    | 7    | 7    | 7    | 7    |

| cookie1   | 2015-04-13  | 3   | 7    | 7    | 7    | 7    | 7    | 4    |

| cookie1   | 2015-04-14  | 2   | 7    | 7    | 7    | 7    | 7    | 4    |

| cookie1   | 2015-04-15  | 4   | 7    | 7    | 7    | 7    | 7    | 4    |

| cookie1   | 2015-04-16  | 4   | 7    | 7    | 7    | 4    | 4    | 4    |

| cookie2   | 2015-04-10  | 6   | 6    | 6    | 7    | 6    | 6    | 7    |

| cookie2   | 2015-04-11  | 5   | 6    | 6    | 7    | 6    | 7    | 7    |

| cookie2   | 2015-04-12  | 7   | 7    | 7    | 7    | 7    | 7    | 7    |

| cookie2   | 2015-04-13  | 4   | 7    | 7    | 7    | 7    | 7    | 5    |

| cookie2   | 2015-04-14  | 3   | 7    | 7    | 7    | 7    | 7    | 5    |

| cookie2   | 2015-04-15  | 5   | 7    | 7    | 7    | 7    | 7    | 5    |

| cookie2   | 2015-04-16  | 5   | 7    | 7    | 7    | 5    | 5    | 5    |

+-----------+-------------+-----+------+------+------+------+------+------+--+

SELECT cookieid,

createtime,

pv,

min(pv) OVER(PARTITION BY cookieid) AS min_pv,

max(pv) OVER(PARTITION BY cookieid) AS max_pv

FROM test_data;

+-----------+-------------+-----+---------+---------+--+

| cookieid  | createtime  | pv  | min_pv  | max_pv  |

+-----------+-------------+-----+---------+---------+--+

| cookie1   | 2015-04-10  | 1   | 1       | 7       |

| cookie1   | 2015-04-16  | 4   | 1       | 7       |

| cookie1   | 2015-04-15  | 4   | 1       | 7       |

| cookie1   | 2015-04-14  | 2   | 1       | 7       |

| cookie1   | 2015-04-13  | 3   | 1       | 7       |

| cookie1   | 2015-04-12  | 7   | 1       | 7       |

| cookie1   | 2015-04-11  | 5   | 1       | 7       |

| cookie2   | 2015-04-16  | 5   | 3       | 7       |

| cookie2   | 2015-04-15  | 5   | 3       | 7       |

| cookie2   | 2015-04-14  | 3   | 3       | 7       |

| cookie2   | 2015-04-13  | 4   | 3       | 7       |

| cookie2   | 2015-04-12  | 7   | 3       | 7       |

| cookie2   | 2015-04-11  | 5   | 3       | 7       |

| cookie2   | 2015-04-10  | 6   | 3       | 7       |

+-----------+-------------+-----+---------+---------+--+

Hive函数：SUM,AVG,MIN,MAX的更多相关文章

  1. Hive分析窗口函数(一) SUM,AVG,MIN,MAX

    Hive分析窗口函数(一) SUM,AVG,MIN,MAX Hive分析窗口函数(一) SUM,AVG,MIN,MAX Hive中提供了越来越多的分析函数,用于完成负责的统计分析.抽时间将所有的分析窗 ...

  2. Hive学习之路 （十三）Hive分析窗口函数(一) SUM,AVG,MIN,MAX

    数据准备 数据格式 cookie1,, cookie1,, cookie1,, cookie1,, cookie1,, cookie1,, cookie1,, 创建数据库及表 create datab ...

  3. MybatisPlus Lambda表达式 聚合查询 分组查询 COUNT SUM AVG MIN MAX GroupBy

    一.序言 众所周知,MybatisPlus在处理单表DAO操作时非常的方便.在处理多表连接连接查询也有优雅的解决方案.今天分享MybatisPlus基于Lambda表达式优雅实现聚合分组查询. 由于视 ...

  4. C# 中奇妙的函数–6. 五个序列聚合运算(Sum, Average, Min, Max,Aggregate)

    今天,我们将着眼于五个用于序列的聚合运算.很多时候当我们在对序列进行操作时,我们想要做基于这些序列执行某种汇总然后,计算结果. Enumerable 静态类的LINQ扩展方法可以做到这一点 .就像之前 ...

  5. SQL模糊查询,sum,AVG,MAX,min函数

    cmd mysql -hlocalhost -uroot -p select * from emp where ename like '___' -- 三个横线, - 代表字符,可以查询 三个enam ...

  6. 三、函数 （SUM、MIN、MAX、COUNT、AVG）

    第八章 使用数据处理函数 8.1 函数 SQL支持利用函数来处理数据.函数一般是在数据上执行的,给数据的转换和处理提供了方便. 每一个DBMS都有特定的函数.只有少数几个函数被所有主要的DBMS等同的 ...

  7. LINQ to SQL Count/Sum/Min/Max/Avg Join

    public class Linq { MXSICEDataContext Db = new MXSICEDataContext(); // LINQ to SQL // Count/Sum/Min/ ...

  8. LINQ to SQL 语句(3) 之 Count/Sum/Min/Max/Avg

    LINQ  to SQL 语句(3) 之  Count/Sum/Min/Max/Avg [1] Count/Sum 讲解 [2] Min 讲解 [3] Max 讲解 [4] Average 和 Agg ...

  9. [转]LINQ语句之Select/Distinct和Count/Sum/Min/Max/Avg

    在讲述了LINQ,顺便说了一下Where操作,这篇开始我们继续说LINQ语句,目的让大家从语句的角度了解LINQ,LINQ包括LINQ to Objects.LINQ to DataSets.LINQ ...

随机推荐

  1. Git分支(5/5) -- 解决合并的冲突

    如果两个分支上都对同一个文件进行了修改, 那么就有可能发生冲突. 首先创建一个分支, 并切换到该分支上: 然后修改index.html, 修改几个地方吧. 然后查看状态, 并commit: 然后切换到 ...

  2. 在线资源--图片/json等

    1. 在线图片: http://image.zhangxinxu.com/image/study/s/s256/mm3.jpg  // mmX.jpg: X可为任意的数字 2. 在线json: 雅虎天 ...

  3. Mycat 配置说明（server.xml)

    server.xml 几乎保存了所有mycat需要的系统配置信息,包括 mycat 用户管理.DML权限管理等,其在代码内直接的映射类为SystemConfig 类. user 标签 该标签主要用于定 ...

  4. 【数据结构】 字符串&KMP子串匹配算法

    字符串 作为人机交互的途径,程序或多或少地肯定要需要处理文字信息.如何在计算机中抽象人类语言的信息就成为一个问题.字符串便是这个问题的答案.虽然从形式上来说,字符串可以算是线性表的一种,其数据储存区存 ...

  5. linux拓展下：批量改扩展名的方法

    [root@oldboy oldboy]# ll total 0 -rw-r--r-- 1 root root 0 Nov 13 19:38 stu_102999_1_.jpg -rw-r--r-- ...

  6. C++ STL 容器之栈的使用

    Stack 栈是种先进后出的容器,C++中使用STL容器Stack<T> 完美封装了栈的常用功能. 下面来个demo 学习下使用栈的使用. //引入IO流头文件 #include<i ...

  7. webpack----webpack4尝鲜

    安装v4.0.0-beta.0 yarn add webpack@next webpack-cli --dev 或者 npm install webpack@next webpack-cli --sa ...

  8. Beta No.4

    今天遇到的困难: 百度位置假死的问题研究发现并不是源于代码的问题,而是直接运行在主线程中会出现诸多问题 Fragment碎片刷新时总产生的固定位置的问题未果 今天完成的任务: 陈甘霖:修复了部分Bug ...

  9. C语言的第二次作业

    一.PTA实验作业 题目1. 计算分段函数 本题目要求计算下列分段函数f(x)的值: 1.本题代码 #include<stdio.h> #include<math.h> int ...

  10. C语言博客作业指针

    一.PTA实验作业 题目1:统计大于等于平均分人数 本题PTA提交列表 设计思路 定义形参s存放s的首地址,n存放人数m的值,aver存放平均分aver的地址 定义i控制循环,count用来统计高于等 ...