Hive函数：LAG,LEAD,FIRST_VALUE,LAST

参考自大数据田地：http://lxw1234.com/archives/2015/04/190.htm

测试数据准备：

create external table test_data (

cookieid string,

createtime string,  --页面访问时间

url string       --被访问页面

) ROW FORMAT DELIMITED

FIELDS TERMINATED BY ','

stored as textfile location '/user/jc_rc_ftp/test_data';

 select * from test_data l;

+-------------+----------------------+---------+--+

| l.cookieid  |     l.createtime     |  l.url  |

+-------------+----------------------+---------+--+

| cookie1     | 2015-04-10 10:00:02  | url2    |

| cookie1     | 2015-04-10 10:00:00  | url1    |

| cookie1     | 2015-04-10 10:03:04  | 1url3   |

| cookie1     | 2015-04-10 10:50:05  | url6    |

| cookie1     | 2015-04-10 11:00:00  | url7    |

| cookie1     | 2015-04-10 10:10:00  | url4    |

| cookie1     | 2015-04-10 10:50:01  | url5    |

| cookie2     | 2015-04-10 10:00:02  | url22   |

| cookie2     | 2015-04-10 10:00:00  | url11   |

| cookie2     | 2015-04-10 10:03:04  | 1url33  |

| cookie2     | 2015-04-10 10:50:05  | url66   |

| cookie2     | 2015-04-10 11:00:00  | url77   |

| cookie2     | 2015-04-10 10:10:00  | url44   |

| cookie2     | 2015-04-10 10:50:01  | url55   |

+-------------+----------------------+---------+--+

LAG
LAG(col,n,DEFAULT) 用于统计窗口内往上第n行值

第一个参数为列名，第二个参数为往上第n行（可选，默认为1），第三个参数为默认值（当往上第n行为NULL时候，取默认值，如不指定，则为NULL）

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

LAG(createtime,1,'1970-01-01 00:00:00') OVER(PARTITION BY cookieid ORDER BY createtime) AS last_1_time,

LAG(createtime,2) OVER(PARTITION BY cookieid ORDER BY createtime) AS last_2_time

FROM test_data;

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

| cookieid  |      createtime      |   url   | rn  |     last_1_time      |     last_2_time      |

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | 1970-01-01 00:00:00  | NULL                 |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | 2015-04-10 10:00:00  | NULL                 |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | 2015-04-10 10:00:02  | 2015-04-10 10:00:00  |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | 2015-04-10 10:03:04  | 2015-04-10 10:00:02  |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | 2015-04-10 10:10:00  | 2015-04-10 10:03:04  |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | 2015-04-10 10:50:01  | 2015-04-10 10:10:00  |

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | 2015-04-10 10:50:05  | 2015-04-10 10:50:01  |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | 1970-01-01 00:00:00  | NULL                 |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | 2015-04-10 10:00:00  | NULL                 |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | 2015-04-10 10:00:02  | 2015-04-10 10:00:00  |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | 2015-04-10 10:03:04  | 2015-04-10 10:00:02  |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | 2015-04-10 10:10:00  | 2015-04-10 10:03:04  |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | 2015-04-10 10:50:01  | 2015-04-10 10:10:00  |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | 2015-04-10 10:50:05  | 2015-04-10 10:50:01  |

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

LEAD

与LAG相反
LEAD(col,n,DEFAULT) 用于统计窗口内往下第n行值
第一个参数为列名，第二个参数为往下第n行（可选，默认为1），第三个参数为默认值（当往下第n行为NULL时候，取默认值，如不指定，则为NULL）

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

LEAD(createtime,1,'1970-01-01 00:00:00') OVER(PARTITION BY cookieid ORDER BY createtime) AS next_1_time,

LEAD(createtime,2) OVER(PARTITION BY cookieid ORDER BY createtime) AS next_2_time

FROM test_data;

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

| cookieid  |      createtime      |   url   | rn  |     next_1_time      |     next_2_time      |

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | 2015-04-10 10:00:02  | 2015-04-10 10:03:04  |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | 2015-04-10 10:03:04  | 2015-04-10 10:10:00  |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | 2015-04-10 10:10:00  | 2015-04-10 10:50:01  |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | 2015-04-10 10:50:01  | 2015-04-10 10:50:05  |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | 2015-04-10 10:50:05  | 2015-04-10 11:00:00  |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | 2015-04-10 11:00:00  | NULL                 |

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | 1970-01-01 00:00:00  | NULL                 |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | 2015-04-10 10:00:02  | 2015-04-10 10:03:04  |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | 2015-04-10 10:03:04  | 2015-04-10 10:10:00  |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | 2015-04-10 10:10:00  | 2015-04-10 10:50:01  |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | 2015-04-10 10:50:01  | 2015-04-10 10:50:05  |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | 2015-04-10 10:50:05  | 2015-04-10 11:00:00  |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | 2015-04-10 11:00:00  | NULL                 |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | 1970-01-01 00:00:00  | NULL                 |

+-----------+----------------------+---------+-----+----------------------+----------------------+--+

FIRST_VALUE

取分组内排序后，截止到当前行，第一个值

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

FIRST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime) AS first1

FROM test_data;

+-----------+----------------------+---------+-----+---------+--+

| cookieid  |      createtime      |   url   | rn  | first1  |

+-----------+----------------------+---------+-----+---------+--+

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | url1    |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | url1    |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | url1    |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | url1    |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | url1    |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | url1    |

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | url1    |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | url11   |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | url11   |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | url11   |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | url11   |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | url11   |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | url11   |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | url11   |

+-----------+----------------------+---------+-----+---------+--+

LAST_VALUE

取分组内排序后，截止到当前行，最后一个值

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

LAST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime) AS last1

FROM test_data;

+-----------+----------------------+---------+-----+---------+--+

| cookieid  |      createtime      |   url   | rn  |  last1  |

+-----------+----------------------+---------+-----+---------+--+

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | url1    |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | url2    |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | 1url3   |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | url4    |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | url5    |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | url6    |

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | url7    |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | url11   |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | url22   |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | 1url33  |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | url44   |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | url55   |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | url66   |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | url77   |

+-----------+----------------------+---------+-----+---------+--+

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

LAST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime DESC) AS last1

FROM test_data;

+-----------+----------------------+---------+-----+---------+--+

| cookieid  |      createtime      |   url   | rn  |  last1  |

+-----------+----------------------+---------+-----+---------+--+

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | url7    |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | url6    |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | url5    |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | url4    |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | 1url3   |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | url2    |

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | url1    |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | url77   |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | url66   |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | url55   |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | url44   |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | 1url33  |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | url22   |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | url11   |

+-----------+----------------------+---------+-----+---------+--+

如果不指定ORDER BY，则默认按照记录在文件中的偏移量进行排序，会出现错误的结果

SELECT cookieid,

createtime,

url,

FIRST_VALUE(url) OVER(PARTITION BY cookieid) AS first2

FROM test_data;

+-----------+----------------------+---------+---------+--+

| cookieid  |      createtime      |   url   | first2  |

+-----------+----------------------+---------+---------+--+

| cookie1   | 2015-04-10 10:00:02  | url2    | url2    |

| cookie1   | 2015-04-10 10:50:01  | url5    | url2    |

| cookie1   | 2015-04-10 10:10:00  | url4    | url2    |

| cookie1   | 2015-04-10 11:00:00  | url7    | url2    |

| cookie1   | 2015-04-10 10:50:05  | url6    | url2    |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | url2    |

| cookie1   | 2015-04-10 10:00:00  | url1    | url2    |

| cookie2   | 2015-04-10 10:50:01  | url55   | url55   |

| cookie2   | 2015-04-10 10:10:00  | url44   | url55   |

| cookie2   | 2015-04-10 11:00:00  | url77   | url55   |

| cookie2   | 2015-04-10 10:50:05  | url66   | url55   |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | url55   |

| cookie2   | 2015-04-10 10:00:00  | url11   | url55   |

| cookie2   | 2015-04-10 10:00:02  | url22   | url55   |

+-----------+----------------------+---------+---------+--+

SELECT cookieid,

createtime,

url,

LAST_VALUE(url) OVER(PARTITION BY cookieid) AS last2

FROM test_data;

+-----------+----------------------+---------+--------+--+

| cookieid  |      createtime      |   url   | last2  |

+-----------+----------------------+---------+--------+--+

| cookie1   | 2015-04-10 10:00:02  | url2    | url1   |

| cookie1   | 2015-04-10 10:50:01  | url5    | url1   |

| cookie1   | 2015-04-10 10:10:00  | url4    | url1   |

| cookie1   | 2015-04-10 11:00:00  | url7    | url1   |

| cookie1   | 2015-04-10 10:50:05  | url6    | url1   |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | url1   |

| cookie1   | 2015-04-10 10:00:00  | url1    | url1   |

| cookie2   | 2015-04-10 10:50:01  | url55   | url22  |

| cookie2   | 2015-04-10 10:10:00  | url44   | url22  |

| cookie2   | 2015-04-10 11:00:00  | url77   | url22  |

| cookie2   | 2015-04-10 10:50:05  | url66   | url22  |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | url22  |

| cookie2   | 2015-04-10 10:00:00  | url11   | url22  |

| cookie2   | 2015-04-10 10:00:02  | url22   | url22  |

+-----------+----------------------+---------+--------+--+

14 rows selected (78.058 seconds)

如果想要取分组内排序后最后一个值，则需要变通一下：

SELECT cookieid,

createtime,

url,

ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,

LAST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime) AS last1,

FIRST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime DESC) AS last2

FROM test_data

ORDER BY cookieid,createtime;

+-----------+----------------------+---------+-----+---------+--------+--+

| cookieid  |      createtime      |   url   | rn  |  last1  | last2  |

+-----------+----------------------+---------+-----+---------+--------+--+

| cookie1   | 2015-04-10 10:00:00  | url1    | 1   | url1    | url7   |

| cookie1   | 2015-04-10 10:00:02  | url2    | 2   | url2    | url7   |

| cookie1   | 2015-04-10 10:03:04  | 1url3   | 3   | 1url3   | url7   |

| cookie1   | 2015-04-10 10:10:00  | url4    | 4   | url4    | url7   |

| cookie1   | 2015-04-10 10:50:01  | url5    | 5   | url5    | url7   |

| cookie1   | 2015-04-10 10:50:05  | url6    | 6   | url6    | url7   |

| cookie1   | 2015-04-10 11:00:00  | url7    | 7   | url7    | url7   |

| cookie2   | 2015-04-10 10:00:00  | url11   | 1   | url11   | url77  |

| cookie2   | 2015-04-10 10:00:02  | url22   | 2   | url22   | url77  |

| cookie2   | 2015-04-10 10:03:04  | 1url33  | 3   | 1url33  | url77  |

| cookie2   | 2015-04-10 10:10:00  | url44   | 4   | url44   | url77  |

| cookie2   | 2015-04-10 10:50:01  | url55   | 5   | url55   | url77  |

| cookie2   | 2015-04-10 10:50:05  | url66   | 6   | url66   | url77  |

| cookie2   | 2015-04-10 11:00:00  | url77   | 7   | url77   | url77  |

+-----------+----------------------+---------+-----+---------+--------+--+

Hive函数：LAG,LEAD,FIRST_VALUE,LAST_VALUE的更多相关文章

pandas实现hive的lag和lead函数以及 first_value和last_value函数
lag和lead VS shift 该函数的格式如下: 第一个参数为列名, 第二个参数为往上第n行(可选,默认为1), 第三个参数为默认值(当往上第n行为NULL时候,取默认值,如不指定,则为NULL ...
Hive 窗口函数LEAD LAG FIRST_VALUE LAST_VALUE
窗口函数(window functions)对多行进行操作,并为查询中的每一行返回一个值. OVER()子句能将窗口函数与其他分析函数(analytical functions)和报告函数(repor ...
oracle listagg函数、lag函数、lead函数实例
Oracle大师Thomas Kyte在他的经典著作中,反复强调过一个实现需求方案选取顺序: “如果你可以使用一句SQL解决的需求,就使用一句SQL:如果不可以,就考虑PL/SQL是否可以:如果PL/ ...
hive函数参考手册
hive函数参考手册原文见:https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF 1.内置运算符1.1关系运算符运 ...
Hive函数以及自定义函数讲解（UDF）
Hive函数介绍HQL内嵌函数只有195个函数(包括操作符,使用命令show functions查看),基本能够胜任基本的hive开发,但是当有较为复杂的需求的时候,可能需要进行定制的HQL函数开发. ...
大数据入门第十一天——hive详解（三）hive函数
一.hive函数 1.内置运算符与内置函数函数分类: 查看函数信息: DESC FUNCTION concat; 常用的分析函数之rank() row_number(),参考:https://www ...
Hadoop生态圈-Hive函数
Hadoop生态圈-Hive函数作者:尹正杰版权声明:原创作品,谢绝转载!否则将追究法律责任.
Hive（四）hive函数与hive shell
一.hive函数 1.hive内置函数 (1)内容较多,见< Hive 官方文档> https://cwiki.apache.org/confluence/displ ...
Hive入门笔记---2.hive函数大全
Hive函数大全–完整版现在虽然有很多SQL ON Hadoop的解决方案,像Spark SQL.Impala.Presto等等,但就目前来看,在基于Hadoop的大数据分析平台.数据仓库中,Hiv ...

随机推荐

C++string类总结
一.string的初始化首先,为了在程序中使用string类型,必须包含头文件 <string>.如下: #include <string> 注意这里不是string.h,s ...
canvas线条的属性
一.线条的帽子lineCap 取值:butt(默认值),round圆头,square方头 var canvas=document.getElementById("canvas"); ...
继续吐槽在net下没有合适的Disk Cache之使用EhCache
说起缓存,大家可能口若悬河,各种类型的缓存都能一一分析,但在net下找到一款合适的Disk Cache貌似还是有一点难度的. 一:背景事情是这样的,最近的一个项目中,需要在web端绘制一些报表,因为 ...
Algorithm --> 顺序打印矩阵
顺序打印矩阵思路参考代码 #include <iostream> using namespace std; ], int row, int col) { || col < ) r ...
Spring Cloud Contract
http://www.infoq.com/cn/news/2017/04/spring-cloud-contract 在默认情况下,我们希望用户以JAR文件的形式将生产者存根和契约发布到Maven库. ...
C语言中数据类型的取值范围
C语言中数据类型的取值范围如下:char -128 ~ +127 (1 Byte)short -32767 ~ + 32768 (2 Bytes)unsigned short 0 ~ 65536 (2 ...
Oracle profile 使用技巧
给scott用户分配一个profile要求如下: 1.尝试登录的次数最多4次: 2.如果4次输入错误,则锁定该用户2天: 3.密码每隔5天修改一次,宽限期为2天: 答: SQL>conn sys ...
lua_cocos精灵的不断闪动
一. 刚开始使用 local blink = cc.Blink:create(1, 10) sprite:runAction(blink) ...
Git常用命令（一）------ 本地操作
本文总结自廖雪峰的网站. 几个名词: 工作区(Working Directory):电脑里能看到的目录版本库(Repository):包含暂存区和master 暂存区(Stage):待放入maste ...
2017-2018-1 Java演绎法第一周作业
团队学习:<构建之法> [团队成员]: 学号姓名负责工作 20162315 马军日常统计,项目部分代码 20162316 刘诚昊项目部分代码,代码质量测试 20162317 袁逸灏 ...

Hive函数：LAG,LEAD,FIRST_VALUE,LAST_VALUE

LEAD

FIRST_VALUE

LAST_VALUE

Hive函数：LAG,LEAD,FIRST_VALUE,LAST_VALUE的更多相关文章

随机推荐

热门专题