hive提前过滤重要性
hive提前过滤
create table sospdm.tmp_yinfei_test_01
(
id string
)
partitioned by (statis_date string)
; create table sospdm.tmp_yinfei_test_02
(
id string
)
partitioned by (statis_date string)
; select t1.*
from tmp_yinfei_test_01 t1
left join tmp_yinfei_test_02 t2
on t1.id=t2.id
where t1.statis_date='' and t2.statis_date=''
;
select t1.*
from tmp_yinfei_test_01 t1
left join tmp_yinfei_test_02 t2
on t1.id=t2.id and t1.statis_date='' and t2.statis_date=''
;
select t1.*
from
(
select * from tmp_yinfei_test_01 where statis_date=''
) t1
left join
(
select * from tmp_yinfei_test_02 where statis_date=''
) t2
on t1.id=t2.id
;
=========================test1===================================== explain select t1.*
from tmp_yinfei_test_01 t1
left join tmp_yinfei_test_02 t2
on t1.id=t2.id
where t1.statis_date='' and t2.statis_date=''
; hive> explain select t1.*
> from tmp_yinfei_test_01 t1
> left join tmp_yinfei_test_02 t2
> on t1.id=t2.id
> where t1.statis_date='' and t2.statis_date=''
> ;
OK
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1 STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: t1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (statis_date = '') (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Reduce Output Operator
key expressions: id (type: string)
sort order: +
Map-reduce partition columns: id (type: string)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
TableScan
alias: t2
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Reduce Output Operator
key expressions: id (type: string)
sort order: +
Map-reduce partition columns: id (type: string)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
value expressions: statis_date (type: string)
Reduce Operator Tree:
Join Operator
condition map:
Left Outer Join0 to 1
keys:
0 id (type: string)
1 id (type: string)
outputColumnNames: _col0, _col6
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (_col6 = '') (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: _col0 (type: string), '' (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
File Output Operator
compressed: true
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink Time taken: 0.399 seconds, Fetched: 58 row(s) 结论:t2表会扫全表
=========================test2=====================================
explain select t1.*
from tmp_yinfei_test_01 t1
left join tmp_yinfei_test_02 t2
on t1.id=t2.id and t1.statis_date='' and t2.statis_date=''
;
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1 STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: t1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Reduce Output Operator
key expressions: id (type: string)
sort order: +
Map-reduce partition columns: id (type: string)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
value expressions: statis_date (type: string)
TableScan
alias: t2
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (statis_date = '') (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Reduce Output Operator
key expressions: id (type: string)
sort order: +
Map-reduce partition columns: id (type: string)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
Left Outer Join0 to 1
filter predicates:
0 {(VALUE._col0 = '')}
1
keys:
0 id (type: string)
1 id (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
File Output Operator
compressed: true
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
结论:t1表会扫全表
=========================test3=====================================
explain select t1.*
from
(
select * from tmp_yinfei_test_01 where statis_date=''
) t1
left join
(
select * from tmp_yinfei_test_02 where statis_date=''
) t2
on t1.id=t2.id
;
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1 STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: tmp_yinfei_test_01
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (statis_date = '') (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: id (type: string), '' (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
value expressions: _col1 (type: string)
TableScan
alias: tmp_yinfei_test_02
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (statis_date = '') (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: id (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
Left Outer Join0 to 1
keys:
0 _col0 (type: string)
1 _col0 (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
File Output Operator
compressed: true
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
hive提前过滤重要性的更多相关文章
- hive -- 协同过滤sql语句
hive -- 协同过滤sql语句 数据: *.3g.qq.com|腾讯应用宝|应用商店 *.91rb.com|91手机助手|应用商店 *.app.qq.com|腾讯应用宝|应用商店 *.haina. ...
- STREAMING HIVE流过滤 官网例子 注意中间用的py脚本
Simple Example Use Cases MovieLens User Ratings First, create a table with tab-delimited text file f ...
- hive条件过滤
where 过滤 %代表任意个字符,_代表一个字符; \\ 转移字符.\\_代表下划线
- Hive计算最大连续登陆天数
目录 一.背景 二.算法 1. 第一步:排序 2. 第二步:第二列与第三列做日期差值 3. 第三步:按第二列分组求和 4. 第四步:求最大次数 三.扩展(股票最大涨停天数) 强哥说他发现了财富密码,最 ...
- hadoop 数据倾斜
数据倾斜是指,map /reduce程序执行时,reduce节点大部分执行完毕,但是有一个或者几个reduce节点运行很慢,导致整个程序的处理时间很长,这是因为某一个key的条数比其他key多很多(有 ...
- orcFile split和读数据原理总结(hive0.13)
http://blog.csdn.net/zhaorongsheng/article/details/72903431 官网关于orcfile的介绍 背景 Hive的rcfile格式已经使用多年,但是 ...
- DataSkew 数据倾斜
date: 2020-04-21 19:38:00 updated: 2020-04-24 10:26:00 DataSkew 数据倾斜 1. Hive 里的数据倾斜 1.1 null值 空值 尽量提 ...
- MySQL之谓词下推
MySQL之谓词下推 什么是谓词 在SQL中,谓词就是返回boolean值即true或者false的函数,或是隐式转换为boolean的函数.SQL中的谓词主要有 LKIE.BETWEEN.IS NU ...
- 大数据SQL中的Join谓词下推,真的那么难懂?
听到谓词下推这个词,是不是觉得很高大上,找点资料看了半天才能搞懂概念和思想,借这个机会好好学习一下吧. 引用范欣欣大佬的博客中写道,以前经常满大街听到谓词下推,然而对谓词下推却总感觉懵懵懂懂,并不明白 ...
随机推荐
- JVM·垃圾收集器与内存分配策略之垃圾收集器!
1.Serial(串行)收集器(新生代都采用复制算法) 这是个单线程的收集器:即 当他工作的时候,会停掉虚拟机所有的线程!(Stop The World)
- STM32L476应用开发之三:串行通讯实验
在我们的项目需求中,有两个串口应用需求,一个是与炭氢传感器的通讯,另一个是与显示屏的通讯.鉴于此,我们需要实验串行通讯. 1.硬件设计 串行通讯一个采用RS232接口,另一个直接采用TTL方式.我们在 ...
- GitHub访问慢
问题描述 一直都觉得访问 GitHub 时速度非常慢,刷新一下都要等好久!于是尝试一下能否解决掉... github.com assets-cdn.github.com avatar2.githubu ...
- Go 开源博客平台 Pipe 1.0.0 发布!
这是 Pipe 博客平台的第一个正式版,欢迎大家使用和反馈建议! 简介 Pipe 是一款小而美的开源博客平台,通过黑客派账号登录即可使用. 动机 产品层面: 市面上缺乏支持多独立博客的平台级系统 实现 ...
- bat如何提取文本指定行的内容
背景:使用CTS框架运行完测试后,会在logs中生成devices_log和host_log,在results中生成相应的结果(报告).根据报告信息我们可以得知失败的用例,但是却不能知道为什么用例会失 ...
- Selenium WebDriver中鼠标事件
鼠标点击操作 鼠标点击事件有以下几种类型: 清单 1. 鼠标左键点击 Actions action = new Actions(driver);action.click();// 鼠标左键在当 ...
- django rest framework(4)
目录 一.分页 二.视图 三.路由 四.渲染器 一.分页 试问如果当数据量特别大的时候,你是怎么解决分页的? 方式a.记录当前访问页数的数据id 方式b.最多显示120页等 方式c.只显示上一页,下一 ...
- lightoj1234 打表技巧:分块打表
/* 打不了那么大的表,所以只记录分块的信息即可 */ #include<bits/stdc++.h> using namespace std; #define maxn 1000005 ...
- ATM机模拟系统
实验要求 https://pan.baidu.com/s/1Sq5joWyVsNhY8DYiyI4UmQ 源代码 /*信1705-1 20173458 王嘉梁*/ package atm; impor ...
- Python属性(@property)
创建用于计算机的属性 在Python中,可以通过@property(装饰器)将一个方法转换为属性,从而实现用于计算的属性.将方法转换为属性后,可以直接通过方法名来访问方法,而不需要再添加一对小括号&q ...