如何快速把hdfs数据动态导入到hive表
1. hdfs 文件
{"retCode":1,"retMsg":"Success","data":[{"secID":"000001.XSHE","ticker":"000001","secShortName":"深发展A","exchangeCD":"XSHE","tradeDate":"1991-10-21","preClosePrice":24,"actPreClosePrice":24,"openPrice":24,"highestPrice":24.4,"lowestPrice":23.85,"closePrice":23.9,"turnoverVol":355700,"turnoverValue":8582250,"turnoverRate":0.0058,"accumAdjFactor":0.0117201563,"negMarketValue":1462295257.8,"marketValue":2145064267.7,"PB":2.2666,"isOpen":1},{"secID":"000002.XSHE","ticker":"000002","secShortName":"深万科A","exchangeCD":"XSHE","tradeDate":"1991-10-21","preClosePrice":8,"actPreClosePrice":8,"openPrice":8,"highestPrice":8,"lowestPrice":7.7,"closePrice":7.9,"turnoverVol":375000,"turnoverValue":2944200,"turnoverRate":0.0066,"accumAdjFactor":0.0117337592,"negMarketValue":451011000,"marketValue":615927450,"PB":1.0001,"isOpen":1},{"secID":"000004.XSHE","ticker":"000004","secShortName":"深安达A","exchangeCD":"XSHE","tradeDate":"1991-10-21","preClosePrice":7.25,"actPreClosePrice":7.25,"openPrice":7.25,"highestPrice":7.25,"lowestPrice":7.2,"closePrice":7.2,"turnoverVol":92000,"turnoverValue":665125,"turnoverRate":0.0078,"accumAdjFactor":0.2649084628,"negMarketValue":84977100,"marketValue":175500000,"PB":7.4199,"isOpen":1},{"secID":"000005.XSHE","ticker":"000005","secShortName":"深原野A","exchangeCD":"XSHE","tradeDate":"1991-10-21","preClosePrice":6.46,"actPreClosePrice":6.46,"openPrice":6.49,"highestPrice":6.49,"lowestPrice":6.49,"closePrice":6.49,"turnoverVol":94500,"turnoverValue":613305,"turnoverRate":0.0021,"accumAdjFactor":0.1016459912,"negMarketValue":287756865,"marketValue":584100000,"PB":9.1783,"isOpen":1},{"secID":"000009.XSHE","ticker":"000009","secShortName":"深宝安A","exchangeCD":"XSHE","tradeDate":"1991-10-21","preClosePrice":5.75,"actPreClosePrice":5.75,"openPrice":5.7,"highestPrice":5.8,"lowestPrice":5.65,"closePrice":5.75,"turnoverVol":767500,"turnoverValue":4382245,"turnoverRate":0.0084,"accumAdjFactor":0.1026538759,"negMarketValue":524745000,"marketValue":1293922500,"PB":2.4503,"isOpen":1},{"secID":"600601.XSHG","ticker":"600601","secShortName":"延中实业","exchangeCD":"XSHG","tradeDate":"1991-10-21","preClosePrice":65.7,"actPreClosePrice":65.7,"openPrice":66.4,"highestPrice":66.4,"lowestPrice":66.4,"closePrice":66.4,"turnoverVol":5333,"turnoverValue":354111,"dealAmount":81,"turnoverRate":0.0053,"accumAdjFactor":0.0010592167,"negMarketValue":66400000,"marketValue":66400000,"PB":40.7703,"isOpen":1},{"secID":"600602.XSHG","ticker":"600602","secShortName":"真空电子","exchangeCD":"XSHG","tradeDate":"1991-10-21","preClosePrice":640.6,"actPreClosePrice":640.6,"openPrice":647,"highestPrice":647,"lowestPrice":647,"closePrice":647,"turnoverVol":2589,"turnoverValue":1675083,"dealAmount":227,"turnoverRate":0.0051,"accumAdjFactor":0.0019640692,"negMarketValue":330552300,"marketValue":1294000000,"PB":287.6707,"isOpen":1},{"secID":"600651.XSHG","ticker":"600651","secShortName":"飞乐音响","exchangeCD":"XSHG","tradeDate":"1991-10-21","preClosePrice":119.6,"actPreClosePrice":119.6,"openPrice":120.8,"highestPrice":120.8,"lowestPrice":120.8,"closePrice":120.8,"turnoverVol":1102,"turnoverValue":133122,"dealAmount":14,"turnoverRate":0.0022,"accumAdjFactor":0.0008192464,"negMarketValue":60400000,"marketValue":60400000,"PB":39.6397,"isOpen":1},{"secID":"600652.XSHG","ticker":"600652","secShortName":"爱使电子","exchangeCD":"XSHG","tradeDate":"1991-10-21","preClosePrice":83.2,"actPreClosePrice":83.2,"openPrice":0,"highestPrice":0,"lowestPrice":0,"closePrice":83.2,"turnoverVol":0,"turnoverValue":0,"dealAmount":0,"turnoverRate":0,"accumAdjFactor":0.0006920481,"negMarketValue":22464000,"marketValue":22464000,"PB":33.8019,"isOpen":0},{"secID":"600653.XSHG","ticker":"600653","secShortName":"申华电工","exchangeCD":"XSHG","tradeDate":"1991-10-21","preClosePrice":103.4,"actPreClosePrice":103.4,"openPrice":104.4,"highestPrice":104.4,"lowestPrice":104.4,"closePrice":104.4,"turnoverVol":240,"turnoverValue":25056,"dealAmount":4,"turnoverRate":0.0005,"accumAdjFactor":0.0009289199,"negMarketValue":52200000,"marketValue":52200000,"PB":97.279,"isOpen":1},{"secID":"600654.XSHG","ticker":"600654","secShortName":"飞乐股份","exchangeCD":"XSHG","tradeDate":"1991-10-21","preClosePrice":633.2,"actPreClosePrice":633.2,"openPrice":639.5,"highestPrice":639.5,"lowestPrice":639.5,"closePrice":639.5,"turnoverVol":101,"turnoverValue":64590,"dealAmount":26,"turnoverRate":0.0048,"accumAdjFactor":0.000663586,"negMarketValue":13429500,"marketValue":134358950,"PB":282.9834,"isOpen":1},{"secID":"600656.XSHG","ticker":"600656","secShortName":"浙江凤凰","exchangeCD":"XSHG","tradeDate":"1991-10-21","preClosePrice":1242.9,"actPreClosePrice":1242.9,"openPrice":1255.3,"highestPrice":1255.3,"lowestPrice":1255.3,"closePrice":1255.3,"turnoverVol":140,"turnoverValue":175742,"dealAmount":7,"turnoverRate":0.0031,"accumAdjFactor":0.0007136096,"negMarketValue":56502308.3,"marketValue":321798665.6,"PB":-604.4303,"isOpen":1}]}
2. 创建 hive 临时表
CREATE EXTERNAL TABLE if not exists sensitop.equd_json_tmp (
retCode string,
retMsg string,
data array<struct<
secID: string,
tradeDate: date,
ticker: string,
secShortName: string,
exchangeCD: string,
preClosePrice: double,
actPreClosePrice: double,
openPrice: double,
highestPrice: double,
lowestPrice: double,
closePrice: double,
turnoverVol: double,
turnoverValue: double,
dealAmount: int,
turnoverRate: double,
accumAdjFactor: double,
negMarketValue: double,
marketValue: double,
PE: double,
PE1: double,
PB: double,
isOpen: int>>)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION 'hdfs://hdfs1.wdp:8020/sensitop/finance/equd';
3. 创建 hive 表
需要把上面表里数组里的数据一条一条放入这个表：
CREATE TABLE if not exists sensitop.equd_h(
secID string,
ticker string,
secShortName string,
exchangeCD string,
tradeDate date,
preClosePrice double,
actPreClosePrice double,
openPrice double,
highestPrice double,
lowestPrice double,
closePrice double,
turnoverVol double,
turnoverValue double,
dealAmount int,
turnoverRate double,
accumAdjFactor double,
negMarketValue double,
marketValue double,
PE double,
PE1 double,
PB double,
isOpen int)
partitioned by (year string)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
然后新建一个最张表
CREATE TABLE if not exists sensitop.equd(
secID string,
ticker string,
secShortName string,
exchangeCD string,
tradeDate date,
preClosePrice double,
actPreClosePrice double,
openPrice double,
highestPrice double,
lowestPrice double,
closePrice double,
turnoverVol double,
turnoverValue double,
dealAmount int,
turnoverRate double,
accumAdjFactor double,
negMarketValue double,
marketValue double,
PE double,
PE1 double,
PB double,
isOpen int)
partitioned by (year string)
注意：这里的字段顺序和上面临时表的顺序要一致。
4. 用 Partition 更新数据
insert overwrite table sensitop.equd_tmp
partition (year='2016')
select b.dt.secID,
b.dt.ticker,
b.dt.secShortName,
b.dt.exchangeCD,
b.dt.tradeDate,
b.dt.preClosePrice,
b.dt.actPreClosePrice,
b.dt.openPrice,
b.dt.highestPrice,
b.dt.lowestPrice,
b.dt.closePrice,
b.dt.turnoverVol,
b.dt.turnoverValue,
b.dt.dealAmount,
b.dt.turnoverRate,
b.dt.accumAdjFactor,
b.dt.negMarketValue,
b.dt.marketValue,
b.dt.PE,
b.dt.PE1,
b.dt.PB,
b.dt.isOpen
from sensitop.equd_json_tmp LATERAL VIEW explode(equd_json_tmp.data) b AS dt
where dt.tradedate >= '2016-01-01' and dt.tradedate <= '2016-12-31';
insert overwrite table sensitop.equd
partition (year='2016')
select secID,
ticker,
secShortName,
exchangeCD,
tradeDate,
preClosePrice,
actPreClosePrice,
openPrice,
highestPrice,
lowestPrice,
closePrice,
turnoverVol,
turnoverValue,
dealAmount,
turnoverRate,
accumAdjFactor,
negMarketValue,
marketValue,
PE,
PE1,
PB,
isOpen
from sensitop.equd_tmp dt
where year = '2016';
5. 用nifi实现动态插入数据
这里有二个分支，左边一个是每天20：00更新当年的partion； 右边一个是更新1990 到 2015 年的数据，而且只需要更新一次。
insert overwrite table sensitop.equd_h
partition (year='${year}')
select b.dt.secID,
b.dt.ticker,
b.dt.secShortName,
b.dt.exchangeCD,
b.dt.tradeDate,
b.dt.preClosePrice,
b.dt.actPreClosePrice,
b.dt.openPrice,
b.dt.highestPrice,
b.dt.lowestPrice,
b.dt.closePrice,
b.dt.turnoverVol,
b.dt.turnoverValue,
b.dt.dealAmount,
b.dt.turnoverRate,
b.dt.accumAdjFactor,
b.dt.negMarketValue,
b.dt.marketValue,
b.dt.PE,
b.dt.PE1,
b.dt.PB,
b.dt.isOpen
from sensitop.equd_json_tmp LATERAL VIEW explode(equd_json_tmp.data) b AS dt
where dt.tradedate >= '${year}-01-01' and dt.tradedate <= '${year}-12-31'
insert overwrite table sensitop.equd
partition (year='${year}')
select secID,
ticker,
secShortName,
exchangeCD,
tradeDate,
preClosePrice,
actPreClosePrice,
openPrice,
highestPrice,
lowestPrice,
closePrice,
turnoverVol,
turnoverValue,
dealAmount,
turnoverRate,
accumAdjFactor,
negMarketValue,
marketValue,
PE,
PE1,
PB,
isOpen
from sensitop.equd_tmp dt
where year = '${year}'
NIFI 中国社区 QQ群：595034369
如何快速把hdfs数据动态导入到hive表的更多相关文章
- 11.把文本文件的数据导入到Hive表中
先在hive里面创建一个表 create table mydb2.t3(id int,name string,age int) row format delimited fields terminat ...
- 使用 sqoop 将mysql数据导入到hive表（import）
Sqoop将mysql数据导入到hive表中 先在mysql创建表 CREATE TABLE `sqoop_test` ( `id` ) DEFAULT NULL, `name` varchar() ...
- mysql中把一个表的数据批量导入另一个表中
mysql中把一个表的数据批量导入另一个表中 不管是在网站开发还是在应用程序开发中,我们经常会碰到需要将MySQL或MS SQLServer某个表的数据批量导入到另一个表的情况,甚至有时还需要指定 ...
- 大数据开发实战：Hive表DDL和DML
1.Hive 表 DDL 1.1.创建表 Hive中创建表的完整语法如下: CREATE [EXTERNAL] TABLE [IF NOT EXISTS] table_name [ (col_nam ...
- 将DataFrame数据如何写入到Hive表中
1.将DataFrame数据如何写入到Hive表中?2.通过那个API实现创建spark临时表?3.如何将DataFrame数据写入hive指定数据表的分区中? 从spark1.2 到spark1.3 ...
- 用sqoop将mysql的数据导入到hive表中
1:先将mysql一张表的数据用sqoop导入到hdfs中 准备一张表 需求 将 bbs_product 表中的前100条数据导 导出来 只要id brand_id和 name 这3个字段 数据存 ...
- 用puthivestreaming把hdfs里的数据流到hive表
全景图: 1. 创建hive表 CREATE TABLE IF NOT EXISTS newsinfo.test( name STRING ) CLUSTERED BY (name)INTO 3 ...
- 用sqoop将mysql的数据导入到hive表
一.先将mysql一张表的数据用sqoop导入到hdfs 1.1.先在mysql中准备一张测试用的表 mysql> desc user_info; +-----------+---------- ...
- 从Oracle导出数据并导入到Hive
1.配置源和目标的数据连接 源(oracle): 目标(Hive 2.1.1),需要事先将hive的驱动程序导入HHDI的lib目录中. Hive2.1.1需要的jar包如下:可根据自身情况更换had ...
随机推荐
- 压缩文本、字节或者文件的压缩辅助类-GZipHelper
下面为大家介绍一.NET下辅助公共类GZipHelper,该工具类主要作用是对文本.字符.文件等进行压缩与解压.该类主要使用命名空间:System.IO.Compression下的GZipStream ...
- MVC5+EF6 简易版CMS（非接口） 第四章：使用业务层方法，以及关联表解决方案
目录 简易版CMS后台管理系统开发流程 MVC5+EF6 简易版CMS(非接口) 第一章:新建项目 MVC5+EF6 简易版CMS(非接口) 第二章:建数据模型 MVC5+EF6 简易版CMS(非接口 ...
- Asp.Net MVC4入门指南（4）：添加一个模型
在本节中,您将添加一些类,这些类用于管理数据库中的电影.这些类是ASP.NET MVC 应用程序中的"模型(Model)". 您将使用.NET Framework 数据访问技术En ...
- sql 中 in与exists的对比
1.exists只能用于子查询,可以替代IN,如果查询到结果则退出内部查询,并将条件标记为TRUE,传回全部结果资料 in 不管匹配到匹配不到,都全部匹配 2.根据上面的解释可以得出结论:如果子查询结 ...
- 微信小程序0.11.122100版本新功能解析
微信小程序0.11.122100版本新功能解析 新版本就不再吐槽了,整的自己跟个愤青似的.人老了,喷不动了,把机会留给年轻人吧.下午随着新版本开放,微信居然破天荒的开放了开发者论坛.我很是担心官方 ...
- Java使用Scanner接收中文并输出时出现乱码
Java中使用Scanner接收输入的中文并输出时会出现乱码现象,怎么解决此问题呢? 1.方法一 在声明Scanner时添加对应的编码格式就可以了,如下所示: Scanner sc = new Sca ...
- 原子操作--ARM架构
说明:内核版本号为3.10.101 一.ARM架构中的原子操作实现 在原子操作(一)中我们已经提到,各个架构组织为“复仇者”联盟,统一了基本的原子变量操作,这里我们就拿atomic_dec(v)来看看 ...
- C#中的BackgroundWorker控件+Delegate.Invoke （委托同步调用）
C#中的BackgroundWorker控件+Delegate.Invoke (委托同步调用) 简单代码,记录一下.一个BackgroundWorker控件 backgroundWorkerRefr ...
- Ajax参数详解
1.url: 要求为String类型的参数,(默认为当前页地址)发送请求的地址. 2.type: 要求为String类型的参数,请求方式(post或get)默认为get.注意其他http请求方法,例如 ...
- WinForm程序关闭窗体后杀死进程
调用this.Close();后监听Browser_FormClosed加作处理 this.FormClosed += new System.Windows.Forms.FormClosedEvent ...