LanguageManual DML

Hive Data Manipulation Language

There are multiple ways to modify data in Hive:

EXPORT and IMPORT commands are also available (as of Hive 0.8).

Loading files into tables

Hive does not do any transformation while loading data into tables. Load operations are currently pure copy/move (纯复制,移动) operations that move datafiles into locations corresponding to Hive tables.

Syntax 语法
LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
Synopsis 简介

Load operations are currently pure copy/move operations that move datafiles into locations corresponding to Hive tables.

  • filepath can be:

    • a relative path, such as project/data1
    • an absolute path, such as /user/hive/project/data1
    • a full URI with scheme and (optionally) an authority, such as hdfs://namenode:9000/user/hive/project/data1
  • The target being loaded to can be a table or a partition. If the table is partitioned, then one must specify a specific partition of the table by specifying values for all of the partitioning columns.
  • filepath can refer to a file (in which case Hive will move the file into the table) or it can be a directory (in which case Hive will move all the files within that directory into the table). In either case,filepath addresses a set of files.
  • If the keyword LOCAL is specified, then:
    • the load command will look for filepath in the local file system. If a relative path is specified, it will be interpreted relative to the user's current working directory. The user can specify a full URI for local files as well - for example: file:///user/hive/project/data1
    • the load command will try to copy all the files addressed by filepath to the target filesystem. The target file system is inferred by looking at the location attribute of the table. The copied data files will then be moved to the table.
  • If the keyword LOCAL is not specified, then Hive will either use the full URI of filepath, if one is specified, or will apply the following rules:
    • If scheme or authority are not specified, Hive will use the scheme and authority from the hadoop configuration variable fs.default.name that specifies the Namenode URI.
    • If the path is not absolute, then Hive will interpret it relative to /user/<username>
    • Hive will move the files addressed by filepath into the table (or partition)
  • If the OVERWRITE keyword is used then the contents of the target table (or partition) will be deleted and replaced by the files referred to by filepath; otherwise the files referred by filepath will be added to the table.
    • Note that if the target table (or partition) already has a file whose name collides(冲突) with any of the filenames contained in filepath, then the existing file will be replaced with the new file.
Notes
  • filepath cannot contain subdirectories.(filepath可以是目录也可以是文件,但是不能包含子目录)
  • If the keyword LOCAL is not given, filepath must refer to files within the same filesystem as the table's (or partition's) location.
  • Hive does some minimal checks to make sure that the files being loaded match the target table. Currently it checks that if the table is stored in sequencefile format, the files being loaded are also sequencefiles, and vice versa.
  • A bug that prevented loading a file when its name includes the "+" character is fixed in release 0.13.0 (HIVE-6048).
  • Please read CompressedStorage if your datafile is compressed.

Inserting data into Hive Tables from queries

Query Results can be inserted into tables by using the insert clause.

Syntax 语法
Standard syntax:
INSERT OVERWRITE TABLE tablename1 [PARTITION (partcol1=val1, partcol2=val2 ...) [IF NOT EXISTS]] select_statement1 FROM from_statement;
INSERT INTO TABLE tablename1 [PARTITION (partcol1=val1, partcol2=val2 ...)] select_statement1 FROM from_statement;
 
Hive extension (multiple inserts):
 
FROM from_statement
INSERT OVERWRITE TABLE tablename1 [PARTITION (partcol1=val1, partcol2=val2 ...) [IF NOT EXISTS]] select_statement1
[INSERT OVERWRITE TABLE tablename2 [PARTITION ... [IF NOT EXISTS]] select_statement2]
[INSERT INTO TABLE tablename2 [PARTITION ...] select_statement2] ...;
 
FROM from_statement
INSERT INTO TABLE tablename1 [PARTITION (partcol1=val1, partcol2=val2 ...)] select_statement1
[INSERT INTO TABLE tablename2 [PARTITION ...] select_statement2]
[INSERT OVERWRITE TABLE tablename2 [PARTITION ... [IF NOT EXISTS]] select_statement2] ...;
 
Hive extension (dynamic partition inserts):
INSERT OVERWRITE TABLE tablename PARTITION (partcol1[=val1], partcol2[=val2] ...) select_statement FROM from_statement;
INSERT INTO TABLE tablename PARTITION (partcol1[=val1], partcol2[=val2] ...) select_statement FROM from_statement;
Synopsis
  • INSERT OVERWRITE will overwrite any existing data in the table or partition

    • unless IF NOT EXISTS is provided for a partition (as of Hive 0.9.0).
  • INSERT INTO will append to the table or partition, keeping the existing data intact(完整无缺的). (Note: INSERT INTO syntax is only available starting in version 0.8.)
    • As of Hive 0.13.0, a table can be made immutable(不可变的) by creating it with TBLPROPERTIES ("immutable"="true"). The default is "immutable"="false".
      INSERT INTO behavior into an immutable table is disallowed if any data is already present, although INSERT INTO still works if the immutable table is empty. The behavior of INSERT OVERWRITE is not affected by the "immutable" table property. (INSERT OVERWRITE不受immutable属性的限制)
      An immutable table is protected against accidental updates due to a script loading data into it being run multiple times by mistake. (避免多次插入和修改) The first insert into an immutable table succeeds and successive inserts fail, resulting in only one set of data in the table, instead of silently succeeding with multiple copies of the data in the table.
  • Inserts can be done to a table or a partition. If the table is partitioned, then one must specify a specific partition of the table by specifying values for all of the partitioning columns.
  • Multiple insert clauses (also known as Multi Table Insert) can be specified in the same query.
  • The output of each of the select statements is written to the chosen table (or partition). Currently the OVERWRITE keyword is mandatory(强制的) and implies(暗示,说明) that the contents of the chosen table or partition are replaced with the output of corresponding(适当的) select statement.
  • The output format and serialization class is determined by the table's metadata (as specified via DDL commands on the table).
  • As of Hive 0.14, if a table has an OutputFormat that implements AcidOutputFormat and the system is configured to use a transaction manager that implements ACID, then INSERT OVERWRITE will be disabled for that table.  This is to avoid users unintentionally overwriting transaction history.  The same functionality can be achieved by using TRUNCATE TABLE (for non-partitioned tables) or DROP PARTITION followed by INSERT INTO.
Notes
  • Multi Table Inserts minimize the number of data scans required. Hive can insert data into multiple tables by scanning the input data just once (and applying different query operators) to the input data.
  • Starting with Hive 0.13.0, the select statement can include one or more common table expressions (CTEs) as shown in the SELECT syntax. For an example, see Common Table Expression.
Dynamic Partition Inserts 动态分区插入

Version information

Icon

This information reflects the situation in Hive 0.12; dynamic partition inserts were added in Hive 0.6.

In the dynamic partition inserts, users can give partial partition specifications, which means just specifying the list of partition column names in the PARTITION clause. The column values are optional. If a partition column value is given, we call this a static partition, otherwise it is a dynamic partition. Each dynamic partition column has a corresponding input column from the select statement. This means that the dynamic partition creation is determined by the value of the input column. The dynamic partition columns must be specified last among the columns in the SELECT statement and in the same order in which they appear in the PARTITION() clause.

Dynamic Partition inserts are disabled by default. These are the relevant(相关的) configuration properties for dynamic partition inserts:

Configuration property

Default

Note

hive.exec.dynamic.partition

false

Needs to be set to true to enable dynamic partition inserts

hive.exec.dynamic.partition.mode

strict

In strict mode, the user must specify at least one static partition in case the user accidentally overwrites all partitions, in nonstrictmode all partitions are allowed to be dynamic

hive.exec.max.dynamic.partitions.pernode

100

Maximum number of dynamic partitions allowed to be created in each mapper/reducer node

hive.exec.max.dynamic.partitions

1000

Maximum number of dynamic partitions allowed to be created in total

hive.exec.max.created.files

100000

Maximum number of HDFS files created by all mappers/reducers in a MapReduce job

hive.error.on.empty.partition

false

Whether to throw an exception if dynamic partition insert generates empty results

Example
FROM page_view_stg pvs
INSERT OVERWRITE TABLE page_view PARTITION(dt='2008-06-08', country)
       SELECT pvs.viewTime, pvs.userid, pvs.page_url, pvs.referrer_url, nullnull, pvs.ip, pvs.cnt

Here the country partition will be dynamically created by the last column from the SELECT clause (i.e. pvs.cnt). Note that the name is not used. In nonstrict mode the dt partition could also be dynamically created.

Additional Documentation

Writing data into the filesystem from queries

Query results can be inserted into filesystem directories by using a slight variation (细微的变化)of the syntax above:

Syntax
Standard syntax:
INSERT OVERWRITE [LOCAL] DIRECTORY directory1
  [ROW FORMAT row_format] [STORED AS file_format] (Note: Only available starting with Hive 0.11.0)
  SELECT ... FROM ...
 
Hive extension (multiple inserts):
FROM from_statement
INSERT OVERWRITE [LOCAL] DIRECTORY directory1 select_statement1
[INSERT OVERWRITE [LOCAL] DIRECTORY directory2 select_statement2] ...
 
 
row_format
  : DELIMITED [FIELDS TERMINATED BY char [ESCAPED BY char]] [COLLECTION ITEMS TERMINATED BY char]
        [MAP KEYS TERMINATED BY char] [LINES TERMINATED BY char]
        [NULL DEFINED AS char] (Note: Only available starting with Hive 0.13)
Synopsis
  • Directory can be a full URI. If scheme or authority are not specified, Hive will use the scheme and authority from the hadoop configuration variable fs.default.name that specifies the Namenode URI.
  • If LOCAL keyword is used, Hive will write data to the directory on the local file system.
  • Data written to the filesystem is serialized as text with columns separated by ^A and rows separated by newlines. If any of the columns are not of primitive type, then those columns are serialized to JSON format.
Notes
  • INSERT OVERWRITE statements to directories, local directories, and tables (or partitions) can all be used together within the same query.
  • INSERT OVERWRITE statements to HDFS filesystem directories are the best way to extract large amounts of data from Hive. Hive can write to HDFS directories in parallel from within a map-reduce job.
  • The directory is, as you would expect, OVERWRITten; in other words, if the specified path exists, it is clobbered and replaced with the output.
  • As of Hive 0.11.0 the separator used can be specified, in earlier versions it was always the ^A character (\001)
  • In Hive 0.14, inserts into ACID compliant tables will deactivate vectorization for the duration of the select and insert.  This will be done automatically.  ACID tables that have data inserted into them can still be queried using vectorization.

Inserting values into tables from SQL

The INSERT...VALUES statement can be used to insert data into tables directly from SQL.

Version Information

Icon

INSERT...VALUES is available starting in Hive 0.14.

Inserting values from SQL statements can only be performed on tables that support ACID. See Hive Transactions for details.

Syntax
Standard Syntax:
INSERT INTO TABLE tablename [PARTITION (partcol1[=val1], partcol2[=val2] ...)] VALUES values_row [, values_row ...]
 
Where values_row is:
( value [, value ...] )
where a value is either null or any valid SQL literal
Synopsis
  • Each row listed in the VALUES clause is inserted into table tablename.
  • Values must be provided for every column in the table.  The standard SQL syntax that allows the user to insert values into only some columns is not yet supported.  To mimic the standard SQL, nulls can be provided for columns the user does not wish to assign a value to.
  • Dynamic partitioning is supported in the same way as for INSERT...SELECT.
  • If the table being inserted into supports ACID and a transaction manager that supports ACID is in use, this operation will be auto-committed upon successful completion.
  • Insert, update, delete operations are not supported on tables that are sorted (tables created with the SORTED BY clause).
  • Hive does not support literals for complex types, so it is not possible to use them in INSERT INTO...VALUES clauses. 
  • Means user cannot insert data into complex datatype [array, map, struct, union] columns using INSERT INTO...VALUES clause.
Examples
CREATE TABLE students (name VARCHAR(64), age INT, gpa DECIMAL(3, 2))
  CLUSTERED BY (age) INTO 2 BUCKETS STORED AS ORC;
 
INSERT INTO TABLE students
  VALUES ('fred flintstone', 35, 1.28), ('barney rubble', 32, 2.32);
 
 
 
CREATE TABLE pageviews (userid VARCHAR(64), link STRING, from STRING)
  PARTITIONED BY (datestamp STRING) CLUSTERED BY (userid) INTO 256 BUCKETS STORED AS ORC;
 
 
INSERT INTO TABLE pageviews PARTITION (datestamp = '2014-09-23')
  VALUES ('jsmith''mail.com''sports.com'), ('jdoe''mail.com'null);
 
 
INSERT INTO TABLE pageviews PARTITION (datestamp)
  VALUES ('tjohnson''sports.com''finance.com''2014-09-23'), ('tlee''finance.com'null'2014-09-21');

Update

Version Information

Icon

UPDATE is available starting in Hive 0.14.

Updates can only be performed on tables that support ACID. See Hive Transactions for details.

Syntax
Standard Syntax:
UPDATE tablename SET column = value [, column = value ...] [WHERE expression]
Synopsis
  • The referenced column must be a column of the table being updated.
  • The value assigned must be an expression that Hive supports in the select clause.  Thus arithmetic operators, UDFs, casts, literals, etc. are supported.  Subqueries are not supported.
  • Only rows that match the WHERE clause will be updated.
  • Partitioning columns cannot be updated.
  • Bucketing columns cannot be updated.
  • In Hive 0.14, upon successful completion of this operation the changes will be auto-committed.
Notes
  • Vectorization will be turned off for update operations.  This is automatic and requires no action on the part of the user.  Non-update operations are not affected.  Updated tables can still be queried using vectorization.
  • In version 0.14 it is recommended that you set hive.optimize.sort.dynamic.partition=false when doing updates, as this produces more efficient execution plans.

Delete

Version Information

Icon

DELETE is available starting in Hive 0.14.

Deletes can only be performed on tables that support ACID. See Hive Transactions for details.

Syntax
Standard Syntax:
DELETE FROM tablename [WHERE expression]
Synopsis
  • Only rows that match the WHERE clause will be deleted.
  • In Hive 0.14, upon successful completion of this operation the changes will be auto-committed.
Notes
    • Vectorization will be turned off for delete operations.  This is automatic and requires no action on the part of the user.  Non-delete operations are not affected.  Tables with deleted data can still be queried using vectorization.
    • In version 0.14 it is recommended that you set hive.optimize.sort.dynamic.partition=false when doing deletes, as this produces more efficient execution plans.

[Hive - LanguageManual] DML: Load, Insert, Update, Delete的更多相关文章

  1. MySQL之DML语句(insert update delete)

    DML主要针对数据库表对象的数据而言的,一般DML完成: 插入新数据 修改已添加的数据 删除不需要的数据 1.insert into插入语句 //主键自增可以不插入,所以用null代替 ); //指定 ...

  2. JDBC基础篇(MYSQL)——使用statement执行DML语句(insert/update/delete)

    注意:其中的JdbcUtil是我自定义的连接工具类:代码例子链接: package day02_statement; import java.sql.Connection; import java.s ...

  3. mysql数据恢复 insert\update\delete 工具MyFlash

    一.简介MyFlash是由美团点评公司技术工程部开发维护的一个回滚DML操作的工具.该工具通过解析v4版本的binlog,完成回滚操作.相对已有的回滚工具,其增加了更多的过滤选项,让回滚更加容易. 该 ...

  4. 关于MyBatis mapper的insert, update, delete返回值

    这里做了比较清晰的解释: http://mybatis.github.io/mybatis-3/java-api.html SqlSession As mentioned above, the Sql ...

  5. PHP5: mysqli 插入, 查询, 更新和删除 Insert Update Delete Using mysqli (CRUD)

    原文: PHP5: mysqli 插入, 查询, 更新和删除  Insert Update Delete Using mysqli (CRUD) PHP 5 及以上版本建议使用以下方式连接 MySQL ...

  6. mysql 事务是专门用来管理insert,update,delete语句的,和select语句一点不相干

    1.mysql 事务是专门用来管理insert,update,delete语句的,和select语句一点不相干 2.一般来说,事务是必须满足4个条件(ACID): Atomicity(原子性).Con ...

  7. insert update delete 语法 以及用法

    insert update delete 被称为 数据定义语句语句 也就是数据的增加 修改 删除 其中不包括查询 譬如: create database -创建数据库 alter database - ...

  8. sql中同一个Trigger里同时包含Insert,Update,Delete

    sql中同一个Trigger里同时包含Insert,Update,Delete SQLServer是靠Inserted表和Deleted表来处理的,判断一下就可以了,只不过比ORACLE麻烦一点 cr ...

  9. mybatis select/insert/update/delete

    这里做了比较清晰的解释: http://mybatis.github.io/mybatis-3/java-api.html SqlSession As mentioned above, the Sql ...

随机推荐

  1. linq 分类

    linq技术为我们开发人员提供了五个比较实用的数据访问类型: LinQ to Object:可以允许对内存中的类对象查询. LinQ to DataSet:可以对内存中的DataSet缓存数据,执行数 ...

  2. Tomcat的SessionID引起的Session Fixation和Session Hijacking问题

    上一篇说到<Spring MVC防御CSRF.XSS和SQL注入攻击>,今天说说SessionID带来的漏洞攻击问题.首先,什么是Session Fixation攻击和Session Hi ...

  3. eclipse的安装

    eclipse的安装只要记住一条就好,那就是eclipse安装的时候版本要一致(就是,安装eclipse 32位的,同时要配套jdk 32位,tomcat 32位的),同理64位的也是一样的. 本人安 ...

  4. webbrowser代理c#代码实现

    微软webbrowser控件也就是IE插件,他的所有功能就像IE类似,当然设置也是一样的,下面介绍下webbrowser如何设置代理,可不要用这个对抗广告联盟哦 You can change the ...

  5. uva10820Send a Table

    筛法. 首先使cnt[i]=sqr(n/i),这样cnt[i]就表示gcd(x,y)大于等于i的数对的个数,然后倒序枚举减去gcd大于i的个数就可以得到ans[i].最终得到ans[1]. 这个算法单 ...

  6. iOS开发:应用生命周期

    iOS应用通过委托对象AppDelegate类在应用周期的不同阶段会回调不同的方法,应用周期分为以下五种状态: Not Running(非运行状态).应用没有运行或被系统终止.   Inactive ...

  7. bzoj1832: [AHOI2008]聚会

    写过的题... #include<cstdio> #include<cstring> #include<iostream> #include<algorith ...

  8. ionic preparing for ios 9

    http://blog.ionic.io/preparing-for-ios-9/ Preparing for iOS 9 Tweet   By Max on September 12, 2015 i ...

  9. apache开源项目-- UIMA

    UIMA (Unstructured Information Management applications) 是一个软件系统,用来分析大量的非结构化信息从而发掘中对最终用户有用的知识点,一个最典型的 ...

  10. CMake实践(1)

    简介: 目录结构t1/main.cpp; t1/CMakeLists.txt 说明: main.cpp: #include <stdio.h> int main(){    printf( ...