kudu导入文件（基于impala）

　　kudu是cloudera开源的运行在hadoop平台上的列式存储系统,拥有Hadoop生态系统应用的常见技术特性，运行在一般的商用硬件上，支持水平扩展,高可用,集成impala后，支持标准sql语句，相对于hbase易用性强，详细介绍。

　　impala是Cloudera公司主导开发的新型查询系统，它提供SQL语义，能查询存储在Hadoop的HDFS和HBase中的PB级大数据。已有的Hive系统虽然也提供了SQL语义，但由于Hive底层执行使用的是MapReduce引擎，仍然是一个批处理过程，难以满足查询的交互性。相比之下，Impala的最大特点也是最大卖点就是它的快速，导入数据实测可达30+W/s,详细介绍。

导入流程：准备数据--》上传hdfs--》导入impala临时表--》导入kudu表

1.准备数据

app@hadoop01:/software/develop/pujh>cat genBiData.sh
#!/usr/bash

date
echo ''>data.txt
chmod  data.txt

for((i=;i<=20593279;i++))
do
 echo "$i|aa$i|aa$i$i|aa$i$i$i" >>data.txt;
done;

date

app@hadoop01:/software/develop/pujh> sed 's/|/,/g' data.txt > temp.csv

app@hadoop01:/software/develop/pujh>chmod 777 tmp.csv

2.上传到hdfs

su - root
su - hdfs
hadoop dfs -mkdir /input/data/pujh
hadoop dfs -chmod -R  /input/data/pujh
hadoop dfs -put /software/develop/pujh /input/data/pujh
hadoop dfs -ls /input/data/pujh
hdfs@hadoop01:>./hadoop dfs -ls /input/data/pujh
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Found  items
-rwxrwxrwx    hdfs supergroup           -- : /input/data/pujh/aa.txt
-rwxrwxrwx    hdfs supergroup  -- : /input/data/pujh/data.txt
-rwxrwxrwx    hdfs supergroup  -- : /input/data/pujh/data2kw.csv
-rwxrwxrwx    hdfs supergroup  -- : /input/data/pujh/data_2kw.txt
-rwxrwxrwx    hdfs supergroup         -- : /input/data/pujh/genBiData.sh

3.导入impala临时表

创建impala临时表

employee_temp

create table employee_temp ( eid int, name String,salary String, destination String) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';hdfs@hadoop02>./impala-shell 

Starting Impala Shell without Kerberos authentication
Connected to hadoop02:
Server version: impala version 2.8.-cdh5.11.2 RELEASE (build f89269c4b96da14a841e94bdf6d4d48821b0d658)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.8.0-cdh5.11.2 (f89269c) built on Fri Aug  :: PDT )

The HISTORY command lists all shell commands in chronological order.
***********************************************************************************
[hadoop02:] > show databases;
Query: show databases
+------------------+----------------------------------------------+
| name             | comment                                      |
+------------------+----------------------------------------------+
| _impala_builtins | System database for Impala builtin functions |
| default          | Default Hive database                        |
| td_test          |                                              |
+------------------+----------------------------------------------+
Fetched  row(s) in .01s
[hadoop02:] > show tables;
Query: show tables
+----------------+
| name           |
+----------------+
| employee       |
| my_first_table |
+----------------+
Fetched  row(s) in .00s

[hadoop02:] > create table employee_temp ( eid int, name String,salary String, destination String) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
Query: create table employee_temp ( eid int, name String,salary String, destination String) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','

Fetched  row(s) in .32s
[hadoop02:] > show tables;
Query: show tables
+----------------+
| name           |
+----------------+
| employee       |
| employee_temp  |
| my_first_table |
+----------------+
Fetched  row(s) in .01s

将hadoop上的文件导入impala临时表

load data inpath '/input/data/pujh/temp.csv' into table employee_temp;

[hadoop02:] > load data inpath '/input/data/pujh/temp.csv' into table employee_temp;
Query: load data inpath '/input/data/pujh/temp.csv' into table employee_temp
ERROR: AnalysisException: Unable to LOAD DATA from hdfs://hadoop01:8020/input/data/pujh/temp.csv because Impala does not have WRITE permissions on its parent directory hdfs://hadoop01:8020/input/data/pujh

[hadoop02:] > load data inpath '/input/data/pujh/temp.csv' into table employee_temp;
Query: load data inpath '/input/data/pujh/temp.csv' into table employee_temp
+----------------------------------------------------------+
| summary                                                  |
+----------------------------------------------------------+
| Loaded  file(s). Total files in destination location:  |
+----------------------------------------------------------+
Fetched  row(s) in .44s
[hadoop02:] > select * from employee_temp limit ;
Query: select * from employee_temp limit
Query submitted at: -- :: (Coordinator: http://hadoop02:25000)
Query progress can be monitored at: http://hadoop02:25000/query_plan?query_id=4246eaa38a3d8bbb:953ce4d300000000
+------+------+--------+-------------+
| eid  | name | salary | destination |
+------+------+--------+-------------+
| NULL | NULL |        |             |
|     | aa1  | aa11   | aa111       |
+------+------+--------+-------------+
Fetched  row(s) in .19s
[hadoop02:] > select * from employee_temp limit ;
Query: select * from employee_temp limit
Query submitted at: -- :: (Coordinator: http://hadoop02:25000)
Query progress can be monitored at: http://hadoop02:25000/query_plan?query_id=cb4c3cf5d647c97a:75d2985f00000000
+------+------+--------+-------------+
| eid  | name | salary | destination |
+------+------+--------+-------------+
| NULL | NULL |        |             |
|     | aa1  | aa11   | aa111       |
|     | aa2  | aa22   | aa222       |
|     | aa3  | aa33   | aa333       |
|     | aa4  | aa44   | aa444       |
|     | aa5  | aa55   | aa555       |
|     | aa6  | aa66   | aa666       |
|     | aa7  | aa77   | aa777       |
|     | aa8  | aa88   | aa888       |
|     | aa9  | aa99   | aa999       |
+------+------+--------+-------------+
Fetched  row(s) in .02s
[hadoop02:] > select count(*) from employee_temp;
Query: select count(*) from employee_temp
Query submitted at: -- :: (Coordinator: http://hadoop02:25000)
Query progress can be monitored at: http://hadoop02:25000/query_plan?query_id=5a4c1107de118395:bfe96a1600000000
+----------+
| count(*) |
+----------+
|  |
+----------+
Fetched  row(s) in .65s

3.从impala临时表employee_temp 导入kudu表employee_kudu

创建kudu表

create table employee_kudu ( eid int, name String,salary String, destination String,PRIMARY KEY(eid)) PARTITION BY HASH PARTITIONS 16 STORED AS KUDU;

[hadoop02:] > create table employee_kudu ( eid int, name String,salary String, destination String,PRIMARY KEY(eid)) PARTITION BY HASH PARTITIONS  STORED AS KUDU;
Query: create table employee_kudu ( eid int, name String,salary String, destination String,PRIMARY KEY(eid)) PARTITION BY HASH PARTITIONS  STORED AS KUDU

Fetched  row(s) in .94s
[hadoop02:] > show tables;
Query: show tables
+----------------+
| name           |
+----------------+
| employee       |
| employee_kudu  |
| employee_temp  |
| my_first_table |

界面查看是否创建成功

从impala临时表employee_temp 导入kudu表employee_kudu

[hadoop02:] > insert into employee_kudu select * from employee_temp;
Query: insert into employee_kudu select * from employee_temp
Query submitted at: -- :: (Coordinator: http://hadoop02:25000)
Query progress can be monitored at: http://hadoop02:25000/query_plan?query_id=2e47536cc5c82392:ef4d552600000000
WARNINGS: Row with null value violates nullability constraint on table 'impala::default.employee_kudu'.

Modified  row(s),  row error(s) in .75s
[hadoop02:] > select count(*) from employee_kudu;
Query: select count(*) from employee_kudu
Query submitted at: -- :: (Coordinator: http://hadoop02:25000)
Query progress can be monitored at: http://hadoop02:25000/query_plan?query_id=6d4bad44a980f229:fd7878d00000000
+----------+
| count(*) |
+----------+
|  |
+----------+
Fetched  row(s) in .18s