将数据导入hive，再将hive表导入hbase

将数据到入hive的无分区表，再将无分区表导入hive的有分区表：

--备份
create table tds_package_secinfobk as select * from tds_package_secinfo;
--导入分区表
tds_package_secinfo

CREATE TABLE  tds_package_secinfo
(package_name string,program_name string,keyhash string)
PARTITIONED BY(risk_label string)
ROW FORMAT delimited fields terminated by '\t'
LOCATION '/user/hive/warehouse/tds_package_secinfo'

LOAD DATA INPATH '/user/hive/warehouse/邪教app.txt' OVERWRITE INTO TABLE tds_package_secinfo PARTITION(risk_label='邪教app');

--无分区表导有分区表
set hive.exec.max.dynamic.partitions= ;
set hive.exec.max.dynamic.partitions.pernode= ;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.support.concurrency=false;
set mapreduce.job.reduce.slowstart.completedmaps=;
insert into table tds_package_secinfo partition(risk_label)
select  package_name ,'' ,'' ,risk_label from  hue_package_secinfo;

hbase_package_secinfo

查询该hive表的包名和标签列：

select package_name,risk_label from tds_package_secinfo

hbase表需要先建表，才能插入数据

建表：

CREATE TABLE hbase_package_secinfo
(package_name string,risk_label string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES
("hbase.columns.mapping" = ":key,info:risk_lable") TBLPROPERTIES ("hbase.table.name"="hbase_package_secinfo");

插入数据：

insert into hbase_package_secinfo
select package_name,risk_label from tds_package_secinfo 

-------------------------------------------------建表完成-----------------------------------------------------------

查询语句：

select count(*),dt from MDS_ENGINE_VIR_URL group by dt order by dt desc; 

select count(*),dt from MDS_ENGINE_WARESOURCE_INFO_NEW group by dt order by dt desc;  

select count(*),dt from MDS_VICTIM_INFO group by dt order by dt desc;   

--------------------------------------------

异常：

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"package_name":"","program_name":"","keyhash":"","risk_label":"__HIVE_DEFAULT_PARTITION__"}
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)
    at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"package_name":"","program_name":"","keyhash":"","risk_label":"__HIVE_DEFAULT_PARTITION__"}
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:)
    ...  more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.serde2.SerDeException: java.lang.IllegalArgumentException: Row length is
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:)
    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:)
    ...  more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: java.lang.IllegalArgumentException: Row length is
    at org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:)
    ...  more
Caused by: java.lang.IllegalArgumentException: Row length is
    at org.apache.hadoop.hbase.client.Mutation.checkRow(Mutation.java:)
    at org.apache.hadoop.hbase.client.Put.<init>(Put.java:)
    at org.apache.hadoop.hbase.client.Put.<init>(Put.java:)
    at org.apache.hadoop.hbase.client.Put.<init>(Put.java:)
    at org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:)
    at org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:)
    ...  more

原因：表中的内容为空，所以插不进去

解决方法：加上条件：

where risk_label!='__HIVE_DEFAULT_PARTITION__'

所以插入语句应该为：

insert into hbase_package_secinfo
select package_name,risk_label from tds_package_secinfo where risk_label!='__HIVE_DEFAULT_PARTITION__'; 

最后删除risk_label":"__HIVE_DEFAULT_PARTITION__的那个分区

ALTER TABLE tds_package_secinfo DROP IF EXISTS PARTITION(risk_label='__HIVE_DEFAULT_PARTITION__' );