Hive学习笔记——metadata

Hive结构体系

https://blog.csdn.net/zhoudaxia/article/details/8855937

可以在hive的jdbc接口中使用getMetaData方法来获取hive表的相关元信息

statement = connection.createStatement();
DatabaseMetaData meta = connection.getMetaData();

参考

https://blog.csdn.net/u010368839/article/details/76358831

hive metadata源码解析可以参考

https://cloud.tencent.com/developer/article/1330250

hive thrift接口可以参考

http://www.laicar.com/book/echapter/5cb0bcfe739207662ac88ed1/links/x_Chapter_16.html/OEBPS/Text/part0024.xhtml

获得表的信息接口，指定tableNamePattern为hive表名

ResultSet tableRet = meta.getTables(null, "%", "ads_nsh_trade", new String[]{"TABLE"});
while (tableRet.next()) {
    System.out.println("TABLE_CAT:" + tableRet.getString("TABLE_CAT"));
    System.out.println("TABLE_SCHEM:" + tableRet.getString("TABLE_SCHEM"));
    System.out.println("TABLE_NAME => " + tableRet.getString("TABLE_NAME"));
    System.out.println("table_type => " + tableRet.getString("table_type"));
    System.out.println("remarks => " + tableRet.getString("remarks"));
    System.out.println("type_cat => " + tableRet.getString("type_cat"));
    System.out.println("type_schem => " + tableRet.getString("type_schem"));
    System.out.println("type_name => " + tableRet.getString("type_name"));
    System.out.println("self_referencing_col_name => " + tableRet.getString("self_referencing_col_name"));
    System.out.println("ref_generation => " + tableRet.getString("ref_generation"));
}

其中的参数可以是

table_cat, table_schem, table_name, table_type, remarks, type_cat, type_schem, type_name, self_referencing_col_name, ref_generation

如果填写不正确将会抛出异常

java.sql.SQLException: Could not find COLUMN_NAME in [table_cat, table_schem, table_name, table_type, remarks, type_cat, type_schem, type_name, self_referencing_col_name, ref_generation]
	at org.apache.hive.jdbc.HiveBaseResultSet.findColumn(HiveBaseResultSet.java:100)
	at org.apache.hive.jdbc.HiveBaseResultSet.getString(HiveBaseResultSet.java:541)

输出的结果

TABLE_CAT:
TABLE_SCHEM:tmp
TABLE_NAME => ads_nsh_trade
table_type => TABLE
remarks => ???????????
type_cat => null
type_schem => null
type_name => null
self_referencing_col_name => null
ref_generation => null

TABLE_CAT:
TABLE_SCHEM:default
TABLE_NAME => ads_nsh_trade
table_type => TABLE
remarks => null
type_cat => null
type_schem => null
type_name => null
self_referencing_col_name => null
ref_generation => null

如果再指定schemaPattern为hive库名

ResultSet tableRet = meta.getTables(null, "default", "ads_nsh_trade", new String[]{"TABLE"});
while (tableRet.next()) {
    System.out.println("TABLE_CAT:" + tableRet.getString("TABLE_CAT"));
    System.out.println("TABLE_SCHEM:" + tableRet.getString("TABLE_SCHEM"));
    System.out.println("TABLE_NAME => " + tableRet.getString("TABLE_NAME"));
    System.out.println("table_type => " + tableRet.getString("table_type"));
    System.out.println("remarks => " + tableRet.getString("remarks"));
    System.out.println("type_cat => " + tableRet.getString("type_cat"));
    System.out.println("type_schem => " + tableRet.getString("type_schem"));
    System.out.println("type_name => " + tableRet.getString("type_name"));
    System.out.println("self_referencing_col_name => " + tableRet.getString("self_referencing_col_name"));
    System.out.println("ref_generation => " + tableRet.getString("ref_generation"));
}

输出结果

TABLE_CAT:
TABLE_SCHEM:default
TABLE_NAME => ads_nsh_trade
table_type => TABLE
remarks => null
type_cat => null
type_schem => null
type_name => null
self_referencing_col_name => null
ref_generation => null

在hive的元数据表中，表的信息主要在TBLS和TABLE_PARAMS这两张表中

参考

https://blog.csdn.net/haozhugogo/article/details/73274832

比如TBLS表

和TABLE_PARAMS表

获得表的字段信息的接口

ResultSet rs1 = meta.getColumns("default", "%", "ads_nsh_trade", "%");
while (rs1.next()) {
    String tableCat = rs1.getString("table_cat");
    String tableSchem = rs1.getString("table_schem");
    String tableName = rs1.getString("table_name");
    String columnName = rs1.getString("COLUMN_NAME");
    String columnType = rs1.getString("TYPE_NAME");
    String remarks = rs1.getString("REMARKS");
    int datasize = rs1.getInt("COLUMN_SIZE");
    int digits = rs1.getInt("DECIMAL_DIGITS");
    int nullable = rs1.getInt("NULLABLE");
    System.out.println(tableCat + " " + tableSchem + " " + tableName + " " + columnName + " " +
                        columnType + " " + datasize + " " + digits + " " + nullable + " " + remarks);
}

其中的参数可以是

table_cat, table_schem, table_name, column_name, data_type, type_name, column_size, buffer_length, decimal_digits, num_prec_radix, nullable, 

remarks, column_def, sql_data_type, sql_datetime_sub, char_octet_length, ordinal_position, is_nullable, scope_catalog, scope_schema, scope_table, source_data_type, is_auto_increment

输出的结果

null default ads_nsh_trade test_string STRING 2147483647 0 1 string??????
null default ads_nsh_trade test_boolean BOOLEAN 0 0 1 boolean??????
null default ads_nsh_trade test_short SMALLINT 5 0 1 short??????
null default ads_nsh_trade test_double DOUBLE 15 15 1 double??????
null default ads_nsh_trade test_byte TINYINT 3 0 1 byte??????
null default ads_nsh_trade test_list array<string> 0 0 1 list<String>????
null default ads_nsh_trade test_map map<string,int> 0 0 1 map<String,Int>????
null default ads_nsh_trade test_int INT 10 0 1 int??????
null default ads_nsh_trade test_set array<bigint> 0 0 1 set<Long>??????
null default ads_nsh_trade col_name DECIMAL 10 2 1 null
null default ads_nsh_trade col_name2 DECIMAL 10 2 1 null
null default ads_nsh_trade test_long BIGINT 19 0 1 null
null tmp ads_nsh_trade test_boolean BOOLEAN 0 0 1 boolean??????
null tmp ads_nsh_trade test_short SMALLINT 5 0 1 short??????
null tmp ads_nsh_trade test_double DOUBLE 15 15 1 double??????
null tmp ads_nsh_trade test_byte TINYINT 3 0 1 byte??????
null tmp ads_nsh_trade test_list array<string> 0 0 1 list<String>????
null tmp ads_nsh_trade test_map map<string,int> 0 0 1 map<String,Int>????
null tmp ads_nsh_trade test_int INT 10 0 1 int??????
null tmp ads_nsh_trade test_set array<bigint> 0 0 1 set<Long>??????
null tmp ads_nsh_trade test_long BIGINT 19 0 1 null
null tmp ads_nsh_trade test_string STRING 2147483647 0 1 null