Hive metastore源码阅读（一）

　　不要问我为什么，因为爱，哈哈哈哈。。。进入正题，最近做项目顺带学习了下hive metastore的源码，进行下知识总结。

　　hive metastore的整体架构如图：

　　

　　一、组成结构：

　　如图我们可以看到，hive metastore的组成结构分为 客户端 服务端 ，那么下来我们逐一进行分析：

　　1、客户端

　　

　　 从代码的角度来看：尼玛太多了。。我们从入口HIVE开始看，可以找到MetaStoreClient客户端的创建：

　　

   private IMetaStoreClient createMetaStoreClient() throws MetaException {

     HiveMetaHookLoader hookLoader = new HiveMetaHookLoader() {
         @Override
         public HiveMetaHook getHook(
           org.apache.hadoop.hive.metastore.api.Table tbl)
           throws MetaException {

           try {
             if (tbl == null) {
               return null;
             }
             HiveStorageHandler storageHandler =
               HiveUtils.getStorageHandler(conf,
                 tbl.getParameters().get(META_TABLE_STORAGE));
             if (storageHandler == null) {
               return null;
             }
             return storageHandler.getMetaHook();
           } catch (HiveException ex) {
             LOG.error(StringUtils.stringifyException(ex));
             throw new MetaException(
               "Failed to load storage handler:  " + ex.getMessage());
           }
         }
       };
     return RetryingMetaStoreClient.getProxy(conf, hookLoader, metaCallTimeMap,
         SessionHiveMetaStoreClient.class.getName());
   }

　　我们可以看到，创建MetaStoreClient中，创建了HiveMetaHook,这个Hook的作用在于，每次对meta进行操作的时候，比如createTable的时候，如果建表的存储方式不是文件，比如集成hbase，HiveMetaStoreClient会调用hook的接口方法preCreateTable，进行建表前的准备，用来判断外部表与内部表，如果中途有失败的话，依旧调用hook中的rollbackCreateTable进行回滚。

   public void createTable(Table tbl, EnvironmentContext envContext) throws AlreadyExistsException,
       InvalidObjectException, MetaException, NoSuchObjectException, TException {
     HiveMetaHook hook = getHook(tbl);
     if (hook != null) {
       hook.preCreateTable(tbl);
     }
     boolean success = false;
     try {
       // Subclasses can override this step (for example, for temporary tables)
       create_table_with_environment_context(tbl, envContext);
       if (hook != null) {
         hook.commitCreateTable(tbl);
       }
       success = true;
     } finally {
       if (!success && (hook != null)) {
         hook.rollbackCreateTable(tbl);
       }
     }
   }

　　在hbase表不存在的情况下，不能create external table ,会报doesn't exist while the table is declared as an external table,那么需直接创建create table 创建一个指向hbase的hive表。

　　建表语句如下：

CREATE TABLE hbase_table_1(key int, value string) /
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "tableName", "hbase.mapred.output.outputtable" = "tableName");

　　代码请查看HBaseStorageHandler的preCreateTable方法，这里就不贴出来啦。

　　随之回归Hive类，Hive类可以说是整个元数据DDL操作的最顶层抽象。HiveMetaStoreClient实现了IMetaStoreClient的接口，在创建HiveMetaStoreClient时，会创建于server段HiveMetaStore的链接，并且会通过检查hive.metastore.local是否为true,来决定是在本地创建服务端，这里为在本地:

   public HiveMetaStoreClient(HiveConf conf, HiveMetaHookLoader hookLoader)
     throws MetaException {

     this.hookLoader = hookLoader;
     if (conf == null) {
       conf = new HiveConf(HiveMetaStoreClient.class);
     }
     this.conf = conf;
     filterHook = loadFilterHooks();

     String msUri = conf.getVar(HiveConf.ConfVars.METASTOREURIS);
     localMetaStore = HiveConfUtil.isEmbeddedMetaStore(msUri);
     if (localMetaStore) {
       // instantiate the metastore server handler directly instead of connecting
       // through the network
       client = HiveMetaStore.newRetryingHMSHandler("hive client", conf, true);
       isConnected = true;
       snapshotActiveConf();
       return;
     }

　　随后，创建server端的HiveMetaStore.HMSHandler，HMSHandler继承自IHMSHandler，而IHMSHandler又继承自ThriftHiveMetastore.Iface，在HMSHandler中实现了所有操作的对外方法：

public class ThriftHiveMetastore {

  /**
   * This interface is live.
   */
  public interface Iface extends com.facebook.fb303.FacebookService.Iface {

    public String getMetaConf(String key) throws MetaException, org.apache.thrift.TException;

    public void setMetaConf(String key, String value) throws MetaException, org.apache.thrift.TException;

    public void create_database(Database database) throws AlreadyExistsException, InvalidObjectException, MetaException, org.apache.thrift.TException;

    public Database get_database(String name) throws NoSuchObjectException, MetaException, org.apache.thrift.TException;

    public void drop_database(String name, boolean deleteData, boolean cascade) throws NoSuchObjectException, InvalidOperationException, MetaException, org.apache.thrift.TException;

    public List<String> get_databases(String pattern) throws MetaException, org.apache.thrift.TException;

    public List<String> get_all_databases() throws MetaException, org.apache.thrift.TException;

    public void alter_database(String dbname, Database db) throws MetaException, NoSuchObjectException, org.apache.thrift.TException;

    public Type get_type(String name) throws MetaException, NoSuchObjectException, org.apache.thrift.TException;

    public boolean create_type(Type type) throws AlreadyExistsException, InvalidObjectException, MetaException, org.apache.thrift.TException;

    public boolean drop_type(String type) throws MetaException, NoSuchObjectException, org.apache.thrift.TException;

    ......

　　在创建HiveMetaStore的init方法中，同时创建了三种Listener---MetaStorePreEventListener，MetaStoreEventListener，MetaStoreEndFunctionListener用于对每一步事件的监听。

       initListeners = MetaStoreUtils.getMetaStoreListeners(
           MetaStoreInitListener.class, hiveConf,
           hiveConf.getVar(HiveConf.ConfVars.METASTORE_INIT_HOOKS));
       for (MetaStoreInitListener singleInitListener: initListeners) {
           MetaStoreInitContext context = new MetaStoreInitContext();
           singleInitListener.onInit(context);
       }

       String alterHandlerName = hiveConf.get("hive.metastore.alter.impl",
           HiveAlterHandler.class.getName());
       alterHandler = (AlterHandler) ReflectionUtils.newInstance(MetaStoreUtils.getClass(
           alterHandlerName), hiveConf);
       wh = new Warehouse(hiveConf);

       synchronized (HMSHandler.class) {
         if (currentUrl == null || !currentUrl.equals(MetaStoreInit.getConnectionURL(hiveConf))) {
           createDefaultDB();
           createDefaultRoles();
           addAdminUsers();
           currentUrl = MetaStoreInit.getConnectionURL(hiveConf);
         }
       }

       if (hiveConf.getBoolean("hive.metastore.metrics.enabled", false)) {
         try {
           Metrics.init();
         } catch (Exception e) {
           // log exception, but ignore inability to start
           LOG.error("error in Metrics init: " + e.getClass().getName() + " "
               + e.getMessage(), e);
         }
       }

       preListeners = MetaStoreUtils.getMetaStoreListeners(MetaStorePreEventListener.class,
           hiveConf,
           hiveConf.getVar(HiveConf.ConfVars.METASTORE_PRE_EVENT_LISTENERS));
       listeners = MetaStoreUtils.getMetaStoreListeners(MetaStoreEventListener.class, hiveConf,
           hiveConf.getVar(HiveConf.ConfVars.METASTORE_EVENT_LISTENERS));
       listeners.add(new SessionPropertiesListener(hiveConf));
       endFunctionListeners = MetaStoreUtils.getMetaStoreListeners(
           MetaStoreEndFunctionListener.class, hiveConf,
           hiveConf.getVar(HiveConf.ConfVars.METASTORE_END_FUNCTION_LISTENERS));

　　同时创建了AlterHandler，它是HiveAlterHandler的接口，是将修改表和修改partition的操作抽离了出来单独实现（修改表很复杂的。。）。

 public interface AlterHandler extends Configurable {

   /**
    * handles alter table
    *
    * @param msdb
    *          object to get metadata
    * @param wh
    *          TODO
    * @param dbname
    *          database of the table being altered
    * @param name
    *          original name of the table being altered. same as
    *          <i>newTable.tableName</i> if alter op is not a rename.
    * @param newTable
    *          new table object
    * @throws InvalidOperationException
    *           thrown if the newTable object is invalid
    * @throws MetaException
    *           thrown if there is any other error
    */
   public abstract void alterTable(RawStore msdb, Warehouse wh, String dbname,
       String name, Table newTable) throws InvalidOperationException,
       MetaException;

   /**
    * handles alter table, the changes could be cascaded to partitions if applicable
    *
    * @param msdb
    *          object to get metadata
    * @param wh
    *          Hive Warehouse where table data is stored
    * @param dbname
    *          database of the table being altered
    * @param name
    *          original name of the table being altered. same as
    *          <i>newTable.tableName</i> if alter op is not a rename.
    * @param newTable
    *          new table object
    * @param cascade
    *          if the changes will be cascaded to its partitions if applicable
    * @throws InvalidOperationException
    *           thrown if the newTable object is invalid
    * @throws MetaException
    *           thrown if there is any other error
    */
   public abstract void alterTable(RawStore msdb, Warehouse wh, String dbname,
       String name, Table newTable, boolean cascade) throws InvalidOperationException,
       MetaException;

　　最重要的是RawStore的创建。RawStore不光是定义了一套最终的物理操作，使用JDO将一个对象当作表进行存储。ObjectStore中的transaction机制也是通过JDO提供的transaction实现的。当commit失败时，将rollback所有操作。

　　

   @Override
   public void createDatabase(Database db) throws InvalidObjectException, MetaException {
     boolean commited = false;
     MDatabase mdb = new MDatabase();
     mdb.setName(db.getName().toLowerCase());
     mdb.setLocationUri(db.getLocationUri());
     mdb.setDescription(db.getDescription());
     mdb.setParameters(db.getParameters());
     mdb.setOwnerName(db.getOwnerName());
     PrincipalType ownerType = db.getOwnerType();
     mdb.setOwnerType((null == ownerType ? PrincipalType.USER.name() : ownerType.name()));
     try {
       openTransaction();
       pm.makePersistent(mdb);
       commited = commitTransaction();
     } finally {
       if (!commited) {
         rollbackTransaction();
       }
     }
   }

   @SuppressWarnings("nls")
   private MDatabase getMDatabase(String name) throws NoSuchObjectException {
     MDatabase mdb = null;
     boolean commited = false;
     try {
       openTransaction();
       name = HiveStringUtils.normalizeIdentifier(name);
       Query query = pm.newQuery(MDatabase.class, "name == dbname");
       query.declareParameters("java.lang.String dbname");
       query.setUnique(true);
       mdb = (MDatabase) query.execute(name);
       pm.retrieve(mdb);
       commited = commitTransaction();
     } finally {
       if (!commited) {
         rollbackTransaction();
       }
     }
     if (mdb == null) {
       throw new NoSuchObjectException("There is no database named " + name);
     }
     return mdb;
   }

　　今晚就到这里。。。自己摸索，如有错误，还望指出谢谢~