hive Metastore contains multiple versions

凌晨接到hive作业异常，hive版本为1.2.1，hadoop版本apache 2.7.1，元数据存储在mysql中，异常信息如下：

Logging initialized using configuration in jar:file:/opt/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.
at
at org.apache.hadoop.util.RunJar.run(RunJar.
at org.apache.hadoop.util.RunJar.main(RunJar.
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.
... 8 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.
at
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.
... 14 more
Caused by: MetaException(message:Metastore contains multiple versions (2) [ version = 1.2.0, comment = Set by MetaStore *@*.*.*.* ] [ version = 1.2.0, comment = Set by MetaStore *@*.*.*.* ])
at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.
at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.
at
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.
at $Proxy9.verifySchema(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient

异常是因为在启动hive命令时会检查hive源数据中有一张VERSION表，如果元数据版本信息获取不到（原因可能是元数据库异常||网络异常||短期内作业量较多操作都会造成查询不到版本信息），这种情况下会判断hive.metastore.schema.verification属性是true还是false，为true时直接抛出MetaException，为false时打出warn警告然后插入一条version数据（这种情况下会造成多条version记录后面的作业会受影响），下面为hive-metastore包中ObjectStore类代码，标红处为造成多条version记录的代码；

 private synchronized void checkSchema() throws MetaException {
    // recheck if it got verified by another thread while we were waiting
    if (isSchemaVerified.get()) {
      return;
    }

    boolean strictValidation =
      HiveConf.getBoolVar(getConf(), HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION);
    // read the schema version stored in metastore db
    String schemaVer = getMetaStoreSchemaVersion();
    if (schemaVer == null) {
      if (strictValidation) {
        throw new MetaException("Version information not found in metastore. ");
      } else {
        LOG.warn("Version information not found in metastore. "
            + HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() +
            " is not enabled so recording the schema version " +
            MetaStoreSchemaInfo.getHiveSchemaVersion());
        setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(),
          "Set by MetaStore " + USER + "@" + HOSTNAME);
      }
    } else {
      // metastore schema version is different than Hive distribution needs
      if (schemaVer.equalsIgnoreCase(MetaStoreSchemaInfo.getHiveSchemaVersion())) {
        LOG.debug("Found expected HMS version of " + schemaVer);
      } else {
        if (strictValidation) {
          throw new MetaException("Hive Schema version "
              + MetaStoreSchemaInfo.getHiveSchemaVersion() +
              " does not match metastore's schema version " + schemaVer +
              " Metastore is not upgraded or corrupt");
        } else {
          LOG.error("Version information found in metastore differs " + schemaVer +
              " from expected schema version " + MetaStoreSchemaInfo.getHiveSchemaVersion() +
              ". Schema verififcation is disabled " +
              HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION + " so setting version.");
          setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(),
            "Set by MetaStore " + USER + "@" + HOSTNAME);
        }
      }
    }
    isSchemaVerified.set(true);
    return;
  }

解决方案：

hive安装好后将hive-site.xml中hive.metastore.schema.verification设置为true，version获取不到时报出异常，不去插入version信息，这样本作业执行失败不会影响下游作业；

开启metastore服务，hive统一连接metastore，由守护进程启动metastore，避免大量hive脚本初始化元数据信息时获取不到版本信息；

优化hive元数据库；

修改观察几天通过grep "Version information not found in metastore"  hive.log发现没有再报找不到version的异常了；