Spark-HBase集成错误之 java.lang.NoClassDefFoundError: org/htrace/Trace

在进行Spark与HBase 集成的过程中遇到以下问题：

java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:)
  at org.apache.spark.sql.SparkSession$$anonfun$sessionState$.apply(SparkSession.scala:)
  at org.apache.spark.sql.SparkSession$$anonfun$sessionState$.apply(SparkSession.scala:)
  at scala.Option.getOrElse(Option.scala:)
  at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:)
  at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$.apply(SparkSession.scala:)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$.apply(SparkSession.scala:)
  at scala.collection.mutable.HashMap$$anonfun$foreach$.apply(HashMap.scala:)
  at scala.collection.mutable.HashMap$$anonfun$foreach$.apply(HashMap.scala:)
  at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:)
  at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:)
  at scala.collection.mutable.HashMap.foreach(HashMap.scala:)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:)
  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:)
  ...  elided
Caused by: java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: org/htrace/Trace when creating Hive client using classpath:......

Please make sure that jars for your version of hive and hadoop are included in the paths passed to spark.sql.hive.metastore.jars.
  at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:)
  at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:)
  at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$.apply$mcZ$sp(HiveExternalCatalog.scala:)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$.apply(HiveExternalCatalog.scala:)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$.apply(HiveExternalCatalog.scala:)
  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:)
  at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:)
  at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:)
  at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:)
  at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:)
  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:)
  ...  more
Caused by: java.lang.reflect.InvocationTargetException: java.lang.NoClassDefFoundError: org/htrace/Trace
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:)
  at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:)
  ...  more
Caused by: java.lang.NoClassDefFoundError: org/htrace/Trace
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:)
  at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
  at java.lang.reflect.Method.invoke(Method.java:)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:)
  at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
  at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:)
  at org.apache.hadoop.hdfs.DistributedFileSystem$.doCall(DistributedFileSystem.java:)
  at org.apache.hadoop.hdfs.DistributedFileSystem$.doCall(DistributedFileSystem.java:)
  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:)
  at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:)
  at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:)
  at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:)
  at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:)
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:)
  at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:)
  ...  more
Caused by: java.lang.ClassNotFoundException: org.htrace.Trace
  at java.net.URLClassLoader.findClass(URLClassLoader.java:)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:)
  ...  more
<console>:: error: not found: value spark
       import spark.implicits._
              ^
<console>:: error: not found: value spark
       import spark.sql
              ^
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.
      /_/

Using Scala version 2.11. (Java HotSpot(TM) -Bit Server VM, Java 1.8.0_60)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

我事先已经将所需要的jar支持包全部拷贝到了各节点下的spark路径下的jar目录，但是在启动bin/spark-shell时总是报以上错误。在网上也看来很多解决方法，比如将缺失的jar包 ：hbase/lib 下 hbase*.jar,metrics*.jar.htrace*.jar  等拷贝到spark/jar目录，但这不正是我的启动准备工作吗，等于没说！

从故障根源分析，造成此错误的原因在于Spark启动的时候无法正常从我们的jar目录加载到它所需要的jar包（此处表现为/htrace-core-3.0.4.jar包）。所以，直接将包添加到jar目录是无法解决此问题的，而我们需要做的是给它指明CLASSPATH路径，在它不能找到所需要的jar包时就可以从此路径寻找。

解决办法：

在spark/conf目录下的spark-env.sh配置文件添加以下配置内容

export HBASE_CLASSPATH='hbase classpath'
export HADOOP_CLASSPATH='hadoop classpath'
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_CLASSPATH

---

以上就是博主为大家介绍的这一板块的主要内容，这都是博主自己的学习过程，希望能给大家带来一定的指导作用，有用的还望大家点个支持，如果对你没用也望包涵，有错误烦请指出。如有期待可关注博主以第一时间获取更新哦，谢谢！同时也欢迎转载，但必须在博文明显位置标注原文地址，解释权归博主所有！