【原创】大叔问题定位分享（8）提交spark任务报错 Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException

spark 2.1.1

一 问题重现

spark-submit --master local[*] --class app.package.AppClass --jars /jarpath/zkclient-0.3.jar --driver-memory 1g app.jar

报错

Java HotSpot(TM) 64-Bit Server VM warning: Setting CompressedClassSpaceSize has no effect when compressed class pointers are not used
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file /export/Logs/hadoop/g1gc.log due to No such file or directory

18/09/14 16:24:38 336 WARN Utils66: Set SPARK_LOCAL_IP if you need to bind to another address
Exception in thread "main" java.lang.NoClassDefFoundError: org/I0Itec/zkclient/exception/ZkNoNodeException
at kafka.utils.ZKGroupDirs.consumerDir(ZkUtils.scala:727)
at kafka.utils.ZKGroupDirs.consumerGroupDir(ZkUtils.scala:728)
at kafka.utils.ZKGroupTopicDirs.consumerOffsetDir(ZkUtils.scala:733)
at app.package.AppClass.main(AppClass.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 15 more

二 问题解析

ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException

1 --jars --jars /jarpath/zkclient-0.3.jar

/jarpath/zkclient-0.3.jar通过--jars参数传递给spark应用

2 类org.I0Itec.zkclient.exception.ZkNoNodeException确实存在/jarpath/zkclient-0.3.jar中

javap -cp /jarpath/zkclient-0.3.jar org.I0Itec.zkclient.exception.ZkNoNodeException
Compiled from "ZkNoNodeException.java"
public class org.I0Itec.zkclient.exception.ZkNoNodeException extends org.I0Itec.zkclient.exception.ZkException {
public org.I0Itec.zkclient.exception.ZkNoNodeException();
public org.I0Itec.zkclient.exception.ZkNoNodeException(org.apache.zookeeper.KeeperException);
public org.I0Itec.zkclient.exception.ZkNoNodeException(java.lang.String, org.apache.zookeeper.KeeperException);
public org.I0Itec.zkclient.exception.ZkNoNodeException(java.lang.String);
}

3 --verbose参数输出

Main class:
app.pacakge.AppClass
Arguments:

System properties:
spark.executor.logs.rolling.maxSize -> 1073741824
spark.driver.memory -> 1g
spark.driver.extraLibraryPath -> /export/App/hadoop-2.6.1/lib/native
spark.eventLog.enabled -> true
spark.eventLog.compress -> true
spark.executor.logs.rolling.time.interval -> daily
SPARK_SUBMIT -> true
spark.app.name -> app.pacakge.AppClass
spark.driver.extraJavaOptions -> -XX:+PrintGCDetails -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-UseCompressedClassPointers -XX:CompressedClassSpaceSize=3G -XX:+PrintGCTimeStamps -Xloggc:/export/Logs/hadoop/g1gc.log
spark.jars -> file:/jarpath/zkclient-0.3.jar,file:/jarpath/app.jar
spark.sql.adaptive.enabled -> true
spark.submit.deployMode -> client
spark.executor.logs.rolling.maxRetainedFiles -> 10
spark.executor.extraClassPath -> /usr/lib/hadoop/lib/hadoop-lzo.jar:/export/App/spark/extlib/*
spark.eventLog.dir -> hdfs://hdfs_name/spark/history
spark.master -> local[*]
spark.sql.crossJoin.enabled -> true
spark.driver.extraClassPath -> /usr/lib/hadoop/lib/hadoop-lzo.jar:/export/App/spark/extlib/*
Classpath elements:
file:/jarpath/app.jar
file:/jarpath/zkclient-0.3.jar

可见/jarpath/zkclient-0.3.jar确实存在classpath中

综上3点，为什么还会报ClassNotFoundException，这里需要两个背景知识：

一个是spark submit过程，详见 https://www.cnblogs.com/barneywill/p/9820684.html

一个是java的classloader，详见 https://www.cnblogs.com/barneywill/p/10111633.html

三 问题回放过程

1 spark-submit
2 spark-class
3 org.apache.spark.launcher.Main
4 org.apache.spark.deploy.SparkSubmit
4.1 初始化MutableURLClassLoader，将--jars和应用jar作为classpath添加到MutableURLClassLoader中（注意此时zkclient-0.3.jar在这里），然后通过MutableURLClassLoader来加载应用class，也就是AppClass，然后反射调用main
4.2 AppClass.main中间接调用kafka.utils.ZKGroupDirs（kafka jar位于spark.driver.extraClassPath中），ZKGroupDirs由MutableURLClassLoader的父classloader，也就是AppClassLoader来加载，kafka.utils.ZKGroupDirs中依赖org.I0Itec.zkclient.exception.ZkNoNodeException，也由AppClassLoader来加载，并不是由MutableURLClassLoader来加载，所以报加载不到，虽然看起来zkclient-0.3.jar确实在classpath中；

问题的解决需要将zkclient-0.3.jar添加到AppClassLoader的classpath中（即spark.driver.extraClassPath）