【原创】大叔问题定位分享(8)提交spark任务报错 Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
spark 2.1.1
一 问题重现
spark-submit --master local[*] --class app.package.AppClass --jars /jarpath/zkclient-0.3.jar --driver-memory 1g app.jar
报错
Java HotSpot(TM) 64-Bit Server VM warning: Setting CompressedClassSpaceSize has no effect when compressed class pointers are not used
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file /export/Logs/hadoop/g1gc.log due to No such file or directory18/09/14 16:24:38 336 WARN Utils66: Set SPARK_LOCAL_IP if you need to bind to another address
Exception in thread "main" java.lang.NoClassDefFoundError: org/I0Itec/zkclient/exception/ZkNoNodeException
at kafka.utils.ZKGroupDirs.consumerDir(ZkUtils.scala:727)
at kafka.utils.ZKGroupDirs.consumerGroupDir(ZkUtils.scala:728)
at kafka.utils.ZKGroupTopicDirs.consumerOffsetDir(ZkUtils.scala:733)
at app.package.AppClass.main(AppClass.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 15 more
二 问题解析
ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
1 --jars --jars /jarpath/zkclient-0.3.jar
/jarpath/zkclient-0.3.jar通过--jars参数传递给spark应用
2 类org.I0Itec.zkclient.exception.ZkNoNodeException确实存在/jarpath/zkclient-0.3.jar中
javap -cp /jarpath/zkclient-0.3.jar org.I0Itec.zkclient.exception.ZkNoNodeException
Compiled from "ZkNoNodeException.java"
public class org.I0Itec.zkclient.exception.ZkNoNodeException extends org.I0Itec.zkclient.exception.ZkException {
public org.I0Itec.zkclient.exception.ZkNoNodeException();
public org.I0Itec.zkclient.exception.ZkNoNodeException(org.apache.zookeeper.KeeperException);
public org.I0Itec.zkclient.exception.ZkNoNodeException(java.lang.String, org.apache.zookeeper.KeeperException);
public org.I0Itec.zkclient.exception.ZkNoNodeException(java.lang.String);
}
3 --verbose参数输出
Main class:
app.pacakge.AppClass
Arguments:System properties:
spark.executor.logs.rolling.maxSize -> 1073741824
spark.driver.memory -> 1g
spark.driver.extraLibraryPath -> /export/App/hadoop-2.6.1/lib/native
spark.eventLog.enabled -> true
spark.eventLog.compress -> true
spark.executor.logs.rolling.time.interval -> daily
SPARK_SUBMIT -> true
spark.app.name -> app.pacakge.AppClass
spark.driver.extraJavaOptions -> -XX:+PrintGCDetails -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-UseCompressedClassPointers -XX:CompressedClassSpaceSize=3G -XX:+PrintGCTimeStamps -Xloggc:/export/Logs/hadoop/g1gc.log
spark.jars -> file:/jarpath/zkclient-0.3.jar,file:/jarpath/app.jar
spark.sql.adaptive.enabled -> true
spark.submit.deployMode -> client
spark.executor.logs.rolling.maxRetainedFiles -> 10
spark.executor.extraClassPath -> /usr/lib/hadoop/lib/hadoop-lzo.jar:/export/App/spark/extlib/*
spark.eventLog.dir -> hdfs://hdfs_name/spark/history
spark.master -> local[*]
spark.sql.crossJoin.enabled -> true
spark.driver.extraClassPath -> /usr/lib/hadoop/lib/hadoop-lzo.jar:/export/App/spark/extlib/*
Classpath elements:
file:/jarpath/app.jar
file:/jarpath/zkclient-0.3.jar
可见/jarpath/zkclient-0.3.jar确实存在classpath中
综上3点,为什么还会报ClassNotFoundException,这里需要两个背景知识:
一个是spark submit过程,详见 https://www.cnblogs.com/barneywill/p/9820684.html
一个是java的classloader,详见 https://www.cnblogs.com/barneywill/p/10111633.html
三 问题回放过程
1 spark-submit
2 spark-class
3 org.apache.spark.launcher.Main
4 org.apache.spark.deploy.SparkSubmit
4.1 初始化MutableURLClassLoader,将--jars和应用jar作为classpath添加到MutableURLClassLoader中(注意此时zkclient-0.3.jar在这里),然后通过MutableURLClassLoader来加载应用class,也就是AppClass,然后反射调用main
4.2 AppClass.main中间接调用kafka.utils.ZKGroupDirs(kafka jar位于spark.driver.extraClassPath中),ZKGroupDirs由MutableURLClassLoader的父classloader,也就是AppClassLoader来加载,kafka.utils.ZKGroupDirs中依赖org.I0Itec.zkclient.exception.ZkNoNodeException,也由AppClassLoader来加载,并不是由MutableURLClassLoader来加载,所以报加载不到,虽然看起来zkclient-0.3.jar确实在classpath中;
问题的解决需要将zkclient-0.3.jar添加到AppClassLoader的classpath中(即spark.driver.extraClassPath)
【原创】大叔问题定位分享(8)提交spark任务报错 Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException的更多相关文章
- maven项目运行tomcat7-maven-plugin:run时出现Caused by: java.lang.ClassNotFoundException: org.codehaus.plexus.util.Scanner(xjl456852原创)
使用tomcat7-maven-plugin插件运行web项目时, 出现下面错误: [WARNING] Error injecting: org.sonatype.plexus.build.incre ...
- 【原创】大叔问题定位分享(16)spark写数据到hive外部表报错ClassCastException: org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to org.apache.hadoop.hive.ql.io.HiveOutputFormat
spark 2.1.1 spark在写数据到hive外部表(底层数据在hbase中)时会报错 Caused by: java.lang.ClassCastException: org.apache.h ...
- 【原创】大叔问题定位分享(17)spark查orc格式数据偶尔报错NullPointerException
spark查orc格式的数据有时会报这个错 Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.orc. ...
- WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, worker1): java.lang.ClassNotFoundException: com.spark.firstApp.HelloSpark$$anonfun$2
进行如下设置,解决报错信息. val conf = new SparkConf().setAppName("helloSpark").setMaster("spark:/ ...
- 【原创】大叔问题定位分享(10)提交spark任务偶尔报错 org.apache.spark.SparkException: A master URL must be set in your configuration
spark 2.1.1 一 问题重现 问题代码示例 object MethodPositionTest { val sparkConf = new SparkConf().setAppName(&qu ...
- 【原创】大叔问题定位分享(9)oozie提交spark任务报 java.lang.NoClassDefFoundError: org/apache/kafka/clients/producer/KafkaProducer
oozie中支持很多的action类型,比如spark.hive,对应的标签为: <spark xmlns="uri:oozie:spark-action:0.1"> ...
- 【原创】大叔问题定位分享(2)spark任务一定几率报错java.lang.NoSuchFieldError: HIVE_MOVE_FILES_THREAD_COUNT
最近用yarn cluster方式提交spark任务时,有时会报错,报错几率是40%,报错如下: 18/03/15 21:50:36 116 ERROR ApplicationMaster91: Us ...
- 【原创】大叔问题定位分享(15)spark写parquet数据报错ParquetEncodingException: empty fields are illegal, the field should be ommited completely instead
spark 2.1.1 spark里执行sql报错 insert overwrite table test_parquet_table select * from dummy 报错如下: org.ap ...
- 【svn】在提交文件是报错:previous operation has not finished;run 'cleanup' if it was interrupted
1.svn在提交文件是报错:previous operation has not finished;run 'cleanup' if it was interrupted2.原因,工作队列被占用,只需 ...
随机推荐
- Do You Kown Asp.Net Core -- Asp.Net Core 2.0 未来web开发新趋势 Razor Page
Razor Page介绍 前言 上周期待已久的Asp.Net Core 2.0提前发布了,一下子Net圈热闹了起来,2.0带来了很多新的特性和新的功能,其中Razor Page引起我的关注,作为web ...
- auth mysql
DROP TABLE IF EXISTS tky_auth_role;CREATE TABLE tky_auth_role ( roleid MEDIUMINT (8) UNSIGNED NOT NU ...
- Educational Codeforces Round 62
A. Detective Book 代码: #include <bits/stdc++.h> using namespace std; ; int N; int a[maxn]; ; in ...
- 2015年旧闻 CNNIC发布伪造CA证书
谷歌称CNNIC发布伪造CA证书 2015-3-24 15:6:17 | 作者: 月光 | 分类: 业界动态 | 评论: 64 | 浏览: 6755 根据谷歌官方安全博客报道和Mozilla官方博 ...
- How to Build a Chat Bot Using Azure Bot Service and Train It with LUIS
Introduction If you haven’t had much programming experience before, building a conversational bot an ...
- 不同系统下的字长------typedef的意义
int的字节长度是由CPU和操作系统编译器共同决定的, 一般情况下,主要是由操作系统决定,比如,你在64位AMD的机器上安装的是32位操作系统,那么,int默认是32位的:如果是64位操作系统,64位 ...
- 【BZOJ5503】[GXOI/GZOI2019]宝牌一大堆(动态规划)
[BZOJ5503][GXOI/GZOI2019]宝牌一大堆(动态规划) 题面 BZOJ 洛谷 题解 首先特殊牌型直接特判. 然后剩下的部分可以直接\(dp\),直接把所有可以存的全部带进去大力\(d ...
- react native环境搭建与生命周期
1.搭建开发环境 英文文档:http://facebook.github.io/react-native/docs/getting-started.html 中文文档:https://reactnat ...
- 深入理解PHP的运行模式
PHP运行模式有4钟:1)cgi 通用网关接口(Common Gateway Interface))2) fast-cgi 常驻 (long-live) 型的 CGI3) cli 命令行运行 ( ...
- 使用Spring Boot Actuator将指标导出到InfluxDB和Prometheus
使用Spring Boot Actuator将指标导出到InfluxDB和Prometheus Spring Boot Actuator是Spring Boot 2发布后修改最多的项目之一.它经过 ...