spark 报错 InvalidClassException: no valid constructor

 2019-03-19 02:50:24 WARN  TaskSetManager:66 - Lost task 1.0 in stage 0.0 (TID 1, 1.2.3.4, executor 1): java.io.InvalidClassException: xxx.xxx.spark.xxx.xxx.Test; no valid constructor
         at java.io.ObjectStreamClass$ExceptionInfo.newInvalidClassException(ObjectStreamClass.java:157)
         at java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:862)
         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2041)
         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
         at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:80)
         at org.apache.spark.scheduler.Task.run(Task.scala:109)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
         at java.lang.Thread.run(Thread.java:748)

如果你执行spark的job时出现上面这个错误，同时满足以下条件

1、存在抽象父类

2、抽象父类中存在有参数的构造方法

3、子类实现了序列化接口，父类没有实现

那么恭喜你，来对地方了，解决方法非常简单，实现抽象父类的默认无参数构造方法就可以

下面代码的第9行

 public abstract class AbBaseThread extends Thread {

     protected SparkSession session;

     protected JavaSparkContext jsc;

     private SparkParam sp;

     public AbBaseThread() {}

     public AbBaseThread(SparkSession session, SparkParam sp) {
         this.session = session;
         this.jsc = new JavaSparkContext(session.sparkContext());
         this.sp = sp;
     }

     public abstract String execute(SparkParam sp);

     public void run() {
         String result = this.execute(sp);
         Jedis jedis = RedisUtil.getJedis();
         jedis.set(sp.getUuid(), result);
         RedisUtil.returnResource(jedis);
     }

 }

为什么会出现上面这个问题，我在这里强行解释一波：

1、子类序列化的时候，因为父类没有实现序列化接口，所以序列化的流中不存在父类相关的信息

2、反序列化子类对象和直接实例化一个子类对象的原理应该差不多，所以在反序列子类的时候也是先有父类对象再有子类对象，父类怎么来的，调用默认的无参构造函数来的。