错误一：

// :: ERROR Executor: Exception in task 0.0 in stage 0.0 (TID )

java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

// :: ERROR TaskSetManager: Task  in stage 0.0 failed  times; aborting job

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task  in stage 0.0 failed  times, most recent failure: Lost task 0.0 in stage 0.0 (TID , localhost, executor driver): java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

Driver stacktrace:

    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)

    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:)

    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:)

    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)

    at scala.Option.foreach(Option.scala:)

    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)

    at org.apache.spark.util.EventLoop$$anon$.run(EventLoop.scala:)

    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:)

    at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:)

    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$head$.apply(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$head$.apply(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$.apply(Dataset.scala:)

    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:)

    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:)

    at org.apache.spark.sql.Dataset.head(Dataset.scala:)

    at org.apache.spark.sql.Dataset.take(Dataset.scala:)

    at org.apache.spark.sql.Dataset.showString(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at RDD_To_DataFrame$.main(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame.main(RDD_To_DataFrame.scala)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

    at java.lang.reflect.Method.invoke(Method.java:)

    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:)

Caused by: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

错误处理：将IDEA中的Scala 改为2.10.4版本

这个问题主要出现在 Spark程序使用 case class 类时

错误二：

Error:(, ) No TypeTag available for (Array[String],)

        val documentDF= spark.createDataFrame(Seq(

错误处理：将IDEA中的Scala 改为2.12.3版本

这个问题主要出现在 Spark程序使用 Seq时：

比如：

val df= spark.createDataFrame(Seq(

         (,Array("soyo","spark","soyo2","soyo","")),

         (,Array("soyo","hadoop","soyo","hadoop","xiaozhou","soyo2","spark","","")),

         (,Array("soyo","spark","soyo2","hadoop","soyo3","")),

         (,Array("soyo","spark","soyo20","hadoop","soyo2","","")),

         (,Array("soyo","","spark","","spark","spark",""))

       )).toDF("id","words")

IDEA Spark程序报错处理的更多相关文章

  1. 解决spark程序报错：Caused by: java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]

    报错信息: 09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO - at org.apache.spark.sql.catalyst.erro ...

  2. eclispe集成Scalas环境后，导入外部Spark包报错：object apache is not a member of package org

    在Eclipse中集成scala环境后,发现导入的Spark包报错,提示是:object apache is not a member of package org,网上说了一大推,其实问题很简单: ...

  3. 运行编译后的程序报错 error while loading shared libraries: lib*.so: cannot open shared object file: No such file or directory

    运行编译后的程序报错  error while loading shared libraries: lib*.so: cannot open shared object file: No such f ...

  4. Window7中Eclipse运行MapReduce程序报错的问题

    按照文档:http://www.micmiu.com/bigdata/hadoop/hadoop2x-eclipse-mapreduce-demo/安装配置好Eclipse后,运行WordCount程 ...

  5. eclipse运行hadoop程序报错:Connection refused: no further information

    eclipse运行hadoop程序报错:Connection refused: no further information log4j:WARN No appenders could be foun ...

  6. WinDbg抓取程序报错dump文件的方法

    程序崩溃的两种主要现象: a. 程序在运行中的时候,突然弹出错误窗口,然后点错误窗口的确定时,程序直接关闭 例如: “应用程序错误” “C++错误之类的窗口” “程序无响应” “假死”等 此种崩溃特点 ...

  7. 记录微信小程序报错 Unexpected end of JSON input;at pages/flow/checkout page getOrderData function

    微信小程序报错 Unexpected end of JSON input;at pages/flow/checkout page getOrderData function 这个报错是在将数组对象通过 ...

  8. 小程序-报错 xxx is not defined （已解决）

    小程序-报错 xxx is not defined (已解决) 问题情境: 这样一段代码,微信的小程序报错 is not defined 我 wxml 想这样调用 //wxml 代码 <view ...

  9. debug运行java程序报错

    debug运行java程序报错 ERROR: transport error 202: connect failed: Connection timed out ERROR: JDWP Transpo ...

随机推荐

  1. Vue组件传值方法调用

    1.子组件改变父组件的值 <father  label="云盘快照" name="name2"> <son :props='rows'   @ ...

  2. Humidex POJ - 3299 （数学）

    题目大意 给定你三个变量中的两个输出剩下的那一个 题解 没有什么,就是把公式推出来即可,完全的数学题 代码 #include <iostream> #include <cmath&g ...

  3. 经典书籍---MySQL经典书籍下载

    以下是一些经典的MySQL书籍电子版,括号内为提取码,若需自取. 欢迎阅读纸质版,尊重作者版权 高性能MySQL_中文版 [ hre3 ] 高性能MySQL_英文版[ m2xj ] MySQL技术内幕 ...

  4. HYSBZ - 1050(旅行comf 并查集Java实现)

    HYSBZ - 1050(旅行comf Java实现) 原题地址 解法:枚举每一条边,对于这条边,我们需要找到集合中和其值相差最小的最大边,这个集合是指与包括i边在内的ST联通集.对于这一要求,我们只 ...

  5. 九度oj 题目1049：字符串去特定字符

    题目1049:字符串去特定字符 时间限制:1 秒 内存限制:32 兆 特殊判题:否 提交:10173 解决:4611 题目描述: 输入字符串s和字符c,要求去掉s中所有的c字符,并输出结果. 输入: ...

  6. [luoguP1403] [AHOI2005]约数研究（这。。。）

    传送门 用类似筛法的原理,就好啦 ——代码 #include <cstdio> int n, ans; int a[1000001]; int main() { int i, j; sca ...

  7. 将Jquery EasyUI中DataGird的数据导入Excel中

    1.第一步获取前台DataGrid中的数据 var rows = $('#tb).datagrid("getRows");            if (rows.length = ...

  8. N的倍数

    题目来源: Ural 1302 基准时间限制:1 秒 空间限制:131072 KB 分值: 40 难度:4级算法题 一个长度为N的数组A,从A中选出若干个数,使得这些数的和是N的倍数. 例如:N = ...

  9. my first emacs custom key binding

    (defun comment-this-level ()  (interactive)  (move-beginning-of-line 1)  (set-mark-command nil)  (fo ...

  10. 线程间的通信----wait/notify机制

    wait/notify机制 实现多个线程之间的通信可以使用wait.notify.notifyAll三个方法.这三个方法都是Object类的方法.wait():导致当前线程等待,直到另一个线程调用此对 ...