错误一：

// :: ERROR Executor: Exception in task 0.0 in stage 0.0 (TID )

java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

// :: ERROR TaskSetManager: Task  in stage 0.0 failed  times; aborting job

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task  in stage 0.0 failed  times, most recent failure: Lost task 0.0 in stage 0.0 (TID , localhost, executor driver): java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

Driver stacktrace:

    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)

    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:)

    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:)

    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)

    at scala.Option.foreach(Option.scala:)

    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)

    at org.apache.spark.util.EventLoop$$anon$.run(EventLoop.scala:)

    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:)

    at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:)

    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$head$.apply(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$head$.apply(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$.apply(Dataset.scala:)

    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:)

    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:)

    at org.apache.spark.sql.Dataset.head(Dataset.scala:)

    at org.apache.spark.sql.Dataset.take(Dataset.scala:)

    at org.apache.spark.sql.Dataset.showString(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at RDD_To_DataFrame$.main(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame.main(RDD_To_DataFrame.scala)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

    at java.lang.reflect.Method.invoke(Method.java:)

    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:)

Caused by: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

错误处理：将IDEA中的Scala 改为2.10.4版本

这个问题主要出现在 Spark程序使用 case class 类时

错误二：

Error:(, ) No TypeTag available for (Array[String],)

        val documentDF= spark.createDataFrame(Seq(

错误处理：将IDEA中的Scala 改为2.12.3版本

这个问题主要出现在 Spark程序使用 Seq时：

比如：

val df= spark.createDataFrame(Seq(

         (,Array("soyo","spark","soyo2","soyo","")),

         (,Array("soyo","hadoop","soyo","hadoop","xiaozhou","soyo2","spark","","")),

         (,Array("soyo","spark","soyo2","hadoop","soyo3","")),

         (,Array("soyo","spark","soyo20","hadoop","soyo2","","")),

         (,Array("soyo","","spark","","spark","spark",""))

       )).toDF("id","words")

IDEA Spark程序报错处理的更多相关文章

  1. 解决spark程序报错：Caused by: java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]

    报错信息: 09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO - at org.apache.spark.sql.catalyst.erro ...

  2. eclispe集成Scalas环境后，导入外部Spark包报错：object apache is not a member of package org

    在Eclipse中集成scala环境后,发现导入的Spark包报错,提示是:object apache is not a member of package org,网上说了一大推,其实问题很简单: ...

  3. 运行编译后的程序报错 error while loading shared libraries: lib*.so: cannot open shared object file: No such file or directory

    运行编译后的程序报错  error while loading shared libraries: lib*.so: cannot open shared object file: No such f ...

  4. Window7中Eclipse运行MapReduce程序报错的问题

    按照文档:http://www.micmiu.com/bigdata/hadoop/hadoop2x-eclipse-mapreduce-demo/安装配置好Eclipse后,运行WordCount程 ...

  5. eclipse运行hadoop程序报错:Connection refused: no further information

    eclipse运行hadoop程序报错:Connection refused: no further information log4j:WARN No appenders could be foun ...

  6. WinDbg抓取程序报错dump文件的方法

    程序崩溃的两种主要现象: a. 程序在运行中的时候,突然弹出错误窗口,然后点错误窗口的确定时,程序直接关闭 例如: “应用程序错误” “C++错误之类的窗口” “程序无响应” “假死”等 此种崩溃特点 ...

  7. 记录微信小程序报错 Unexpected end of JSON input;at pages/flow/checkout page getOrderData function

    微信小程序报错 Unexpected end of JSON input;at pages/flow/checkout page getOrderData function 这个报错是在将数组对象通过 ...

  8. 小程序-报错 xxx is not defined （已解决）

    小程序-报错 xxx is not defined (已解决) 问题情境: 这样一段代码,微信的小程序报错 is not defined 我 wxml 想这样调用 //wxml 代码 <view ...

  9. debug运行java程序报错

    debug运行java程序报错 ERROR: transport error 202: connect failed: Connection timed out ERROR: JDWP Transpo ...

随机推荐

  1. 2018最新Python小白入门教程，30天学会Python

    随着Python的技术的流行,Python在为人们带来工作与生活上带来了很多的便捷,因为Python简单,学起来快,也是不少新手程序员入门的首选语言.作为一名Python爱好者,我也想跟大家分享分享我 ...

  2. vue-cli的项目加入骨架屏

    在原生APP中我们经常可以看到,打开app时候,内容还没出来,app会被别的内容替代,这样很好的提升了用户体验.那么在webApp中,我们如何避免白屏的尴尬情况呢?可以通过 vue-skeleton- ...

  3. 移动端响应式rem

    (function (doc, win) { var docEl = doc.documentElement, resizeEvt = 'orientationchange' in window ? ...

  4. php利用32进制实现对id加密解密

    前言 最近在项目中遇到一个问题,当前用户分享一个邀请码给好友,好友根据邀请码注册成为新用户之后,则成为当前用户的下级,特定条件下,可以得到下级用户的一系列返利.这里要实现的就是根据当前用户的id,生成 ...

  5. 【Python实践-8】和为S的两个数字

    (剑指offer)输入一个递增排序的数组和一个数字S,在数组中查找两个数,使得他们的和正好是S,如果有多对数字的和等于S,输出两个数的乘积最小的. 思路:选定第一个数字,然后遍历后面的数字求和并与S比 ...

  6. 18年多校-1002 Balanced Sequence

    >>点击进入原题测试<< 思路:自己写没写出来,想不通该怎么排序好,看了杜神代码后补题A掉的.重新理解了一下优先队列中重载小于号的含义,这里记录一下这种排序方式. #inclu ...

  7. java 访问对象私有变量

    Captcha captcha = getCaptcha(captchaId); // 通过反射获取验证码值 Class<?> classType = captcha.getClass() ...

  8. 【03】全局 CSS 样式

    全局 CSS 样式 设置全局 CSS 样式:基本的 HTML 元素均可以通过 class 设置样式并得到增强效果:还有先进的栅格系统. 概览 深入了解 Bootstrap 底层结构的关键部分,包括我们 ...

  9. 【Codeforces 484A】Bits

    [链接] 我是链接,点我呀:) [题意] 让你求出l~r当中二进制表示1的个数最多的数x [题解] 最多有64位 我们可以从l开始一直增大到r 怎么增大? 找到l的二进制表示当中0所在的位置 假设i这 ...

  10. [luoguP1316] 丢瓶盖（二分答案）

    传送门 二分答案再判断即可 ——代码 #include <cstdio> #include <iostream> #include <algorithm> #def ...