IDEA Spark程序报错处理

错误一：

// :: ERROR Executor: Exception in task 0.0 in stage 0.0 (TID )

java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

// :: ERROR TaskSetManager: Task  in stage 0.0 failed  times; aborting job

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task  in stage 0.0 failed  times, most recent failure: Lost task 0.0 in stage 0.0 (TID , localhost, executor driver): java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

Driver stacktrace:

    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)

    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:)

    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:)

    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)

    at scala.Option.foreach(Option.scala:)

    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)

    at org.apache.spark.util.EventLoop$$anon$.run(EventLoop.scala:)

    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:)

    at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:)

    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$head$.apply(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$head$.apply(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$.apply(Dataset.scala:)

    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:)

    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:)

    at org.apache.spark.sql.Dataset.head(Dataset.scala:)

    at org.apache.spark.sql.Dataset.take(Dataset.scala:)

    at org.apache.spark.sql.Dataset.showString(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at RDD_To_DataFrame$.main(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame.main(RDD_To_DataFrame.scala)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

    at java.lang.reflect.Method.invoke(Method.java:)

    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:)

Caused by: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

错误处理：将IDEA中的Scala 改为2.10.4版本

这个问题主要出现在 Spark程序使用 case class 类时

错误二：

Error:(, ) No TypeTag available for (Array[String],)

        val documentDF= spark.createDataFrame(Seq(

错误处理：将IDEA中的Scala 改为2.12.3版本

这个问题主要出现在 Spark程序使用 Seq时：

比如：

val df= spark.createDataFrame(Seq(

         (,Array("soyo","spark","soyo2","soyo","")),

         (,Array("soyo","hadoop","soyo","hadoop","xiaozhou","soyo2","spark","","")),

         (,Array("soyo","spark","soyo2","hadoop","soyo3","")),

         (,Array("soyo","spark","soyo20","hadoop","soyo2","","")),

         (,Array("soyo","","spark","","spark","spark",""))

       )).toDF("id","words")

IDEA Spark程序报错处理的更多相关文章

解决spark程序报错：Caused by: java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]
报错信息: 09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO - at org.apache.spark.sql.catalyst.erro ...
eclispe集成Scalas环境后，导入外部Spark包报错：object apache is not a member of package org
在Eclipse中集成scala环境后,发现导入的Spark包报错,提示是:object apache is not a member of package org,网上说了一大推,其实问题很简单: ...
运行编译后的程序报错 error while loading shared libraries: lib*.so: cannot open shared object file: No such file or directory
运行编译后的程序报错 error while loading shared libraries: lib*.so: cannot open shared object file: No such f ...
Window7中Eclipse运行MapReduce程序报错的问题
按照文档:http://www.micmiu.com/bigdata/hadoop/hadoop2x-eclipse-mapreduce-demo/安装配置好Eclipse后,运行WordCount程 ...
eclipse运行hadoop程序报错:Connection refused: no further information
eclipse运行hadoop程序报错:Connection refused: no further information log4j:WARN No appenders could be foun ...
WinDbg抓取程序报错dump文件的方法
程序崩溃的两种主要现象: a. 程序在运行中的时候,突然弹出错误窗口,然后点错误窗口的确定时,程序直接关闭例如: “应用程序错误” “C++错误之类的窗口” “程序无响应” “假死”等此种崩溃特点 ...
记录微信小程序报错 Unexpected end of JSON input;at pages/flow/checkout page getOrderData function
微信小程序报错 Unexpected end of JSON input;at pages/flow/checkout page getOrderData function 这个报错是在将数组对象通过 ...
小程序-报错 xxx is not defined （已解决）
小程序-报错 xxx is not defined (已解决) 问题情境: 这样一段代码,微信的小程序报错 is not defined 我 wxml 想这样调用 //wxml 代码 <view ...
debug运行java程序报错
debug运行java程序报错 ERROR: transport error 202: connect failed: Connection timed out ERROR: JDWP Transpo ...

随机推荐

微服务网关从零搭建——（三）Ocelot网关 + identity4
增加验证服务 1.创建名为AuthService 的core 空项目 2.修改startup文件 using System; using System.Collections.Generic; usi ...
Getting start with dbus in systemd (01) - Interface, method, path
Getting start with dbus in systemd (01) 基本概念几个概念 dbus name: connetion: 如下,第一行,看到的就是 "dbus name ...
07Html、CSS
07Html.CSS-2018/07/17 1.HTML是用来描述网页的一种标记语言,是一套标记标签.HTML用使用标记标签来描述网页.超文本标记语言. 2.格式 <html> < ...
服务器做ssh免秘钥登陆
集群内服务器做非root用户免秘钥登陆:1.node1新建用户abc1,制作公钥.私钥(一路回车键即可)ssh-keygen –t rsa将自动在/home/abc1/.ssh/目录下创建公私钥文件如 ...
FileOutputStream将从一个文件中读取的内容写到另一个文件中
package com.janson.day2018082 import java.io.FileInputStream; import java.io.FileNotFoundException; ...
Ubuntu下安装oh-my-zsh
本文参考 https://www.jianshu.com/p/9a5c4cb0452d 此文已在ubuntu下确实安装成功,只不过懒得截图了,可以参照上述地址,我在他原基础上进行了一些更改. 安装Oh ...
勇者斗恶龙 UVA 11292
Once upon a time, in the Kingdom of Loowater, a minor nuisance turned into a major problem. The shor ...
【Codeforces 1129A】Toy Train
[链接] 我是链接,点我呀:) [题意] 火车从1,2,3...n->1的方式绕圈走.(即每次从i走到i+1) 有一些点有货物需要装载,但是每个点只能装上去一个货物. 每个货物都有目标点卸货点( ...
【Codeforces 1091D】New Year and the Permutation Concatenation
[链接] 我是链接,点我呀:) [题意] 把1~n的n!种排列依次连接成一个长度为nn!的序列. 让你在这个序列当中找长度为n的连续段,使得连续段中的数字的和为n(n-1)/2 输出符合要求的连续段的 ...
sheepdog简介
1.corosync,single ring最多支持50个节点:zookeeper,500个节点可稳定支撑,1000-1500个节点挑战比较大,需要优化消息传递机制. 2.sheepdog一开始为分布 ...

IDEA Spark程序报错处理

IDEA Spark程序报错处理的更多相关文章

随机推荐

热门专题