spark提示Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Lscala.collection.immutable.Map;

起因

编写了一个处理两列是否相等的UDF,这两列的数据结构是一样的,但是结构比较复杂,如下:

|-- list: array (nullable = true)
| |-- element: map (containsNull = true)
| | |-- key: string
| | |-- value: array (valueContainsNull = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- Date: integer (nullable = true)
| | | | |-- Name: string (nullable = true)
|-- list2: array (nullable = true)
| |-- element: map (containsNull = true)
| | |-- key: string
| | |-- value: array (valueContainsNull = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- Date: integer (nullable = true)
| | | | |-- Name: string (nullable = true)

也就是说Array里嵌着Map,Map里还嵌着一个Array,只能依次去比较,编写的UDF如下:

case class AppList(Date: Int, versionCode: Int, Name:String)

    def isMapEqual(map1: Map[String, Array[AppList]], map2:Map[String, Array[AppList]]): Boolean = {
try{
if (map1.size != map2.size){
return false
} else{
for ( x <- map1.keys){
if (map1(x) != map2(x)){
return false
}
}
return true
}
} catch {
case e: Exception => false
}
} def isListEqual(list1: Array[Map[String, Array[AppList]]], list2:Seq[Map[String, Seq[AppList]]]): Boolean = {
try {
if (list1.length != list2.length){
return false
} else if (list1.length == 0 || list2.length == 0){
return false
} else {
return isMapEqual(list1(0), list2(0))
}
} catch {
case e: Exception => false
}
} val isColumnEqual = udf((list1: Array[Map[String, Array[AppList]]], list2:Array[Map[String, Array[AppList]]]) => {
isListEqual(list1, list2)
})

然后我就贴到spark-shell里去执行了下面语句:

val dat = df.withColumn("equal", isColumnEqual($"list1", $"list2"))
dat.show()

此时就出现了如下错误:

Caused by: org.apache.spark.SparkException: Failed to execute user defined function($anonfun$1: (array<map<string,array<struct<Date:int,Name:string>>>>, array<map<string,array<struct<Date:int,Name:string>>>>) => boolean)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Lscala.collection.immutable.Map;
at $anonfun$1.apply(<console>:42)
... 16 more
解决办法

所谓的解决办法,自然是去谷歌了…

这里看到,说把Array改成Seq就好了,囧,尝试了一下,果然就好了

原因

这里说:

So it looks like the ArrayType on Dataframe "idDF" is really a WrappedArray and not an Array - So the function call to "filterMapKeysWithSet" failed as it expected an Array but got a WrappedArray/ Seq instead (which doesn't implicitly convert to Array in Scala 2.8 and above).

意思是,此Array非Scala中的原生Array,而是封装了一下的Array(有错的一定指出来,我都没写过Scala,慌

参考

spark提示Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Lscala.collection.immutable.Map;的更多相关文章

  1. hive orc压缩数据异常java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.hive.ql.io.orc.OrcSerde$OrcSerdeRow

    hive表在创建时候指定存储格式 STORED AS ORC tblproperties ('orc.compress'='SNAPPY'); 当insert数据到表时抛出异常 Caused by: ...

  2. sbt spark2.3.1 idea打包Caused by: java.lang.ClassNotFoundException: scala.Product$class

    今天同事在服务区上面装的是最新版本的hadoop3.10和spark2.3.1,因为用scala开发, 所以我想用sbt进行开发.过程中遇到各种坑,刚开始用的jdk10,结果也报错,后来改成jdk1. ...

  3. 关于android使用ksoap2报Caused by: java.lang.ClassCastException: org.ksoap2.SoapFault cannot be cast to org.ksoap2.serialization.SoapObject

    Caused by: java.lang.ClassCastException: org.ksoap2.SoapFault cannot be cast to org.ksoap2.serializa ...

  4. Caused by: java.lang.ClassCastException: org.springframework.web.SpringServletContainerInitializer cannot be cast to javax.servlet.ServletContainerInitializer错误解决办法

    严重: Failed to initialize end point associated with ProtocolHandler ["http-bio-8080"] java. ...

  5. Caused by: java.lang.ClassCastException: org.springframework.web.SpringServletContainerInitializer cannot be cast to javax.servlet.ServletContainerInitializer

    A child container failed during startjava.util.concurrent.ExecutionException: org.apache.catalina.Li ...

  6. Caused by: java.lang.ClassCastException: class java.lang.Double cannot be cast to class org.apache.hadoop.io.WritableComparable

    错误: Caused by: java.lang.ClassCastException: class java.lang.Double cannot be cast to class org.apac ...

  7. java.lang.ClassCastException: net.sf.ezmorph.bean.MorphDynaBean cannot be cast to

    Java.lang.ClassCastException: net.sf.ezmorph.bean.MorphDynaBean cannot be cast to 在使用JSONObject.toBe ...

  8. SSH整合时执行hibernate查询报错:java.lang.ClassCastException: com.ch.hibernate.Department_$$_javassist_0 cannot be cast to javassist.util.proxy

    今天在整合ssh三个框架时,有一个功能,是查询所有员工信息,且员工表和部门表是多对一的映射关系,代码能正常运行到查询得到一个List集合,但在页面展示的时候,就报异常了, java.lang.Clas ...

  9. java.lang.ClassCastException: org.springframework.web.filter.CharacterEncodingFilter cannot be cast to javax.servlet.Filter

    java.lang.ClassCastException: org.springframework.web.filter.CharacterEncodingFilter cannot be cast ...

随机推荐

  1. spring加载jar包中多个配置文件(转)

    转自:http://evan0625.iteye.com/blog/1598366 在使用spring加载jar包中的配置文件时,不支持通配符,需要一个一个引入,如下所示: Java代码 <co ...

  2. Git 获取仓库(分布式版本控制系统)

    1.在现有目录中初始化仓库 如果你打算使用 Git 来对现有的项目进行管理,你只需要进入该项目目录并输入以下命令. # 初始化仓库 $ git init 该命令将创建一个名为 .git 的子目录,这个 ...

  3. .NET压缩图片保存 .NET CORE WebApi Post跨域提交 C# Debug和release判断用法 tofixed方法 四舍五入 (function($){})(jQuery); 使用VUE+iView+.Net Core上传图片

    .NET压缩图片保存   需求: 需要将用户后买的图片批量下载打包压缩,并且分不同的文件夹(因:购买了多个用户的图片情况) 文章中用到了一个第三方的类库,Nuget下载 SharpZipLib 目前用 ...

  4. 《JAVA与模式》之迭代器模式

    定义:提供一种方法访问一个容器对象中各个元素,而又不暴露该对象的内部细节. 类型:行为类模式 类图: public static void main(String[] args) { List< ...

  5. JDBC连接SQLServer出现的异常

    数据库连接.  question1. java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver 异常 ...

  6. Win7 安装 MongoDB

    MongoDB 下载 MongoDB 提供了可用于 32 位和 64 位系统的预编译二进制包,你可以从MongoDB官网下载安装,MongoDB 预编译二进制包下载地址:https://www.mon ...

  7. jQuery -&gt; 删除/替换DOM元素

    删除 删除操作很easy,直接在结果集后链式调用remove()方法就可以. 比如.要删除下面html脚本中全部的a元素.直接通过 $('a'.remove(); 就能够做到了. <h3> ...

  8. 在C语言中除法运算为什么没有小数部分?

    原文链接: http://wenda.tianya.cn/question/4e096f010317a93d 除法运算符" / ",如果是两个整数相除结果为整数如果需要保留小数时 ...

  9. struts系列:校验(二)自定义校验器

    一.自定义校验类 public class PasswordValidator extends FieldValidatorSupport { @Override public void valida ...

  10. Cross compiling coreutils and generate the manpages

    When we cross compiling coreutils, there is an problem of generating man pages, because the source s ...