spark-2.4.2
kudu-1.7.0

开始尝试

1)自己手工将jar加到classpath

spark-2.4.2-bin-hadoop2.6
+
kudu-spark2_2.11-1.7.0-cdh5.16.1.jar

# bin/spark-shell
scala> val df = spark.read.options(Map("kudu.master" -> "master:7051", "kudu.table" -> "impala::test.tbl_test")).format("kudu").load
java.lang.ClassNotFoundException: Failed to find data source: kudu. Please find packages at http://spark.apache.org/third-party-projects.html
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:660)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:194)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
... 49 elided
Caused by: java.lang.ClassNotFoundException: kudu.DefaultSource
at scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:72)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$5(DataSource.scala:634)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$4(DataSource.scala:634)
at scala.util.Failure.orElse(Try.scala:224)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:634)
... 51 more

2)采用官方的方式(将kudu版本改为1.7.0)

spark-2.4.2-bin-hadoop2.6

# bin/spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.7.0

same error

3)采用官方的方式(不修改)

spark-2.4.2-bin-hadoop2.6

# bin/spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.9.0
scala> val df = spark.read.options(Map("kudu.master" -> "master:7051", "kudu.table" -> "impala::test.tbl_test")).format("kudu").load
java.lang.NoClassDefFoundError: scala/Product$class
at org.apache.kudu.spark.kudu.Upsert$.<init>(OperationType.scala:41)
at org.apache.kudu.spark.kudu.Upsert$.<clinit>(OperationType.scala)
at org.apache.kudu.spark.kudu.DefaultSource$$anonfun$getOperationType$2.apply(DefaultSource.scala:217)
at org.apache.kudu.spark.kudu.DefaultSource$$anonfun$getOperationType$2.apply(DefaultSource.scala:217)
at scala.Option.getOrElse(Option.scala:138)
at org.apache.kudu.spark.kudu.DefaultSource.getOperationType(DefaultSource.scala:217)
at org.apache.kudu.spark.kudu.DefaultSource.createRelation(DefaultSource.scala:104)
at org.apache.kudu.spark.kudu.DefaultSource.createRelation(DefaultSource.scala:87)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
... 49 elided
Caused by: java.lang.ClassNotFoundException: scala.Product$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 61 more

看起来是scala版本冲突,到spark下载页面发现一句话:

Note that, Spark is pre-built with Scala 2.11 except version 2.4.2, which is pre-built with Scala 2.12.

4)kudu-spark改为scala2.12

spark-2.4.2-bin-hadoop2.6

# bin/spark-shell --packages org.apache.kudu:kudu-spark2_2.12:1.9.0

        ::::::::::::::::::::::::::::::::::::::::::::::

        ::          UNRESOLVED DEPENDENCIES         ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: org.apache.kudu#kudu-spark2_2.12;1.9.0: not found

        ::::::::::::::::::::::::::::::::::::::::::::::

好吧,下载2.4.3

5)采用官方的方式(继续)

spark-2.4.3-bin-hadoop2.6

# bin/spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.9.0
scala> val df = spark.read.options(Map("kudu.master" -> "master:7051", "kudu.table" -> "impala::test.tbl_test")).format("kudu").load
df: org.apache.spark.sql.DataFrame = [order_no: string, id: bigint ... 28 more fields]

正常了

6)采用官方的方式(将kudu版本改为1.7.0)

spark-2.4.3-bin-hadoop2.6

# bin/spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.7.0

same error

看来spark连接kudu只能采用scala2.11+kudu-spark2_2.11:1.9.0

参考:
https://kudu.apache.org/docs/developing.html
http://spark.apache.org/downloads.html

【原创】大叔经验分享(55)spark连接kudu报错的更多相关文章

  1. 【原创】大叔经验分享(53)kudu报错unable to find SASL plugin: PLAIN

    kudu安装后运行不正常,master中找不到任何tserver,查看tserver日志发现有很多报错: Failed to heartbeat to master:7051: Invalid arg ...

  2. 【原创】大叔经验分享(51)docker报错Exited (137)

    docker container启动失败,报错:Exited (137) *** ago,比如 Exited (137) 16 seconds ago 这时通过docker logs查不到任何日志,从 ...

  3. 【原创】大叔经验分享(63)kudu vs parquet

    一 对比 存储空间对比: 查询性能对比: 二 设计方案 将数据拆分为:历史数据(hdfs+parquet+snappy)+ 近期数据(kudu),可以兼具各种优点: 1)整体低于10%的磁盘占用: 2 ...

  4. 【原创】大叔经验分享(61)kudu rebalance报错

    kudu rebalance命令报错 terminate called after throwing an instance of 'std::regex_error' what(): regex_e ...

  5. 【原创】大叔经验分享(62)kudu副本数量

    kudu的副本数量是在表上设置,可以通过命令查看 # sudo -u kudu kudu cluster ksck $master ... Summary by table Name | RF | S ...

  6. 【原创】大叔经验分享(59)kudu查看table size

    kudu并没有命令可以直接查看每个table占用的空间,可以从cloudera manager上间接查看 CM is scrapping and aggregating the /metrics pa ...

  7. 【原创】大叔经验分享(58)kudu写入压力大时报错

    kudu写入压力大时报错 19/05/18 16:53:12 INFO AsyncKuduClient: Invalidating location fd52e4f930bc45458a8f29ed1 ...

  8. 【原创】大叔经验分享(38)beeline连接hiveserver2报错impersonate

    beeline连接hiveserver2报错 Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost: ...

  9. 【原创】大叔问题定位分享(33)beeline连接presto报错

    hive2.3.4 presto0.215 使用hive2.3.4的beeline连接presto报错 $ beeline -d com.facebook.presto.jdbc.PrestoDriv ...

随机推荐

  1. MySQL datetime 和 timestamp 的区别

    [转载]:MySQL中有关TIMESTAMP和DATETIME的总结 1. datetime 和 timestamp 的相同点 两者都可以用来表示YYYY-MM-DD HH:MM:SS[.fracti ...

  2. 开源运维自动化平台-opendevops

    开源运维自动化平台-opendevops 简介 官网 | Github|  在线体验 CODO是一款为用户提供企业多混合云.自动化运维.完全开源的云管理平台. CODO前端基于Vue iview开发. ...

  3. list元素按照日期排序

    private static void ListSort(List<AppClassInfoVo> list) { Collections.sort(list, new Comparato ...

  4. [eclipse]如何修改Eclipse编辑器的字体

    步骤如下, 菜单->"Window"->“Preference”->“General”->“Appearance”->“Colors & Fo ...

  5. vmalloc详解

    vmalloc是一个接口函数, 内核代码使用它来分配在虚拟内存中连续但在物理内存中不一定连续的内存. 只需要一个参数,以字节为单位. 使用vmalloc的最著名的实例是内核对模块的实现. 因为模块可能 ...

  6. where in 的参数化查询实现

    身为一名小小的程序猿,在日常开发中不可以避免的要和where in和like打交道,在大多数情况下我们传的参数不多简单做下单引号.敏感字符转义之后就直接拼进了SQL,执行查询,搞定.若有一天你不可避免 ...

  7. GitHub:Alibaba

    ylbtech-GitHub:Alibaba 1.返回顶部   2.返回顶部   3.返回顶部   4.返回顶部   5.返回顶部 1. https://github.com/alibaba 2.   ...

  8. js图片轮播效果实现代码

    首先给大家看一看js图片轮播效果,如下图 具体思路: 一.页面加载.获取整个容器.所有放数字索引的li及放图片列表的ul.定义放定时器的变量.存放当前索引的变量index 二.添加定时器,每隔2秒钟i ...

  9. 爬虫 lxml 模块

    Xpath 在 XML 文档中查找信息的语言, 同样适用于 HTML 辅助工具 Xpath Helper Chrome插件  快捷键 Ctrl + shift + x XML Quire xpath ...

  10. CentOS7.x 更换 yum 镜像

    163 镜像地址 http://mirrors.163.com/.help/centos.html 163 镜像是国内最好的yum 镜像之一,这里使用的是163的yum 镜像备份 镜像 mv /etc ...