首次尝试SeaTunnel同步Doris至Hive？这些坑你不能不避
笔者使用SeaTunnel 2.3.2版本将Doris数据同步到Hive(cdh-6.3.2)首次运行时有如下报错，并附上报错的解决方案：
- java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
- java.lang.NoClassDefFoundError: org/apache/thrift/TBase
- java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
- java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
- java.lang.OutOfMemoryError: Java heap space
目录：
java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException
1.1 解决办法
java.lang.NoClassDefFoundError: org/apache/thrift/TBase
2.1 解决办法
java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
3.1 解决办法
java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
4.1 解决办法
java.lang.OutOfMemoryError: Java heap space
5.1 解决办法
1、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.MetaException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 11 more
1.1 解决办法
原因是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可：
hive-metastore-2.1.1-cdh6.3.2.jar # cdh版本的记得将这个也复制上
hive-metastore.jar
2、java.lang.NoClassDefFoundError: org/apache/thrift/TBase
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/TBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2.1 解决办法
是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可
libthrift-0.9.3-1.jar
3、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:48)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
3.1 解决办法
是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可
hive-common-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-common.jar
hive-exec-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-exec.jar
4、java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
Exception in thread "main" java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 25 more
4.1 解决办法
是缺少对应的包，这个我的Hive里面的libthrift-0.9.3-1.jar包里面没有，所以报了这个错，去下面这个地址直接下载即可
下载地址：https://repo1.maven.org/maven2/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar
5、java.lang.OutOfMemoryError: Java heap space
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:379)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4129)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4115)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:563)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:303)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:219)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
5.1 解决办法
去SeaTunnel的config里面增加对应值，截图如下：
将值修改大一点就可以了
原文链接：https://blog.csdn.net/qq_43224174/article/details/131430223
本文由 白鲸开源 提供发布支持！
首次尝试SeaTunnel同步Doris至Hive？这些坑你不能不避的更多相关文章
- Mongodb同步数据到hive（二）
Mongodb同步数据到hive(二) 1. 概述 上一篇文章主要介绍了mongodb-based,通过直连mongodb的方式进行数据映射来进行数据查询,但是那种方式会对线上的 ...
- 【Luogu P1714】切蛋糕（面向对象编程首次尝试？）
Luogu P1714 题目的大意就是给定一个长度为n的序列,求出这个序列中长度不超过m的子串的最大和 很容易想出的一个解法就是枚举起点终点,直接暴力扫一遍得出答案. 当然也很容易发现这种做法肯定会T ...
- Hive 填坑指南
Hive 填坑指南 目录 Hive 填坑指南 数据表备份 数据表备份 方法1:create table 表名_new as select * from 原表 create table 表名_new a ...
- Spiking-YOLO : 前沿性研究，脉冲神经网络在目标检测的首次尝试 | AAAI 2020
论文提出Spiking-YOLO,是脉冲神经网络在目标检测领域的首次成功尝试,实现了与卷积神经网络相当的性能,而能源消耗极低.论文内容新颖,比较前沿,推荐给大家阅读 来源:晓飞的算法工程笔记 公众 ...
- 臣妾做不到啊---记团队内首次尝试HOLACRACY管理会议
"易穷则变,变则通.通则久". --<系辞传>下篇的第二章 实践敏捷方法接近两年,瓶颈效应逐渐显现.计穷力竭之时.有幸了解到HOLACRACY.购买了市面上唯一一本中文 ...
- REmap首次尝试--PC端
在REchart项目中,实验此网址的内容,得到若干实例 针对此我针对北京市做了若干实验 第一:绘制北京市的填充地图 代码如下: >BJqu<-mapNames("beijing& ...
- python首次尝试——hello world
百忙之中抽空学习python,但是令人惊讶的是连hello world都写不出来.
- 首次尝试LINUX下的ssh命令：登录和退出
1:我现在本机安装了centos虚拟机,然后在windows桌面下使用SecureCRT ssh客户端登录我的本地虚拟机,再然后 通过centos下的ssh命令登录局域网内测试机192.168.0.1 ...
- 首次尝试Flink的一些感受
最近打算研究研究 Flink,根据官方文档写个 Hello,World.入门还是比较容易的,不需要复杂的安装环境.配置.这篇文章简单介绍 Flink 的使用感受以及入门. 感受 搭建环境方便:Flin ...
- Evolution of Image Classifiers，进化算法在神经网络结构搜索的首次尝试 | ICML 2017
论文提出使用进化算法来进行神经网络结构搜索,整体搜索逻辑十分简单,结合权重继承,搜索速度很快,从实验结果来看,搜索的网络准确率挺不错的.由于论文是个比较早期的想法,所以可以有很大的改进空间,后面的很大 ...
随机推荐
- 一文学完所有的Hive Sql（两万字最全详解）
Hive Sql 大全 本文基本涵盖了Hive日常使用的所有SQL,因为SQL太多,所以将SQL进行了如下分类: 一.DDL语句(数据定义语句): 对数据库的操作:包含创建.修改数据库 对数据表的操作 ...
- Kubernetes(K8s)最新版搭建
Kubernetes简单介绍 Kubernetes意为舵手,简称K8s. 前身是Google的Borg.所以一开源就吸引了一大批注意力. 因为谷歌,所以墙.在国内搭建K8s非常头疼. 下面我就来介绍一 ...
- springboot使用mail提示没有该类型的bean
@Autowired private JavaMailSenderImpl javaMailSender; 自动注入时提示没有该类型的Bean. 原因 没有配置邮件发送相关的配置信息. spring: ...
- linux 下新建显示器分辨率
1. 输入cvt 1920 1080 (假设需要添加的分辨率为1920x1080), 获取Mode Line # 1920x1080 59.96 Hz (CVT 2.07M9) hsync: 67.1 ...
- 设备树DTS 学习：3-驱动开发中常用的 DTS api
背景 本章的内容是为了实现在驱动中的开发,通过调用有关的api来寻找设备树节点熟悉,从而达到使用设备树进行驱动开发的目的. 参考:Linux内核 设备树操作常用API Linux设备树语法详解一文中介 ...
- IPv6地址的文本表示规范
背景 随着IPv6越来越普及,经常要跟IPv6地址打交道,迫切需要一个统一的IPv6地址文本表示规范. RFC4291简单的说明了如何将IPv6地址表示成文本形式,但有很多有歧义和不周全的地方. RF ...
- HDFS的特点和目标，不适合场景
HDFS的特点和目标: HDFS设计优点: (一)高可靠性:Hadoop按位存储和处理数据的能力值得人们信赖; (二)高扩展性:Hadoop是在可用的计算机集簇间分配数据并完成计算任务的,这些集簇可 ...
- 【Mysql】Mysql字符集CHARACTER和COLLATE
一:设置服务器默认字符集为 utf8mb4 创建数据库时,如果没有指定字符集,会采用服务器的默认字符集.设置服务器默认字符集为 utf8mb4 可以提高便利性. 编辑 MySQL 的配置文件,修改服务 ...
- 洛谷P2864
来一发在洛谷的第一篇题解 解析 首先从原点出发回到原点会形成一个环 要计算在环上的路程,首先我们要破环 利用建墙法破环 可以选取任意一个边缘上的树,往上或者往下或者往左往右建立一堵墙'|' 这样利用b ...
- 洛谷P1747
这个题被坑麻了,题目居然不给棋盘的范围,评论区居然有人说棋盘是无限大的,我想说的是如果真是这样那么第9个点答案应该是2而不是3,这个棋盘绝对是有大小的. #include<iostream> ...