首次尝试SeaTunnel同步Doris至Hive？这些坑你不能不避
笔者使用SeaTunnel 2.3.2版本将Doris数据同步到Hive(cdh-6.3.2)首次运行时有如下报错，并附上报错的解决方案：
- java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
- java.lang.NoClassDefFoundError: org/apache/thrift/TBase
- java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
- java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
- java.lang.OutOfMemoryError: Java heap space
目录：
java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException
1.1 解决办法
java.lang.NoClassDefFoundError: org/apache/thrift/TBase
2.1 解决办法
java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
3.1 解决办法
java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
4.1 解决办法
java.lang.OutOfMemoryError: Java heap space
5.1 解决办法
1、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.MetaException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 11 more
1.1 解决办法
原因是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可：
hive-metastore-2.1.1-cdh6.3.2.jar # cdh版本的记得将这个也复制上
hive-metastore.jar
2、java.lang.NoClassDefFoundError: org/apache/thrift/TBase
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/TBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2.1 解决办法
是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可
libthrift-0.9.3-1.jar
3、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:48)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
3.1 解决办法
是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可
hive-common-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-common.jar
hive-exec-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-exec.jar
4、java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
Exception in thread "main" java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 25 more
4.1 解决办法
是缺少对应的包，这个我的Hive里面的libthrift-0.9.3-1.jar包里面没有，所以报了这个错，去下面这个地址直接下载即可
下载地址：https://repo1.maven.org/maven2/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar
5、java.lang.OutOfMemoryError: Java heap space
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:379)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4129)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4115)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:563)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:303)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:219)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
5.1 解决办法
去SeaTunnel的config里面增加对应值，截图如下：
将值修改大一点就可以了
原文链接：https://blog.csdn.net/qq_43224174/article/details/131430223
本文由 白鲸开源 提供发布支持！
首次尝试SeaTunnel同步Doris至Hive？这些坑你不能不避的更多相关文章
- Mongodb同步数据到hive（二）
Mongodb同步数据到hive(二) 1. 概述 上一篇文章主要介绍了mongodb-based,通过直连mongodb的方式进行数据映射来进行数据查询,但是那种方式会对线上的 ...
- 【Luogu P1714】切蛋糕（面向对象编程首次尝试？）
Luogu P1714 题目的大意就是给定一个长度为n的序列,求出这个序列中长度不超过m的子串的最大和 很容易想出的一个解法就是枚举起点终点,直接暴力扫一遍得出答案. 当然也很容易发现这种做法肯定会T ...
- Hive 填坑指南
Hive 填坑指南 目录 Hive 填坑指南 数据表备份 数据表备份 方法1:create table 表名_new as select * from 原表 create table 表名_new a ...
- Spiking-YOLO : 前沿性研究，脉冲神经网络在目标检测的首次尝试 | AAAI 2020
论文提出Spiking-YOLO,是脉冲神经网络在目标检测领域的首次成功尝试,实现了与卷积神经网络相当的性能,而能源消耗极低.论文内容新颖,比较前沿,推荐给大家阅读 来源:晓飞的算法工程笔记 公众 ...
- 臣妾做不到啊---记团队内首次尝试HOLACRACY管理会议
"易穷则变,变则通.通则久". --<系辞传>下篇的第二章 实践敏捷方法接近两年,瓶颈效应逐渐显现.计穷力竭之时.有幸了解到HOLACRACY.购买了市面上唯一一本中文 ...
- REmap首次尝试--PC端
在REchart项目中,实验此网址的内容,得到若干实例 针对此我针对北京市做了若干实验 第一:绘制北京市的填充地图 代码如下: >BJqu<-mapNames("beijing& ...
- python首次尝试——hello world
百忙之中抽空学习python,但是令人惊讶的是连hello world都写不出来.
- 首次尝试LINUX下的ssh命令：登录和退出
1:我现在本机安装了centos虚拟机,然后在windows桌面下使用SecureCRT ssh客户端登录我的本地虚拟机,再然后 通过centos下的ssh命令登录局域网内测试机192.168.0.1 ...
- 首次尝试Flink的一些感受
最近打算研究研究 Flink,根据官方文档写个 Hello,World.入门还是比较容易的,不需要复杂的安装环境.配置.这篇文章简单介绍 Flink 的使用感受以及入门. 感受 搭建环境方便:Flin ...
- Evolution of Image Classifiers，进化算法在神经网络结构搜索的首次尝试 | ICML 2017
论文提出使用进化算法来进行神经网络结构搜索,整体搜索逻辑十分简单,结合权重继承,搜索速度很快,从实验结果来看,搜索的网络准确率挺不错的.由于论文是个比较早期的想法,所以可以有很大的改进空间,后面的很大 ...
随机推荐
- 支付宝spi接口设计验签和返回结果加签注意点,支付宝使用JSONObject对象
支付宝spi接口设计验签和返回结果加签注意点,支付宝使用JSONObject对象 SPI 三方服务接入指南https://opendocs.alipay.com/isv/spiforisv 服务端实现 ...
- Linux 提权-Capabilities
本文通过 Google 翻译 Capabilities – Linux Privilege Escalation - Juggernaut-Sec 这篇文章所产生,本人仅是对机器翻译中部分表达别扭的字 ...
- 15-BFC
01 BFC官方解释 https://www.w3.org/TR/CSS2/visuren.html#normal-flow 大致意思如下 02 什么情况下会形成BFC MDN解释 https://d ...
- 云服务器通过内网穿透的方式ssh访问内网服务器
云服务器通过内网穿透的方式ssh访问内网服务器 背景 买了一台云服务器,了解到可以通过外部服务器连接到公司内部服务器. 为了加快办公的效率,配置了一下. 以Ubuntu为例. 原文(有删改):http ...
- hypernetwork在SD中是怎么工作的
大家在stable diffusion webUI中可能看到过hypernetwork这个词,那么hypernetwork到底是做什么用的呢? 简单点说,hypernetwork模型是用于修改样式的小 ...
- 实测14us，Linux-RT实时性能及开发案例分享—基于全志T507-H国产平台
本文带来的是基于全志T507-H(硬件平台:创龙科技TLT507-EVM评估板),Linux-RT内核的硬件GPIO输入和输出实时性测试及应用开发案例的分享.本次演示的开发环境如下: Windows开 ...
- Java定时任务实现优惠码
在Java中实现定时任务来发放优惠码,我们可以使用多种方法,比如使用java.util.Timer类.ScheduledExecutorService接口,或者更高级的框架如Spring的@Sched ...
- Node.js 的ORM(Sequelize) 的使用
Sequelize是一个Node.js 的ORM.什么是ORM呢?对象关系映射(Object Relational Mapping).什么意思?就是在编程语言中,很容易创建对象,如果在面向对象的语言中 ...
- Java常用JDK类库和第三方类库
以下是收集的一些有用的第三方库,Java开发人员可以在其应用程序中使用它们来完成许多有用的任务.为了使用这些库,Java开发人员也应该熟悉这些类库. jdk自带的常用类库 java.lang包 jav ...
- Mac VMware Fusion 11.5 虚拟机带密钥
虚拟机 链接: https://pan.baidu.com/s/19V20p5ZV-1U5lFNHrotLgw 密码: rdun CentOS 6.5 链接: https://pan.baidu.co ...