首次尝试SeaTunnel同步Doris至Hive？这些坑你不能不避
笔者使用SeaTunnel 2.3.2版本将Doris数据同步到Hive(cdh-6.3.2)首次运行时有如下报错，并附上报错的解决方案：
- java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
- java.lang.NoClassDefFoundError: org/apache/thrift/TBase
- java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
- java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
- java.lang.OutOfMemoryError: Java heap space
目录：
java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException
1.1 解决办法
java.lang.NoClassDefFoundError: org/apache/thrift/TBase
2.1 解决办法
java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
3.1 解决办法
java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
4.1 解决办法
java.lang.OutOfMemoryError: Java heap space
5.1 解决办法
1、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.MetaException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 11 more
1.1 解决办法
原因是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可：
hive-metastore-2.1.1-cdh6.3.2.jar # cdh版本的记得将这个也复制上
hive-metastore.jar
2、java.lang.NoClassDefFoundError: org/apache/thrift/TBase
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/TBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2.1 解决办法
是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可
libthrift-0.9.3-1.jar
3、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:48)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
3.1 解决办法
是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可
hive-common-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-common.jar
hive-exec-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-exec.jar
4、java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
Exception in thread "main" java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 25 more
4.1 解决办法
是缺少对应的包，这个我的Hive里面的libthrift-0.9.3-1.jar包里面没有，所以报了这个错，去下面这个地址直接下载即可
下载地址：https://repo1.maven.org/maven2/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar
5、java.lang.OutOfMemoryError: Java heap space
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:379)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4129)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4115)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:563)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:303)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:219)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
5.1 解决办法
去SeaTunnel的config里面增加对应值，截图如下：
将值修改大一点就可以了
原文链接：https://blog.csdn.net/qq_43224174/article/details/131430223
本文由 白鲸开源 提供发布支持！
首次尝试SeaTunnel同步Doris至Hive？这些坑你不能不避的更多相关文章
- Mongodb同步数据到hive（二）
Mongodb同步数据到hive(二) 1. 概述 上一篇文章主要介绍了mongodb-based,通过直连mongodb的方式进行数据映射来进行数据查询,但是那种方式会对线上的 ...
- 【Luogu P1714】切蛋糕（面向对象编程首次尝试？）
Luogu P1714 题目的大意就是给定一个长度为n的序列,求出这个序列中长度不超过m的子串的最大和 很容易想出的一个解法就是枚举起点终点,直接暴力扫一遍得出答案. 当然也很容易发现这种做法肯定会T ...
- Hive 填坑指南
Hive 填坑指南 目录 Hive 填坑指南 数据表备份 数据表备份 方法1:create table 表名_new as select * from 原表 create table 表名_new a ...
- Spiking-YOLO : 前沿性研究，脉冲神经网络在目标检测的首次尝试 | AAAI 2020
论文提出Spiking-YOLO,是脉冲神经网络在目标检测领域的首次成功尝试,实现了与卷积神经网络相当的性能,而能源消耗极低.论文内容新颖,比较前沿,推荐给大家阅读 来源:晓飞的算法工程笔记 公众 ...
- 臣妾做不到啊---记团队内首次尝试HOLACRACY管理会议
"易穷则变,变则通.通则久". --<系辞传>下篇的第二章 实践敏捷方法接近两年,瓶颈效应逐渐显现.计穷力竭之时.有幸了解到HOLACRACY.购买了市面上唯一一本中文 ...
- REmap首次尝试--PC端
在REchart项目中,实验此网址的内容,得到若干实例 针对此我针对北京市做了若干实验 第一:绘制北京市的填充地图 代码如下: >BJqu<-mapNames("beijing& ...
- python首次尝试——hello world
百忙之中抽空学习python,但是令人惊讶的是连hello world都写不出来.
- 首次尝试LINUX下的ssh命令：登录和退出
1:我现在本机安装了centos虚拟机,然后在windows桌面下使用SecureCRT ssh客户端登录我的本地虚拟机,再然后 通过centos下的ssh命令登录局域网内测试机192.168.0.1 ...
- 首次尝试Flink的一些感受
最近打算研究研究 Flink,根据官方文档写个 Hello,World.入门还是比较容易的,不需要复杂的安装环境.配置.这篇文章简单介绍 Flink 的使用感受以及入门. 感受 搭建环境方便:Flin ...
- Evolution of Image Classifiers，进化算法在神经网络结构搜索的首次尝试 | ICML 2017
论文提出使用进化算法来进行神经网络结构搜索,整体搜索逻辑十分简单,结合权重继承,搜索速度很快,从实验结果来看,搜索的网络准确率挺不错的.由于论文是个比较早期的想法,所以可以有很大的改进空间,后面的很大 ...
随机推荐
- Java邮件发送解决ssl及邮件中文乱码
Java邮件发送解决ssl及邮件中文乱码 org.apache.commons.mail.EmailException: Sending the email to the following serv ...
- IDEA生成类和方法注释模板详细说明 绝对好用
吐槽 今天心血来潮,将使用了很久的IDEA旗舰版卸载了,想换社区版用一段时间,毕竟社区版开源免费.精简不卡顿,如果够用的话以后就省去了破解的烦恼,而且可以紧跟官网使用最新版 旧的IDEA配置忘记保存了 ...
- 内网服务器通过单台外网服务器实现外网访问，iptables NAT
环境: servera: 外网服务器 serverb: 内网服务器 servera内网网关(GATEWAY)要设置为外网IP,其IP地址作为其它内网服务器的网关 servera 内网网卡配置 ...
- bugly进阶01-集成bugly时的相关参数
bugly进阶01-集成bugly时的相关参数 个人github CSDN博客 前言 bugly的集成十分的简单,在代码中只需要简单的一个语句就可以轻松集成: - (BOOL)application: ...
- Django部署在CENTOS7上
项目结构 /data/playback_project/├── PlayBack└── script /data/playback_project/PlayBack├── app01├── db.sq ...
- 高通mm-camera平台 Camera bring up基本调试思路
原文:https://www.cnblogs.com/thjfk/p/4086001.html 确定硬件 1.首先对照原理图,检查camera module的pin脚连接是否正确. 2.用示波器量Ca ...
- Xilinx SDK 开发Linux APP
Xilinx SDK 开发Linux APP 步骤 配置环境变量 将工具链需要的程序的所在目录添加到 系统环境变量中,例如: D:\Xilinx_201803\SDK\2018.3\gnu\micro ...
- ReST，以及RESTful的 简单介绍
什么是 ReST 阮一峰说的比较清楚,具体见他的博客文章. 二.名称 ReST这个词,是[Roy Thomas Fielding](http://en.wikipedia.org/wiki/Roy_F ...
- Asp .Net Core 系列：基于 Castle DynamicProxy + Autofac 实践 AOP 以及实现事务、用户填充功能
目录 什么是 AOP ? .Net Core 中 有哪些 AOP 框架? 基于 Castle DynamicProxy 实现 AOP IOC中使用 Castle DynamicProxy 实现事务管理 ...
- Linux 文件夹和文件操作【Linux 常用命令系列一】
〇.前言 本文首先介绍了 Linux 中文件的结构,将全部文件夹罗列并介绍了大概的用途: 然后通过实例介绍了文件夹相关的常用操作,仅供参考. 一.Linux 系统的文件结构 列一下系统全部文件夹: / ...