解决kylin sync table报错：MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found
在kylin-gui中sync表default.customer_visit时报错：
-- ::, ERROR [http-bio--exec-] controller.BasicController: :
java.lang.RuntimeException: cannot get HiveTableMeta
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:)
at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:)
at org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$543fa169.reloadHiveTable(<generated>)
at org.apache.kylin.rest.controller.TableController.loadHiveTable(TableController.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:)
at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:)
如果此表已被别的project引入，在另一个project中sync时报错为：
java.lang.RuntimeException: Failed to fetch metadata of 'DEFAULT.CUSTOMER_VISIT': java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not fou
nd
at org.apache.kylin.source.hive.SchemaChecker$CheckResult.raiseExceptionWhenInvalid(SchemaChecker.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:)
at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:)
下面是此表的定义，ClassNotFoundException中的类就是此表用到的JsonSerDe:
CREATE EXTERNAL TABLE `customer_visit`(
`age_m` int COMMENT 'from deserializer',
`discharge_date` date COMMENT 'from deserializer',
`address_code` string COMMENT 'from deserializer',
`admission_date` date COMMENT 'from deserializer',
`birth_date` date COMMENT 'from deserializer',
`outcome` string COMMENT 'from deserializer',
`age` int COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hive.hcatalog.data.JsonSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'hdfs://xxx/user/data/abc.db/customer_visit'
TBLPROPERTIES (
'COLUMN_STATS_ACCURATE'='false',
'numFiles'='',
'numRows'='-1',
'rawDataSize'='-1',
'totalSize'='',
'transient_lastDdlTime'='')
解决方案：
相关jar需要放到hive metastore服务的$HIVE_HOME/lib/。从第一个报错的Caused by部分可以看到报错是从org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore抛出来的，提示了是metastore的报错。
2. 另外kylin服务器上的$HIVE_HOME/lib/也需要放置此jar，否则执行第一步报错，这是hive cli需要用的：
FAILED: RuntimeException MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found)
at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:92)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.createFlatHiveTable(CreateFlatHiveTableStep.java:90)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(CreateFlatHiveTableStep.java:121)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
执行build第一步时报如下错，需要配置hive-env.sh 中的HIVE_AUX_JARS_PATH，把用到的jar包路径配置到其中，这个是给MR用的，HIVE_AUX_JARS_PATH的说明是：Folder containing extra ibraries required for hive compilation/execution
2017-04-27 19:34:33,040 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328)
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109)
... 22 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141)
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
... 24 more
一点收获：
1. kylin使用hive 依赖的是本地的hive客户端
2. 鉴于第一条，调试Kylin和hive相关的东西时，只需对本地hive客户端进行修改，不需要hive，这样可以节省不少时间
3. kylin sync table是通过hive metastore服务执行的，所以需要相关表在hive metastore服务上能正常访问，是否能正常访问可以在metastore服务的机器上执行相关hql（主要是元数据相关的hql）验证。
解决kylin sync table报错：MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found的更多相关文章
- 【原创】大叔问题定位分享（8）提交spark任务报错 Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
spark 2.1.1 一 问题重现 spark-submit --master local[*] --class app.package.AppClass --jars /jarpath/zkcli ...
- javafx:JavaFX Scene Builder 2.0打开含有第三方jar包的fxml文件报错 Caused by: java.lang.ClassNotFoundException
报错如下: java.io.IOException: javafx.fxml.LoadException: /C:/User.................test.fxml at com.orac ...
- 【spring cloud】spring cloud 使用feign调用，1.fallback熔断器不起作用，2.启动报错Caused by: java.lang.ClassNotFoundException: com.netflix.hystrix.contrib.javanica.aop.aspectj.Hystri解决
示例GitHub源码地址:https://github.com/AngelSXD/springcloud 1.首先使用feign调用,需要配置熔断器 2.配置熔断器需要将熔断器注入Bean,熔断器类上 ...
- eclipse 报错问题：java.lang.ClassNotFoundException:
解决方法:https://www.cnblogs.com/whatlonelytear/articles/5921978.html
- Tomcat报错java.lang.ClassNotFoundException: 2localhost.org.apache.juli.FileHandler
Can't load log handler "1catalina.org.apache.juli.FileHandler" java.lang.ClassNotFoundExce ...
- 报错：maven:java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
运行报错: maven:java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets 找不到Charsets这个类 上网查了以后,是因为 ...
- Tomcat报java.lang.ClassNotFoundException: 1catalina.org.apache.juli.FileHandler
最近在生产环境部署Tomcat的时候,在启动的时候,在控制台报"java.lang.ClassNotFoundException: 1catalina.org.apache.juli.Fil ...
- 用JSON报的一个错误java.lang.ClassNotFoundException: org.apache.commons.lang.exception.NestableRuntimeExcep
以前在做项目的时候就曾接触过JSON的技术,但那个时候是项目经理把所有该配制的都配了,工具类也提供了,如何使用也跟我们说了,那个时候只是觉得很好用,倒没有研究过. 今天自己写了一个JSON的例子,可以 ...
- 解决dwr报错【 Error: java.lang.SecurityException: No class by name: service】
打开包含dwr的网页时后台报错: 警告: Names of known classes are: __System DwrQueryService 十二月 11, 2015 10:24:44 上午 o ...
随机推荐
- drools7 (四、FactHandle 介绍)
先看代码 Base.java package cn.xiaojf.drools7.base; import org.apache.commons.lang3.StringUtils; import o ...
- Numpy的简单用法
Numpy的简单用法 import numpy as np 一.创建ndarray对象 列表转换成ndarray: >>> a = [1,2,3,4,5] >>> ...
- TypeScript的数据类型总结
全局下载:npm i typescript -g 编译.ts文件:tsc 1.ts自动生成对应js文件 基本类型: 1.布尔,数字,字符串 ts文件 //布尔类型 var a:boolean=tru ...
- Windows 中打开 shell
按住 Shift,单机鼠标右键"在当前目录打开命令窗口"
- 【面试总结】网易2019秋招一站式面试总结（等offer中……）
岗位:运维工程师(网易杭州) 面试时间:一天 上午十一点二十,准时开启面试,初面面试官是个看起来就像是主管的人,厚实的身体,中气浑厚的声音,整齐朴素的衬衫. 简要问题摘录如下:(后续补充答案内容) 1 ...
- HDU - 1027 Ignatius and the Princess II 全排列
Ignatius and the Princess II Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 65536/32768 K ( ...
- shell if [ -d filename]
[ -a FILE ] 如果 FILE 存在则为真. [ -b FILE ] 如果 FILE 存在且是一个块特殊文件则为真. [ -c FILE ] 如果 FILE 存在且是一个字特殊文件则为真. [ ...
- codevs 1462 素数和
1462 素数和 时间限制: 1 s 空间限制: 64000 KB 题目等级 : 青铜 Bronze 题目描述 Description 给定2个整数a,b 求出它们之间(不含a,b)所有 ...
- [LOJ2541][PKUWC2018]猎人杀(容斥+分治+FFT)
https://blog.csdn.net/Maxwei_wzj/article/details/80714129 n个二项式相乘可以用分治+FFT的方法,使用空间回收可以只开log个数组. #inc ...
- Loj10164 数字游戏1
题目描述 科协里最近很流行数字游戏.某人命名了一种不降数,这种数字必须满足从左到右各位数字成小于等于的关系,如 123,446.现在大家决定玩一个游戏,指定一个整数闭区间 [a,b][a,b][a,b ...