解决kylin sync table报错：MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found
在kylin-gui中sync表default.customer_visit时报错：
-- ::, ERROR [http-bio--exec-] controller.BasicController: :
java.lang.RuntimeException: cannot get HiveTableMeta
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:)
at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:)
at org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$543fa169.reloadHiveTable(<generated>)
at org.apache.kylin.rest.controller.TableController.loadHiveTable(TableController.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:)
at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:)
如果此表已被别的project引入，在另一个project中sync时报错为：
java.lang.RuntimeException: Failed to fetch metadata of 'DEFAULT.CUSTOMER_VISIT': java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not fou
nd
at org.apache.kylin.source.hive.SchemaChecker$CheckResult.raiseExceptionWhenInvalid(SchemaChecker.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:)
at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:)
下面是此表的定义，ClassNotFoundException中的类就是此表用到的JsonSerDe:
CREATE EXTERNAL TABLE `customer_visit`(
`age_m` int COMMENT 'from deserializer',
`discharge_date` date COMMENT 'from deserializer',
`address_code` string COMMENT 'from deserializer',
`admission_date` date COMMENT 'from deserializer',
`birth_date` date COMMENT 'from deserializer',
`outcome` string COMMENT 'from deserializer',
`age` int COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hive.hcatalog.data.JsonSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'hdfs://xxx/user/data/abc.db/customer_visit'
TBLPROPERTIES (
'COLUMN_STATS_ACCURATE'='false',
'numFiles'='',
'numRows'='-1',
'rawDataSize'='-1',
'totalSize'='',
'transient_lastDdlTime'='')
解决方案：
相关jar需要放到hive metastore服务的$HIVE_HOME/lib/。从第一个报错的Caused by部分可以看到报错是从org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore抛出来的，提示了是metastore的报错。
2. 另外kylin服务器上的$HIVE_HOME/lib/也需要放置此jar，否则执行第一步报错，这是hive cli需要用的：
FAILED: RuntimeException MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found)
at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:92)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.createFlatHiveTable(CreateFlatHiveTableStep.java:90)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(CreateFlatHiveTableStep.java:121)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
执行build第一步时报如下错，需要配置hive-env.sh 中的HIVE_AUX_JARS_PATH，把用到的jar包路径配置到其中，这个是给MR用的，HIVE_AUX_JARS_PATH的说明是：Folder containing extra ibraries required for hive compilation/execution
2017-04-27 19:34:33,040 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328)
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109)
... 22 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141)
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
... 24 more
一点收获：
1. kylin使用hive 依赖的是本地的hive客户端
2. 鉴于第一条，调试Kylin和hive相关的东西时，只需对本地hive客户端进行修改，不需要hive，这样可以节省不少时间
3. kylin sync table是通过hive metastore服务执行的，所以需要相关表在hive metastore服务上能正常访问，是否能正常访问可以在metastore服务的机器上执行相关hql（主要是元数据相关的hql）验证。
解决kylin sync table报错：MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found的更多相关文章
- 【原创】大叔问题定位分享（8）提交spark任务报错 Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
spark 2.1.1 一 问题重现 spark-submit --master local[*] --class app.package.AppClass --jars /jarpath/zkcli ...
- javafx:JavaFX Scene Builder 2.0打开含有第三方jar包的fxml文件报错 Caused by: java.lang.ClassNotFoundException
报错如下: java.io.IOException: javafx.fxml.LoadException: /C:/User.................test.fxml at com.orac ...
- 【spring cloud】spring cloud 使用feign调用，1.fallback熔断器不起作用，2.启动报错Caused by: java.lang.ClassNotFoundException: com.netflix.hystrix.contrib.javanica.aop.aspectj.Hystri解决
示例GitHub源码地址:https://github.com/AngelSXD/springcloud 1.首先使用feign调用,需要配置熔断器 2.配置熔断器需要将熔断器注入Bean,熔断器类上 ...
- eclipse 报错问题：java.lang.ClassNotFoundException:
解决方法:https://www.cnblogs.com/whatlonelytear/articles/5921978.html
- Tomcat报错java.lang.ClassNotFoundException: 2localhost.org.apache.juli.FileHandler
Can't load log handler "1catalina.org.apache.juli.FileHandler" java.lang.ClassNotFoundExce ...
- 报错：maven:java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
运行报错: maven:java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets 找不到Charsets这个类 上网查了以后,是因为 ...
- Tomcat报java.lang.ClassNotFoundException: 1catalina.org.apache.juli.FileHandler
最近在生产环境部署Tomcat的时候,在启动的时候,在控制台报"java.lang.ClassNotFoundException: 1catalina.org.apache.juli.Fil ...
- 用JSON报的一个错误java.lang.ClassNotFoundException: org.apache.commons.lang.exception.NestableRuntimeExcep
以前在做项目的时候就曾接触过JSON的技术,但那个时候是项目经理把所有该配制的都配了,工具类也提供了,如何使用也跟我们说了,那个时候只是觉得很好用,倒没有研究过. 今天自己写了一个JSON的例子,可以 ...
- 解决dwr报错【 Error: java.lang.SecurityException: No class by name: service】
打开包含dwr的网页时后台报错: 警告: Names of known classes are: __System DwrQueryService 十二月 11, 2015 10:24:44 上午 o ...
随机推荐
- 聊一聊FE面试那些事【原创】
最近公司由于业务的扩展.技术的延伸需要招一批有能力的小伙伴加入,而我有幸担任"技术面试官"的角色前前后后面试了不下50多位候选人,如同见证了50多位前端开发者的经历一样,在面试的过 ...
- 【转载】Node.js 教程（菜鸟教程系列）
很好的一篇教程:Node.js 教程 简单做下笔记 概述 简单的说 Node.js 就是运行在服务端的 JavaScript. Node.js 是一个基于Chrome JavaScript 运行时建立 ...
- Codeforces Round #334 (Div. 1) B. Moodular Arithmetic
B - Moodular Arithmetic 题目大意:题意:告诉你p和k,其中(0<=k<=p-1),x属于{0,1,2,3,....,p-1},f函数要满足f(k*x%p)=k*f( ...
- django.db.utils.OperationalError: 1045错误
django.db.utils.OperationalError: (1045, "Access denied for user 'root'@'localhost' (using pass ...
- String 的常用操作
String 类,我可以不负责的说在 Java 中这个类应该是使用最频繁的类了.然而关于它的常用的操作,我却不甚了解.整理这个东西很反人性的,这些方法,你看到他的时候感觉很简单,但是真正用的时候还是不 ...
- JAVAEE——SSH项目实战04：联系人添加、列表显示和修改
作者: kent鹏 转载请注明出处: http://www.cnblogs.com/xieyupeng/p/7159337.html 一.联系人添加 1.添加页面设计 linkman/list. ...
- tomcat如何利用waf进行防护
近期某一实验室遇到一个问题:web环境是windows+tomcat+mysql,检测到cookie注入,此时又不想修改代码.此时两种方案进行解决: 1.利用安软(waf)类进行检测防御.这里国内主要 ...
- CentOS7配置ssh证书登录无效
今天配置A服务器使用root用户ssh免密登录服务器B,配置过程很简单,由于这两台服务器开发人员之前生成过证书,我就直接在A服务器执行如下命令即可 # ssh-copy-id -i ~/.ssh/id ...
- C#剪切板
C#剪切板 Clipboard类 我们现在先来看一下官方文档的介绍 位于:System.Windows.Forms 命名空间下 Provides methods to place data on an ...
- 1006 Sign In and Sign Out (25)（25 point(s)）
problem At the beginning of every day, the first person who signs in the computer room will unlock t ...