解决kylin sync table报错：MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found
在kylin-gui中sync表default.customer_visit时报错：
-- ::, ERROR [http-bio--exec-] controller.BasicController: :
java.lang.RuntimeException: cannot get HiveTableMeta
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:)
at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:)
at org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$543fa169.reloadHiveTable(<generated>)
at org.apache.kylin.rest.controller.TableController.loadHiveTable(TableController.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:)
at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:)
如果此表已被别的project引入，在另一个project中sync时报错为：
java.lang.RuntimeException: Failed to fetch metadata of 'DEFAULT.CUSTOMER_VISIT': java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not fou
nd
at org.apache.kylin.source.hive.SchemaChecker$CheckResult.raiseExceptionWhenInvalid(SchemaChecker.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:)
at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:)
下面是此表的定义，ClassNotFoundException中的类就是此表用到的JsonSerDe:
CREATE EXTERNAL TABLE `customer_visit`(
`age_m` int COMMENT 'from deserializer',
`discharge_date` date COMMENT 'from deserializer',
`address_code` string COMMENT 'from deserializer',
`admission_date` date COMMENT 'from deserializer',
`birth_date` date COMMENT 'from deserializer',
`outcome` string COMMENT 'from deserializer',
`age` int COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hive.hcatalog.data.JsonSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'hdfs://xxx/user/data/abc.db/customer_visit'
TBLPROPERTIES (
'COLUMN_STATS_ACCURATE'='false',
'numFiles'='',
'numRows'='-1',
'rawDataSize'='-1',
'totalSize'='',
'transient_lastDdlTime'='')
解决方案：
相关jar需要放到hive metastore服务的$HIVE_HOME/lib/。从第一个报错的Caused by部分可以看到报错是从org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore抛出来的，提示了是metastore的报错。
2. 另外kylin服务器上的$HIVE_HOME/lib/也需要放置此jar，否则执行第一步报错，这是hive cli需要用的：
FAILED: RuntimeException MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found)
at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:92)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.createFlatHiveTable(CreateFlatHiveTableStep.java:90)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(CreateFlatHiveTableStep.java:121)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
执行build第一步时报如下错，需要配置hive-env.sh 中的HIVE_AUX_JARS_PATH，把用到的jar包路径配置到其中，这个是给MR用的，HIVE_AUX_JARS_PATH的说明是：Folder containing extra ibraries required for hive compilation/execution
2017-04-27 19:34:33,040 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328)
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109)
... 22 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141)
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
... 24 more
一点收获：
1. kylin使用hive 依赖的是本地的hive客户端
2. 鉴于第一条，调试Kylin和hive相关的东西时，只需对本地hive客户端进行修改，不需要hive，这样可以节省不少时间
3. kylin sync table是通过hive metastore服务执行的，所以需要相关表在hive metastore服务上能正常访问，是否能正常访问可以在metastore服务的机器上执行相关hql（主要是元数据相关的hql）验证。
解决kylin sync table报错：MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found的更多相关文章
- 【原创】大叔问题定位分享（8）提交spark任务报错 Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
spark 2.1.1 一 问题重现 spark-submit --master local[*] --class app.package.AppClass --jars /jarpath/zkcli ...
- javafx:JavaFX Scene Builder 2.0打开含有第三方jar包的fxml文件报错 Caused by: java.lang.ClassNotFoundException
报错如下: java.io.IOException: javafx.fxml.LoadException: /C:/User.................test.fxml at com.orac ...
- 【spring cloud】spring cloud 使用feign调用，1.fallback熔断器不起作用，2.启动报错Caused by: java.lang.ClassNotFoundException: com.netflix.hystrix.contrib.javanica.aop.aspectj.Hystri解决
示例GitHub源码地址:https://github.com/AngelSXD/springcloud 1.首先使用feign调用,需要配置熔断器 2.配置熔断器需要将熔断器注入Bean,熔断器类上 ...
- eclipse 报错问题：java.lang.ClassNotFoundException:
解决方法:https://www.cnblogs.com/whatlonelytear/articles/5921978.html
- Tomcat报错java.lang.ClassNotFoundException: 2localhost.org.apache.juli.FileHandler
Can't load log handler "1catalina.org.apache.juli.FileHandler" java.lang.ClassNotFoundExce ...
- 报错：maven:java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
运行报错: maven:java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets 找不到Charsets这个类 上网查了以后,是因为 ...
- Tomcat报java.lang.ClassNotFoundException: 1catalina.org.apache.juli.FileHandler
最近在生产环境部署Tomcat的时候,在启动的时候,在控制台报"java.lang.ClassNotFoundException: 1catalina.org.apache.juli.Fil ...
- 用JSON报的一个错误java.lang.ClassNotFoundException: org.apache.commons.lang.exception.NestableRuntimeExcep
以前在做项目的时候就曾接触过JSON的技术,但那个时候是项目经理把所有该配制的都配了,工具类也提供了,如何使用也跟我们说了,那个时候只是觉得很好用,倒没有研究过. 今天自己写了一个JSON的例子,可以 ...
- 解决dwr报错【 Error: java.lang.SecurityException: No class by name: service】
打开包含dwr的网页时后台报错: 警告: Names of known classes are: __System DwrQueryService 十二月 11, 2015 10:24:44 上午 o ...
随机推荐
- abtest分流随机链接方法（javascript）
¶¹¸¯¸ÉËêµÄ·¨¹úµçÊÓ¸²¸Ç --> 代码如下 <!DOCTYPE HTML> <html> <head> <script type=& ...
- 面试题41：和为s的两个数字 || 和为s的连续正数序列
和为s的两个数字 题目:输入一个递增排序的数组和一个数字s,在数组中查找两个数,使得它们的和正好是s.如果有多对数字的和等于s,输出任意一对即可. 有点类似于夹逼的思想 注意两个int相加的和要用lo ...
- [翻译]Gulp.js简介
我们讨论了很多关于怎么减少页面体积,提高重网站性能的方法.有些是操作是一劳永逸的,如开启服务器的gzip压缩,使用适当的图片格式,或删除一些不必要的字符.但有一些任务是每次工作都必须反复执行的.如 新 ...
- USACO 6.1 Cow XOR
Cow XORAdrian Vladu -- 2005 Farmer John is stuck with another problem while feeding his cows. All of ...
- NHibernate 错误
Unable to locate persister for the entity named 'Model.Customer'.The persister define the persistenc ...
- Data时间格式化
//时间戳转时间 function timeStamp2String(time) { var datetime = new Date(); datetime.setTime(time); var ye ...
- 本机ip和127.0.0.1的区别
简介 连接上因特网的每一台计算机都会有一个IP地址, 在linux下可以使用命令ifconfig来查看本机的ip地址(windows为ipconfig), 比如我当前电脑的ip地址为192.168.0 ...
- thinkphp3.2中开启静态缓存后对404页面的处理方法
静态缓存很实用但是有时有些不需要静态缓存,如404页面,第一次访问返回404页面并缓存,第二次换回的状态就是200,属于正常访问,虽然人眼可以看出是404页面,但是搜索引擎不会的,而是把这个页面当成正 ...
- poj-3268最短路
title: poj-3268最短路 date: 2018-10-13 15:54:34 tags: acm 刷题 categories: ACM-最短路 概述 这是一道最短路的模板题,,,不过虽然是 ...
- [转载] 你所不知道的TIME_WAIT和CLOSE_WAIT
前言 本文转载自 https://mp.weixin.qq.com/s/FrEfx_Yvv0fkLG97dMSTqw.很久前看到Vincent Bernat在博客中写了一遍关于TCP time-wai ...