解决kylin sync table报错：MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found
在kylin-gui中sync表default.customer_visit时报错：
-- ::, ERROR [http-bio--exec-] controller.BasicController: :
java.lang.RuntimeException: cannot get HiveTableMeta
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:)
at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:)
at org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$543fa169.reloadHiveTable(<generated>)
at org.apache.kylin.rest.controller.TableController.loadHiveTable(TableController.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:)
at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:)
如果此表已被别的project引入，在另一个project中sync时报错为：
java.lang.RuntimeException: Failed to fetch metadata of 'DEFAULT.CUSTOMER_VISIT': java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not fou
nd
at org.apache.kylin.source.hive.SchemaChecker$CheckResult.raiseExceptionWhenInvalid(SchemaChecker.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:)
at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:)
下面是此表的定义，ClassNotFoundException中的类就是此表用到的JsonSerDe:
CREATE EXTERNAL TABLE `customer_visit`(
`age_m` int COMMENT 'from deserializer',
`discharge_date` date COMMENT 'from deserializer',
`address_code` string COMMENT 'from deserializer',
`admission_date` date COMMENT 'from deserializer',
`birth_date` date COMMENT 'from deserializer',
`outcome` string COMMENT 'from deserializer',
`age` int COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hive.hcatalog.data.JsonSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'hdfs://xxx/user/data/abc.db/customer_visit'
TBLPROPERTIES (
'COLUMN_STATS_ACCURATE'='false',
'numFiles'='',
'numRows'='-1',
'rawDataSize'='-1',
'totalSize'='',
'transient_lastDdlTime'='')
解决方案：
相关jar需要放到hive metastore服务的$HIVE_HOME/lib/。从第一个报错的Caused by部分可以看到报错是从org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore抛出来的，提示了是metastore的报错。
2. 另外kylin服务器上的$HIVE_HOME/lib/也需要放置此jar，否则执行第一步报错，这是hive cli需要用的：
FAILED: RuntimeException MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found)
at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:92)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.createFlatHiveTable(CreateFlatHiveTableStep.java:90)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(CreateFlatHiveTableStep.java:121)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
执行build第一步时报如下错，需要配置hive-env.sh 中的HIVE_AUX_JARS_PATH，把用到的jar包路径配置到其中，这个是给MR用的，HIVE_AUX_JARS_PATH的说明是：Folder containing extra ibraries required for hive compilation/execution
2017-04-27 19:34:33,040 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328)
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109)
... 22 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141)
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
... 24 more
一点收获：
1. kylin使用hive 依赖的是本地的hive客户端
2. 鉴于第一条，调试Kylin和hive相关的东西时，只需对本地hive客户端进行修改，不需要hive，这样可以节省不少时间
3. kylin sync table是通过hive metastore服务执行的，所以需要相关表在hive metastore服务上能正常访问，是否能正常访问可以在metastore服务的机器上执行相关hql（主要是元数据相关的hql）验证。
解决kylin sync table报错：MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found的更多相关文章
- 【原创】大叔问题定位分享（8）提交spark任务报错 Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
spark 2.1.1 一 问题重现 spark-submit --master local[*] --class app.package.AppClass --jars /jarpath/zkcli ...
- javafx:JavaFX Scene Builder 2.0打开含有第三方jar包的fxml文件报错 Caused by: java.lang.ClassNotFoundException
报错如下: java.io.IOException: javafx.fxml.LoadException: /C:/User.................test.fxml at com.orac ...
- 【spring cloud】spring cloud 使用feign调用，1.fallback熔断器不起作用，2.启动报错Caused by: java.lang.ClassNotFoundException: com.netflix.hystrix.contrib.javanica.aop.aspectj.Hystri解决
示例GitHub源码地址:https://github.com/AngelSXD/springcloud 1.首先使用feign调用,需要配置熔断器 2.配置熔断器需要将熔断器注入Bean,熔断器类上 ...
- eclipse 报错问题：java.lang.ClassNotFoundException:
解决方法:https://www.cnblogs.com/whatlonelytear/articles/5921978.html
- Tomcat报错java.lang.ClassNotFoundException: 2localhost.org.apache.juli.FileHandler
Can't load log handler "1catalina.org.apache.juli.FileHandler" java.lang.ClassNotFoundExce ...
- 报错：maven:java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
运行报错: maven:java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets 找不到Charsets这个类 上网查了以后,是因为 ...
- Tomcat报java.lang.ClassNotFoundException: 1catalina.org.apache.juli.FileHandler
最近在生产环境部署Tomcat的时候,在启动的时候,在控制台报"java.lang.ClassNotFoundException: 1catalina.org.apache.juli.Fil ...
- 用JSON报的一个错误java.lang.ClassNotFoundException: org.apache.commons.lang.exception.NestableRuntimeExcep
以前在做项目的时候就曾接触过JSON的技术,但那个时候是项目经理把所有该配制的都配了,工具类也提供了,如何使用也跟我们说了,那个时候只是觉得很好用,倒没有研究过. 今天自己写了一个JSON的例子,可以 ...
- 解决dwr报错【 Error: java.lang.SecurityException: No class by name: service】
打开包含dwr的网页时后台报错: 警告: Names of known classes are: __System DwrQueryService 十二月 11, 2015 10:24:44 上午 o ...
随机推荐
- 【LOJ】#2430. 「POI2014」沙拉餐厅 Salad Bar
题解 波兰人的j是苹果,p是橘子 还真是跟中国过不去啊= =写的时候很难受 我们先求出每个点作为起点,能延伸到的最大长度,这个可以处理成前缀和,查询一下区间最小值是不是小于0,用st表实现,如果区间最 ...
- 【POJ】1067.取石子游戏
题解 这道题让我对SG函数有了更深刻的理解,这是道打表找规律题 我们打出来SG函数似乎是 1 2必败 3 5必败 4 7必败 6 10必败 8 13必败 哇我找到规律了-- 然而,我显然不会通项 后来 ...
- USACO 5.3 Window Area
Window AreaIV Balkan Olympiad You've just be assigned the project of implemented a windowing interfa ...
- bzoj [SDOI2009]学校食堂Dining
感觉这个状压dp比较难想.. dp[ i ][ s ][ k ] 表示前i - 1个都排好了, 从i开始的7个的取没取的状态为s, 且最后一个相对i的位置为k的最少花费. 状态转移方程 if(s &a ...
- Bad connection to FS. command aborted. exception: Call to chaoren/192.168.80.100:9000 failed on connection exception: java.net.ConnectException: Connection refused
Bad connection to FS. command aborted. exception: Call to chaoren/192.168.80.100:9000 failed on conn ...
- python二叉树简单实现
二叉树简单实现: class Node: def __init__(self,item): self.item = item self.child1 = None self.child2 = None ...
- 【Java】 int与String类型间的相互转化
public class Test { public static void main(String[] args) { /* * int类型转String类型 */ int n1 = 9; //1. ...
- SQL_异化
select a.pk_accasoa from bd_accasoa a; --下级科目原来主键: 0001Z0100000000001A2 --执行该语句后下级科目异化了(替换的意思) , '@@ ...
- Openstack_通用模块_Oslo_vmware 创建 vCenter 虚拟机快照
创建虚拟机快照 vSphere Create Snapshot 文档 Snapshot 是虚拟机磁盘文件(VMDK)在某个点及时的复本.包含了虚拟机所有虚拟磁盘上的数据状态和这个虚拟机的电源状态(on ...
- ArrayList，Vector，LinkedList
韩梦飞沙 韩亚飞 313134555@qq.com yue31313 han_meng_fei_sha 数组列表 和 向量Vector, 都是使用数组方式存储. 向量 使用了 同步synchr ...