SupportKB

Problem Description:
After upgrading to HDP 3.0, the HiveServer2 fails to start and the following error is displayed in the HiveServer2 logs:

  1. WARN [main]: metastore.RetryingMetaStoreClient (:()) -
  2. MetaStoreClient lost connection. Attempting to reconnect (1 of 24) after 5s. getCurrentNotificationEventId
  3. org.apache.thrift.TApplicationException: Internal error processing get_current_notificationEventId
  4. at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
  5. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  6. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  7. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  8. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_current_notificationEventId
  9. (ThriftHiveMetastore.java:5575) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  10. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_current_notificationEventId
  11. (ThriftHiveMetastore.java:5563) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  12. at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getCurrentNotificationEventId
  13. (HiveMetaStoreClient.java:2723) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  14. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
  15. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
  16. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
  17. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
  18. at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
  19. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  20. at com.sun.proxy.$Proxy40.getCurrentNotificationEventId(Unknown Source) ~[?:?]
  21. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
  22. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
  23. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
  24. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
  25. at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke
  26. (HiveMetaStoreClient.java:2773) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  27. at com.sun.proxy.$Proxy40.getCurrentNotificationEventId(Unknown Source) ~[?:?]
  28. at org.apache.hadoop.hive.metastore.messaging.EventUtils$MSClientNotificationFetcher.
  29. getCurrentNotificationEventId(EventUtils.java:73)
  30. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  31. at org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll.<init>
  32. (NotificationEventPoll.java:103)
  33. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  34. at org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll.initialize
  35. (NotificationEventPoll.java:59)
  36. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  37. at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:275)
  38. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  39. at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1063)
  40. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  41. at org.apache.hive.service.server.HiveServer2.access$1600(HiveServer2.java:140)
  42. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  43. at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1332)
  44. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  45. at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1176)
  46. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  47. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
  48. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  49. ~[?:1.8.0_112]
  50. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
  51. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
  52. at org.apache.hadoop.util.RunJar.run(RunJar.java:318) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
  53. at org.apache.hadoop.util.RunJar.main(RunJar.java:232) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]

In hive-metastore.log, the following message is displayed at the same timeline:

  1. ERROR [pool-6-thread-200]: metastore.HiveMetaStore (HiveMetaStore.java:get_current_notificationEventId(7617))
  2. - Not authorized to make the get_current_notificationEventId call.
  3. You can try to disable metastore.metastore.event.db.notification.api.auth
  4. org.apache.hadoop.hive.metastore.api.MetaException: User hive is not allowed to perform this API call
  5. at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.authorizeProxyPrivilege
  6. (HiveMetaStore.java:7655)
  7. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  8. at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_current_notificationEventId(HiveMetaStore.java:7615)
  9. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  10. at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) ~[?:?]
  11. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  12. ~[?:1.8.0_112]
  13. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
  14. at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
  15. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  16. at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
  17. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  18. at com.sun.proxy.$Proxy34.get_current_notificationEventId(Unknown Source) [?:?]
  19. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_current_notificationEventId.getResult
  20. (ThriftHiveMetastore.java:18364)
  21. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  22. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_current_notificationEventId.getResult
  23. (ThriftHiveMetastore.java:18349) [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  24. at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
  25. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  26. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
  27. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  28. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
  29. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  30. at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_112]
  31. at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_112]
  32. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
  33. [hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
  34. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
  35. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  36. at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
  37. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  38. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  39. [?:1.8.0_112]
  40. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  41. [?:1.8.0_112]
  42. at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

Cause:
The connection to Metastore from HiveServer2 fails after upgrading the cluster when hive.metastore.event.db.notification.api.auth property is set to true.
Solution:
This is a known issue (HIVE-19740).

Workaround In HDFS > Configs > Custom core-site.xml, add the following:

  1. hadoop.proxyuser.hive.hosts=HS2_HOST
  2. hadoop.proxyuser.hive.groups=*

Or

In Hive >  Configs > Custom hive-site.xml, add the following:

  1. hive.metastore.event.db.notification.api.auth=false

Restart the required services.

For more details about this property, see HiveReplicationv2Development

About:
This article created by Hortonworks Support (Article: 000007703) on 2018-08-06 07:32
OS: All
Type: Cluster_Administration
Version: H3.0.0

Support ID: 000007703

Error:"MetaStoreClient lost connection. Attempting to reconnect (1 of 24) after 5s. getCurrentNotificationEventId" occurs as HiveServer2 fails to start as it cannot connect to Metastore in HDP 3.0的更多相关文章

  1. WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

    [2017-05-19 13:32:14,933] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient) ...

  2. WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) java.net.ConnectException: Connection refused

    1.启动kafka的脚本程序报如下所示的错误: [hadoop@slaver1 script_hadoop]$ kafka-start.sh start kafkaServer... [-- ::,] ...

  3. hadoop mapreduce 写入hbase报错 Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

    现象:map任务构造数据正常,reduce任务,开始也正常,速度很快 ,在hbase 的管理界面,可以看到,5W以上的请求数 当reduce 执行到 70% 左右的时候,就堵住了,查看yarn的web ...

  4. Linux下启动tomcat报错，WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException:

    tomcat启动完了之后,一直不停的打印这种错误信息,看表面上,应该是zk节点下的数据是空的,连接不上服务,所以一直在尝试连接,然后一直又连不上: 完整的错误信息: 407662 [usf-ZooKe ...

  5. R语言保存文件 Error in save error writing to connection

    Error in save(filtered, file = paste(sampleName, "filtered", sep = "_")) :   err ...

  6. urllib2.URLError: <urlopen error [Errno 104] Connection reset by peer>

    http://www.dianping.com/shop/8010173 File "综合商场1.py", line 152, in <module>    httpC ...

  7. STM8学习 无法仿真原因Starting debug session... -> Emulator reset (usb://usb)... ** Connection error (usb://usb): swim error [30200]: st-link connection error

    刚调试程序时,STlink总是连不上,一直提示: Starting debug session...-> Emulator reset (usb://usb)...** Connection e ...

  8. mysqldump导出报错"mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table `file_storage` at row: 29"

    今天mysql备份的crontab自动运行的时候,出现了报警,报警内容如下 mysqldump: Error 2013: Lost connection to MySQL server during ...

  9. Putty出现 Network error:Software caused connection abort

    使用centos7.5 用Putty连接使用没多久就会出现 Network error:Software caused connection abort #修改sshd配置文件.修改3项配置即可 vi ...

随机推荐

  1. 服务器配置tomcat部署项目

    部署项目首先你需要把你的java web项目打包成war文件 在需要打包的项目上右键>选择[Export] 选中[Web]下面的[WAR file],点击[Next] 通过[Browse]选择保 ...

  2. nexus私服搭建及maven生命周期

    一.maven找库流程 从流程上看创建nexus私服,能够优化流程,而且更加快速 二.nexus下载.安装 1.nexus下载地址 https://sonatype-download.global.s ...

  3. linux ulimit 调优

    概要:linux系统默认open files数目为1024, 有时应用程序会报Too many open files的错误,是因为open files 数目不够.这就需要修改ulimit和file-m ...

  4. 超实用的JavaScript代码段 Item5 --图片滑动效果实现

    先上图 鼠标滑过那张图,显示完整的哪张图,移除则复位: 简单的CSS加JS操作DOM实现: <!doctype html> <html> <head> <me ...

  5. text-shadow 详解及示例

    text-shadow  [tɛkst] - [ˈʃædoʊ]   定义: text-shadow: none | <shadow> [,<shadow>]* <shad ...

  6. Android 进阶 教你打造 Android 中的 IOC 框架 【ViewInject】 （上）

    转载请标明出处:http://blog.csdn.net/lmj623565791/article/details/39269193,本文出自:[张鸿洋的博客] 1.概述 首先我们来吹吹牛,什么叫Io ...

  7. Nginx 入门学习教程

    昨天听一个前同事说他们公司老大让他去研究下关于Nginx 方面的知识,我想了下Nginx 在如今的开发技术栈中应该会很大可能会用到,所以写篇博文记录总结下官网学习教程吧. 1. 什么是Nginx? 我 ...

  8. java中&和&& | 和||的区别

    我想很多人在学习java的时候,或者其他语言(如:C#,.Net等)都会遇到 &和&& 然而,如果你没有真正的理解他们的意思,这会给你思路上面带来很大的麻烦 在这篇blog中, ...

  9. bzoj3594 方伯伯的玉米田 树状数组优化dp

    f[i][j]表示到第i位,使用了j次机会的最长不下降子序列长度 转移:f[i][j]=max(f[x][y])+1; x<i; y<=j; a[x]+y<=a[i]+j; 所以根据 ...

  10. Nginx重新编译添加新模块

    找到nginx的安装包目录,如果没有的话去官网重新下载 查看ngixn版本极其编译参数 /usr/local/nginx/sbin/nginx -V 进入nginx源码目录,重新设置nginx ./c ...