SupportKB

Problem Description:
After upgrading to HDP 3.0, the HiveServer2 fails to start and the following error is displayed in the HiveServer2 logs:

  1. WARN [main]: metastore.RetryingMetaStoreClient (:()) -
  2. MetaStoreClient lost connection. Attempting to reconnect (1 of 24) after 5s. getCurrentNotificationEventId
  3. org.apache.thrift.TApplicationException: Internal error processing get_current_notificationEventId
  4. at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
  5. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  6. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  7. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  8. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_current_notificationEventId
  9. (ThriftHiveMetastore.java:5575) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  10. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_current_notificationEventId
  11. (ThriftHiveMetastore.java:5563) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  12. at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getCurrentNotificationEventId
  13. (HiveMetaStoreClient.java:2723) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  14. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
  15. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
  16. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
  17. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
  18. at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
  19. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  20. at com.sun.proxy.$Proxy40.getCurrentNotificationEventId(Unknown Source) ~[?:?]
  21. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
  22. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
  23. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
  24. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
  25. at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke
  26. (HiveMetaStoreClient.java:2773) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  27. at com.sun.proxy.$Proxy40.getCurrentNotificationEventId(Unknown Source) ~[?:?]
  28. at org.apache.hadoop.hive.metastore.messaging.EventUtils$MSClientNotificationFetcher.
  29. getCurrentNotificationEventId(EventUtils.java:73)
  30. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  31. at org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll.<init>
  32. (NotificationEventPoll.java:103)
  33. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  34. at org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll.initialize
  35. (NotificationEventPoll.java:59)
  36. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  37. at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:275)
  38. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  39. at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1063)
  40. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  41. at org.apache.hive.service.server.HiveServer2.access$1600(HiveServer2.java:140)
  42. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  43. at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1332)
  44. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  45. at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1176)
  46. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  47. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
  48. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  49. ~[?:1.8.0_112]
  50. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
  51. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
  52. at org.apache.hadoop.util.RunJar.run(RunJar.java:318) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
  53. at org.apache.hadoop.util.RunJar.main(RunJar.java:232) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]

In hive-metastore.log, the following message is displayed at the same timeline:

  1. ERROR [pool-6-thread-200]: metastore.HiveMetaStore (HiveMetaStore.java:get_current_notificationEventId(7617))
  2. - Not authorized to make the get_current_notificationEventId call.
  3. You can try to disable metastore.metastore.event.db.notification.api.auth
  4. org.apache.hadoop.hive.metastore.api.MetaException: User hive is not allowed to perform this API call
  5. at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.authorizeProxyPrivilege
  6. (HiveMetaStore.java:7655)
  7. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  8. at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_current_notificationEventId(HiveMetaStore.java:7615)
  9. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  10. at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) ~[?:?]
  11. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  12. ~[?:1.8.0_112]
  13. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
  14. at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
  15. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  16. at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
  17. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  18. at com.sun.proxy.$Proxy34.get_current_notificationEventId(Unknown Source) [?:?]
  19. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_current_notificationEventId.getResult
  20. (ThriftHiveMetastore.java:18364)
  21. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  22. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_current_notificationEventId.getResult
  23. (ThriftHiveMetastore.java:18349) [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  24. at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
  25. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  26. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
  27. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  28. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
  29. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  30. at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_112]
  31. at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_112]
  32. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
  33. [hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
  34. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
  35. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  36. at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
  37. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  38. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  39. [?:1.8.0_112]
  40. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  41. [?:1.8.0_112]
  42. at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

Cause:
The connection to Metastore from HiveServer2 fails after upgrading the cluster when hive.metastore.event.db.notification.api.auth property is set to true.
Solution:
This is a known issue (HIVE-19740).

Workaround In HDFS > Configs > Custom core-site.xml, add the following:

  1. hadoop.proxyuser.hive.hosts=HS2_HOST
  2. hadoop.proxyuser.hive.groups=*

Or

In Hive >  Configs > Custom hive-site.xml, add the following:

  1. hive.metastore.event.db.notification.api.auth=false

Restart the required services.

For more details about this property, see HiveReplicationv2Development

About:
This article created by Hortonworks Support (Article: 000007703) on 2018-08-06 07:32
OS: All
Type: Cluster_Administration
Version: H3.0.0

Support ID: 000007703

Error:"MetaStoreClient lost connection. Attempting to reconnect (1 of 24) after 5s. getCurrentNotificationEventId" occurs as HiveServer2 fails to start as it cannot connect to Metastore in HDP 3.0的更多相关文章

  1. WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

    [2017-05-19 13:32:14,933] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient) ...

  2. WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) java.net.ConnectException: Connection refused

    1.启动kafka的脚本程序报如下所示的错误: [hadoop@slaver1 script_hadoop]$ kafka-start.sh start kafkaServer... [-- ::,] ...

  3. hadoop mapreduce 写入hbase报错 Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

    现象:map任务构造数据正常,reduce任务,开始也正常,速度很快 ,在hbase 的管理界面,可以看到,5W以上的请求数 当reduce 执行到 70% 左右的时候,就堵住了,查看yarn的web ...

  4. Linux下启动tomcat报错，WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException:

    tomcat启动完了之后,一直不停的打印这种错误信息,看表面上,应该是zk节点下的数据是空的,连接不上服务,所以一直在尝试连接,然后一直又连不上: 完整的错误信息: 407662 [usf-ZooKe ...

  5. R语言保存文件 Error in save error writing to connection

    Error in save(filtered, file = paste(sampleName, "filtered", sep = "_")) :   err ...

  6. urllib2.URLError: <urlopen error [Errno 104] Connection reset by peer>

    http://www.dianping.com/shop/8010173 File "综合商场1.py", line 152, in <module>    httpC ...

  7. STM8学习 无法仿真原因Starting debug session... -> Emulator reset (usb://usb)... ** Connection error (usb://usb): swim error [30200]: st-link connection error

    刚调试程序时,STlink总是连不上,一直提示: Starting debug session...-> Emulator reset (usb://usb)...** Connection e ...

  8. mysqldump导出报错"mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table `file_storage` at row: 29"

    今天mysql备份的crontab自动运行的时候,出现了报警,报警内容如下 mysqldump: Error 2013: Lost connection to MySQL server during ...

  9. Putty出现 Network error:Software caused connection abort

    使用centos7.5 用Putty连接使用没多久就会出现 Network error:Software caused connection abort #修改sshd配置文件.修改3项配置即可 vi ...

随机推荐

  1. j2ee中spring的分布式事务实现及解决方案

    1 java事务类型 Java事务的类型有三种:JDBC事务.JTA(Java Transaction API)事务.容器事务. 常见的容器事务如Spring事务,容器事务主要是J2EE应用服务器提供 ...

  2. SSM-MyBatis-08：Mybatis中SqlSession的commit方法为什么会造成事物的提交

      ------------吾亦无他,唯手熟尔,谦卑若愚,好学若饥------------- 如题目所示,本小章节讨论为什么SqlSession的commit会造成事物的提交 首先先看SqlSessi ...

  3. Java 读书笔记 (十六) Java 继承

    例: 开发动物类,其中动物分别为企鹅以及老鼠,要求如下: 企鹅: 属性(姓名,id), 方法(吃,睡,自我介绍) 老鼠: 属性(姓名,id), 方法(吃,睡,自我介绍) 企鹅类: public cla ...

  4. Java 读书笔记 (十二) Java Character 类

    在实际开发过程中, 我们经常会遇到需要使用对象,而不是内置数据类型的情况. 为了解决这个问题, Java语言为内置数据类型char提供了包装类Character类. 可以使用Character的构造方 ...

  5. C++中“wchar_t* ”和“ char * ”之间的相互转换

    把char*转换为wchar_t* 用stdlib.h中的mbstowcs_s函数,可以通过下面的例子了解其用法: char *CStr = "string to convert" ...

  6. [NOI赛前训练]——专项测试3·数学

    由于并不想写T1和T2的题解……所有只有T3的题解了. T3 由于内部题就只写题解了. 好吧,我是一点都不想写…… 说一下这zz题解哪里写错了吧…… ……不想写…… 就说一个吧…… $n-\frac{ ...

  7. BZOJ_3282_Tree_LCT

    BZOJ_3282_Tree_LCT Description 给定N个点以及每个点的权值,要你处理接下来的M个操作. 操作有4种.操作从0到3编号.点从1到N编号. 0:后接两个整数(x,y),代表询 ...

  8. HTML——元素

    HTML 元素 HTML 文档由 HTML 元素定义. HTML 元素 开始标签 * 元素内容 结束标签 * <p> 这是一个段落 </p> <a href=" ...

  9. MYSQL—— Insert的几种用法！

    向表中插入数据 标题头示例图如下: 用insert插入值得方式: 1.使用如下语句进行插入值操作,要求:插入值必须与表头给出列数值一致,否则报:[Err] 1136 - Column count do ...

  10. Java调用Javascript、Python算法总结

    最近项目中经常需要将Javascript或者Python中的算法发布为服务,而发布Tomcat服务则需要在Java中调用这些算法,因此就不免要进行跨语言调用,即在Java程序中调用这些算法. 不管是调 ...