Hive执行count函数失败

1.现象：

0: jdbc:hive2://192.168.137.12:10000> select count(*) from emp;

INFO  : Number of reduce tasks determined at compile time: 1

INFO  : In order to change the average load for a reducer (in bytes):

INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>

INFO  : In order to limit the maximum number of reducers:

INFO  :   set hive.exec.reducers.max=<number>

INFO  : In order to set a constant number of reducers:

INFO  :   set mapreduce.job.reduces=<number>

ERROR : Job Submission failed with exception 'org.apache.hadoop.security.AccessControlException(Permission denied: user=anonymous, access=EXECUTE, inode="/tmp/hadoop-yarn/staging/anonymous/.staging":bigdata:supergroup:drwx------

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1720)

        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3855)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1011)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

)'

org.apache.hadoop.security.AccessControlException: Permission denied: user=anonymous, access=EXECUTE, inode="/tmp/hadoop-yarn/staging/anonymous/.staging":bigdata:supergroup:drwx------

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1720)

        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3855)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1011)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)

        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2110)

        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)

        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)

        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)

        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)

        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)

        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:144)

        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)

        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)

        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)

        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)

        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)

        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:431)

        at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)

        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)

        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)

        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)

        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)

        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)

        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)

        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)

        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)

        at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)

        at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

        at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=anonymous, access=EXECUTE, inode="/tmp/hadoop-yarn/staging/anonymous/.staging":bigdata:supergroup:drwx------

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1720)

        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3855)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1011)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

        at org.apache.hadoop.ipc.Client.call(Client.java:1475)

        at org.apache.hadoop.ipc.Client.call(Client.java:1412)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)

        at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)

        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

        at com.sun.proxy.$Proxy19.getFileInfo(Unknown Source)

        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)

        ... 41 more

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1)

0: jdbc:hive2://192.168.137.12:10000>

2.问题分析：

根据日志：

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=anonymous, access=EXECUTE, inode="/tmp/hadoop-yarn/staging/anonymous/.staging":bigdata:supergroup:drwx------

​ 显示对于“/tmp/hadoop-yarn/staging/anonymous/.staging”目录的权限为700，匿名用户没有访问权限，那就给个权限吧！这个目录是用来存放job信息的。

3.解决方法：为“/tmp/hadoop-yarn/staging/anonymous/.staging”赋予读写权限

 hadoop fs -chmod -R 777 /tmp/

参考链接：

https://www.cnblogs.com/zqyw/p/10524103.html

Hive执行count函数失败，Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException)的更多相关文章

  1. Hive JDBC:java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate anonymous

    今天使用JDBC来操作Hive时,首先启动了hive远程服务模式:hiveserver2 &(表示后台运行),然后到eclipse中运行程序时出现错误: java.sql.SQLExcepti ...

  2. Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):

    用windows连接hadoop集群执行mapreduce任务的时候出现以下错误: org.apache.hadoop.security.AccessControlException:Permissi ...

  3. 异常-Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hdfs, access=WRITE, inode="/hbase":root:supergroup:drwxr-xr-x

    1 详细异常 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlExce ...

  4. 你遇到了吗？Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException)

    我在使用 Structured Streaming 的 ForeachWriter,写 HDFS 文件时,出现了这个异常 这个异常出现的原因是HDFS作为一个分布式文件系统,支持多线程读,但是不支持多 ...

  5. hive运行query语句时提示错误：org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException:

    hive> select product_id, track_time from trackinfo limit 5; Total MapReduce jobs = 1 Launching Jo ...

  6. Hive执行过程中出现Caused by ： java.lang.ClassNotFoundException: org.cloudera.htrace.Trace的错误解决办法（图文详解）

    不多说,直接上干货! 问题详情 如下 这个错误的意思是缺少 htrace-core-2.04.jar. 解决办法: 将$HBASE_HOME/lib下的htrace-core-2.04.jar拷贝到$ ...

  7. hive执行结果moveTask操作失败

    hive执行结果moveTask操作失败 Apache Hive 2.1.0 ,在执行"INSERT OVERWRITE TABLE ...... select "或者 " ...

  8. [PHP源码阅读]count函数

    在PHP编程中,在遍历数组的时候经常需要先计算数组的长度作为循环结束的判断条件,而在PHP里面对数组的操作是很频繁的,因此count也算是一个常用函数,下面研究一下count函数的具体实现. 我在gi ...

  9. 在MySQL的InnoDB存储引擎中count(*)函数的优化

    写这篇文章之前已经看过了很多数据库方面的优化内容,大部分都是加索引.使用事务.要什么select什么等等.然而,只是停留在阅读的层面上,很少有实践,因为没有遇到真实的项目,一切都是纸上谈兵.实践是检验 ...

随机推荐

  1. C#递归的简单实例

    using System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Threa ...

  2. 轻松应对并发问题，Newbe.Claptrap 框架中 State 和 Event 应该如何理解？

    Newbe.Claptrap 框架中 State 和 Event 应该如何理解?最近整理了一下项目的术语表.今天就谈谈什么是 Event 和 State. Newbe.Claptrap 是一个用于轻松 ...

  3. HTML5其他标签应用

    HTML5 是下一代 HTML 标准. HTML5 多媒体 音频标签 <audio src=" "></audio> 视频标签 <video src= ...

  4. iOS APP下载安装时，如果出现此时无法下载安装APP的字样时，一些解决思路

    1.在iosAPP下载安装时,如果出现此时无法下载安装APP的字样时,可能是苹果系统进行了支持更新,并需要我们确认条约.至于如何判断是否是苹果系统进行了更改,只需要我们进入开发者账号,进入我的账户(A ...

  5. vue“欺骗”ueditor，实现图片上传

    一.环境介绍 @vue/cli 4.3.1 webpack 4.43.0 ueditor1.4.3.3 jsp版 二.springboot集成ueditor,实现分布式图片上传 参考我的另一篇博客,& ...

  6. Django学习路35_视图使用方法(复制的代码) + 简单总结

    from django.shortcuts import render,redirect from django.http import HttpResponse,JsonResponse from ...

  7. Numpy改变数组的形状

    import numpy as np n = np.arange(10) # array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) # 查看数组的大小 n.size # # 将数 ...

  8. PHP mkdir() 函数

    定义和用法 mkdir() 函数创建目录. 如果成功该函数返回 TRUE,如果失败则返回 FALSE. 语法 mkdir(path,mode,recursive,context) 参数 描述 path ...

  9. PHP imagecolorclosestalpha - 取得与指定的颜色加透明度最接近的颜色的索引

    imagecolorclosestalpha — 取得与指定的颜色加透明度最接近的颜色的索引.高佣联盟 www.cgewang.com 语法 int imagecolorclosestalpha ( ...

  10. PDO::exec

    PDO::exec — 执行一条 SQL 语句,并返回受影响的行数(PHP 5 >= 5.1.0, PECL pdo >= 0.1.0) 说明 语法 int PDO::exec ( str ...