用户使用的sql: select count( distinct patient_id ) from argus.table_aa000612_641cd8ce_ceff_4ea0_9b27_0a3a743f0fe3;

下面做不同的测试：

1.beeline -u jdbc:hive2://0.0.0.0:10000 -e "select count( distinct patient_id ) from argus.table_aa000612_641cd8ce_ceff_4ea0_9b27_0a3a743f0fe3;"

报错：

INFO  : Starting Job = job_1494385775332_0822, Tracking URL = http://hadoop1.tj2.yiducloud.cn:8088/proxy/application_1494385775332_0822/

INFO  : Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1494385775332_0822

ERROR : Ended Job = job_1494385775332_0822 with exception 'java.io.IOException(Job status not available )'

java.io.IOException: Job status not available

        at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:)

        at org.apache.hadoop.mapreduce.Job.getJobState(Job.java:)

        at org.apache.hadoop.mapred.JobClient$NetworkedJob.getJobState(JobClient.java:)

        at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:)

        at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:)

        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:)

        at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:)

        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:)

        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:)

        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:)

        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:)

        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:)

        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:)

        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:)

        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:)

        at org.apache.hive.service.cli.operation.SQLOperation.access$(SQLOperation.java:)

        at org.apache.hive.service.cli.operation.SQLOperation$$.run(SQLOperation.java:)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

        at org.apache.hive.service.cli.operation.SQLOperation$.run(SQLOperation.java:)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:)

        at java.util.concurrent.FutureTask.run(FutureTask.java:)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

        at java.lang.Thread.run(Thread.java:)

Error: Error while processing statement: FAILED: Execution Error, return code  from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=)

yarn上报错：

Diagnostics: 
Staging dir does not exist /mr-history/anonymous/.staging

2. beeline -u jdbc:hive2://0.0.0.0:10000 -n rd -e "select count( distinct patient_id ) from argus.table_aa000612_641cd8ce_ceff_4ea0_9b27_0a3a743f0fe3;"

报错栈：

INFO  : Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1494385775332_0823

INFO  : Hadoop job information for Stage-: number of mappers: ; number of reducers:

INFO  : -- ::, Stage- map = %,  reduce = %

ERROR : Ended Job = job_1494385775332_0823 with exception 'java.io.IOException(java.io.IOException: Unknown Job job_1494385775332_0823

        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:)

        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:)

        at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:)

        at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$.callBlockingMethod(MRClientProtocol.java:)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)

        at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

        at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)

)'

java.io.IOException: java.io.IOException: Unknown Job job_1494385775332_0823

        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:)

        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:)

        at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:)

        at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$.callBlockingMethod(MRClientProtocol.java:)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)

        at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

        at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)

        at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:)

        at org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:)

        at org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:)

        at org.apache.hadoop.mapreduce.Job$.run(Job.java:)

        at org.apache.hadoop.mapreduce.Job$.run(Job.java:)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

        at org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:)

        at org.apache.hadoop.mapred.JobClient$NetworkedJob.getTaskCompletionEvents(JobClient.java:)

        at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.computeReducerTimeStatsPerJob(HadoopJobExecHelper.java:)

        at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:)

        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:)

        at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:)

        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:)

        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:)

        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:)

        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:)

        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:)

        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:)

        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:)

        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:)

        at org.apache.hive.service.cli.operation.SQLOperation.access$(SQLOperation.java:)

        at org.apache.hive.service.cli.operation.SQLOperation$$.run(SQLOperation.java:)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

        at org.apache.hive.service.cli.operation.SQLOperation$.run(SQLOperation.java:)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:)

        at java.util.concurrent.FutureTask.run(FutureTask.java:)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

        at java.lang.Thread.run(Thread.java:)

Caused by: java.io.IOException: Unknown Job job_1494385775332_0823

        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:)

        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:)

        at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:)

        at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$.callBlockingMethod(MRClientProtocol.java:)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)

        at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

        at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:)

        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:)

        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:)

        at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.unwrapAndThrowException(MRClientProtocolPBClientImpl.java:)

        at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBClientImpl.java:)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

        at java.lang.reflect.Method.invoke(Method.java:)

        at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:)

        ...  more

Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unknown Job job_1494385775332_0823

        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:)

        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:)

        at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:)

        at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$.callBlockingMethod(MRClientProtocol.java:)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)

        at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

        at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)

        at org.apache.hadoop.ipc.Client.call(Client.java:)

        at org.apache.hadoop.ipc.Client.call(Client.java:)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:)

        at com.sun.proxy.$Proxy47.getTaskAttemptCompletionEvents(Unknown Source)

        at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBClientImpl.java:)

        ...  more

Error: Error while processing statement: FAILED: Execution Error, return code  from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=)

yarn上报错：

Diagnostics: 
Job init failed : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://tijmu/mr-history/rd/.staging/job_1494385775332_0823/job.splitmetainfo    

3. beeline -u jdbc:hive2://0.0.0.0:10000 -n work -e "xxx"： 报错和1一样

4. hive -e "select count( distinct patient_id ) from argus.table_aa000612_641cd8ce_ceff_4ea0_9b27_0a3a743f0fe3;" 正常结束

5. 怀疑是权限调试时改了代谢病的hdfs配置

6. 解决方案：根据第二步Diagnostics的报错信息搜索到一篇文章：https://community.hortonworks.com/questions/17489/job-init-fail-job-splitmetainfo-file-does-not-exis.html

 

对比了hive和hs2中此属性的值，果然不一样，在hs2中采用hive中的值后(set yarn.app.mapreduce.am.staging-dir=/user/history;)hql执行正常。

另一个集群的HS2上此属性值也一样是/mr-history, hdfs同样存在目录/mr-history/rd/.staging，权限也一样，但执行就没问题，因此估计还是和第5点有关。
抽查了以前的集群，值都为/user。问题关键是需要两个地方配置成一样。

解决hiveserver2报错：java.io.IOException: Job status not available - Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask的更多相关文章

  1. 执行Hive sql 报FAILED:Execution Error，return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

    在hive Beeline命令行使用insert into ... select ...向hive表插入数据时,报FAILED:Execution Error,return code 2 from o ...

  2. java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

    执行Hive查询: Console是这样报错的 java.sql.SQLException: Error from org.apache.hadoop.hive.ql.exec.mr.MapRedTa ...

  3. Hive创建表格报【Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException】引发的血案

    在成功启动Hive之后感慨这次终于没有出现Bug了,满怀信心地打了长长的创建表格的命令,结果现实再一次给了我一棒,报了以下的错误Error, return code 1 from org.apache ...

  4. Hive的Shell里hive> 执行操作时，出现FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask错误的解决办法（图文详解）

    不多说,直接上干货! 这个问题,得非 你的hive和hbase是不是同样都是CDH版本,还是一个是apache版本,一个是CDH版本. 问题详情 [kfk@bigdata-pro01 apache-h ...

  5. hive报错 Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections,

    学习hive 使用mysql作为元数据  hive创建数据库和切换数据库都是可以的 但是创建表就是出问题 百度之后发现 是编码问题 特别记录一下~~~ 1.报错前如图: 2.在mysql数据库中执行如 ...

  6. hive 字符集问题 报错 Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections,

    学习hive 使用mysql作为元数据  hive创建数据库和切换数据库都是可以的 但是创建表就是出问题 百度之后发现 是编码问题 特别记录一下~~~ 1.报错前如图: 2.在mysql数据库中执行如 ...

  7. hive_异常_01_（未解决）FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.hadoop.hbase.HTableDescriptor.addFamily(Lorg/apache/hadoop/hbase/HColumnDescriptor;)V

    一.如果出现如下错误需要编译源码 需要重新编译Hbase-handler源码 步骤如下: 准备Jar包: 将Hbase 中lib下的jar包和Hive中lib下的jar包全部导入到一起. 记得删除里面 ...

  8. 解决kylin报错 ClassCastException org.apache.hadoop.hive.ql.exec.ConditionalTask cannot be cast to org.apache.hadoop.hive.ql.exec.mr.MapRedTask

    方法:去掉参数SET hive.auto.convert.join=true; 从配置文件$KYLIN_HOME/conf/kylin_hive_conf.xml删掉 或 kylin-gui的cube ...

  9. Spark报错java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

    Spark 读取 JSON 文件时运行报错 java.io.IOException: Could not locate executable null\bin\winutils.exe in the ...

随机推荐

  1. HBase（十）HBase性能调优总结

    一. HBase的通用优化 1 高可用 在 HBase 中 Hmaster 负责监控 RegionServer 的生命周期,均衡 RegionServer 的负载,如果 Hmaster 挂掉了,那么整 ...

  2. 【POJ】1286.Necklace of Beads

    题解 群论,我们只要找出所有的置换群的所有循环节 具体可参照算法艺术与信息学竞赛 旋转的置换有n个,每一个的循环节个数是gcd(N,i),i的范围是0到N - 1 翻转,对于奇数来说固定一个点,然后剩 ...

  3. LoadRunner 参数化之 连接数据库进行参数化

    LoadRunner 参数化之 连接数据库进行参数化 Loadrunner(简称“LR”)对性能测试的脚本进行参数化时,由于数据量偏大,大家往往都会把数据录入到数据库表里,然后关联到LR,本文将详细介 ...

  4. CSUOJ 1018 Avatar

    Description In the planet Pandora, Jake found an old encryption algorithm. The plaintext, key and ci ...

  5. Java工具类之浮点精确计算

    public class Arith { // 默认除法运算精度 private static final int DEF_DIV_SCALE = 10; // 构造器私有,让这个类不能实例化 pri ...

  6. iOS 9应用开发教程之使用代码添加按钮美化按钮

    iOS 9应用开发教程之使用代码添加按钮美化按钮 丰富的用户界面 在iOS9中提供了很多的控件以及视图来丰富用户界面,对于这些视图以及控件我们在上一章中做了简单的介绍.本章我们将详细讲解这些视图. i ...

  7. Revit二次开发示例：CancelSave

    在Revit程序中注册文件操作事件,保存新建或打开文件的信息.当保存时,如果当前文件内容和之前的一致时,则弹出对话框提示并取消保存.对话框中有一个功能链接,点击可打开插件所在目录. #region N ...

  8. codevs 1462 素数和

    1462 素数和  时间限制: 1 s  空间限制: 64000 KB  题目等级 : 青铜 Bronze     题目描述 Description 给定2个整数a,b 求出它们之间(不含a,b)所有 ...

  9. UVA1378 A funny stone game

    博弈论. 就是有一堆石子你拿走一堆中的一个,然后再向后面两堆中加两个问胜负 i<j<=k 所以我们可以直接通过sg函数计算,考虑问题的奇偶性,如果这一位是奇的我们才考虑,偶的可以模仿 然后 ...

  10. Educational Codeforces Round 45 (Div 2) (A~G)

    目录 Codeforces 990 A.Commentary Boxes B.Micro-World C.Bracket Sequences Concatenation Problem D.Graph ...