执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds its limit [2048]

原因分析

shell脚本中一次提交的hql-mr作业量太大，其中包含的信息超过oozie launcher一次容许的最大值2K（2K是默认值）

解决办法

1）修改oozie-site.xml：
<property>
<name>oozie.action.max.output.data</name>
<value>204800</value>
</property>
2）CDH集群中修改 oozie-site.xml 的 Oozie Server 高级配置代码段（安全阀），如图：

参考：https://blog.csdn.net/abysscarry/article/details/82120855

异常日志

18/09/15 16:21:47 INFO CuratorFrameworkSingleton: Closing ZooKeeper client.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
18/09/15 16:21:58 INFO client.RMProxy: Connecting to ResourceManager at master.prodcdh.com/172.18.205.119:8032
18/09/15 16:21:58 INFO client.RMProxy: Connecting to ResourceManager at master.prodcdh.com/172.18.205.119:8032
Launcher AM execution failed
java.io.IOException: output.properties data exceeds its limit [2048]
at org.apache.oozie.action.hadoop.LocalFsOperations.getLocalFileContentAsString(LocalFsOperations.java:86)
at org.apache.oozie.action.hadoop.LauncherAM.processActionData(LauncherAM.java:521)
at org.apache.oozie.action.hadoop.LauncherAM.handleActionData(LauncherAM.java:501)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:229)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
Exception in thread "main" java.io.IOException: output.properties data exceeds its limit [2048]
at org.apache.oozie.action.hadoop.LocalFsOperations.getLocalFileContentAsString(LocalFsOperations.java:86)
at org.apache.oozie.action.hadoop.LauncherAM.processActionData(LauncherAM.java:521)
at org.apache.oozie.action.hadoop.LauncherAM.handleActionData(LauncherAM.java:501)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:229)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)

troubleshooting-执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds its limit [2048]的更多相关文章

  1. Oozie java.io.IOException: output.properties data exceeds its limit [2048]

    在使用oozie调用sqoop时,报了下边这个错 Launcher AM execution failed java.io.IOException: output.properties data ex ...

  2. hive使用python脚本导致java.io.IOException: Broken pipe异常退出

    反垃圾rd那边有一个hql,在执行过程中出现错误退出,报java.io.IOException: Broken pipe异常,hql中使用到了python脚本,hql和python脚本最近没有人改过, ...

  3. hive对于lzo文件处理异常Caused by: java.io.IOException: Compressed length 842086665 exceeds max block size 67108864 (probably corrupt file)

    hive查询lzo数据格式文件的表时,抛 Caused by: java.io.IOException: Compressed length 842086665 exceeds max block s ...

  4. hive报错：Failed with exception java.io.IOException: rename for src path:

    在hive中,会有这样一种情形: 1.创建一个分区外部表A(比如A表有5个字段),并且向A表里指定的分区(比如20160928这个分区)里插入数据 2.发现A表缺少一些字段,因为存在元数据不实时更新的 ...

  5. hive运行query语句时提示错误：org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException:

    hive> select product_id, track_time from trackinfo limit 5; Total MapReduce jobs = 1 Launching Jo ...

  6. 解决hiveserver2报错：java.io.IOException: Job status not available - Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

    用户使用的sql: select count( distinct patient_id ) from argus.table_aa000612_641cd8ce_ceff_4ea0_9b27_0a3a ...

  7. 解决方案--java执行cmd命令ProcessBuilder--出错Exception in thread "main" java.io.IOException: Cannot run program "dir d:\": CreateProcess error=2(xjl456852原创)

    当我尝试在java中通过ProcessBuilder运行window的cmd命令时出现错误: public static void main(String [] args) throws IOExce ...

  8. AIO 开始不定时的抛异常： java.io.IOException: 指定的网络名不再可用

    一天里会抛出几个这样的错误,但发现服务还在正常的运行. java.io.IOException: 指定的网络名不再可用. at sun.nio.ch.Iocp.translateErrorToIOEx ...

  9. ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

    sqoop从mysql导入到hive报错: 18/08/22 13:30:53 ERROR tool.ImportTool: Import failed: java.io.IOException: j ...

随机推荐

  1. 微软、谷歌、亚马逊、Facebook等硅谷大厂91个开源软件盘点（附下载地址）

    开源软件中有大量专家构建的代码,大大节省了开发人员的时间和成本,热衷于开源的大厂们总是能够带给我们新的惊喜.2016年9月GitHub报告显示,GitHub已经有超过 520 万的用户和超 30 万的 ...

  2. 【紫书】BigInteger 高精度类型 原书上有一个bug：A+B！=B+A

    存个代码 struct BigInterger { static const int BASE = 1e8; ; vector<int> s; BigInterger() { *this ...

  3. iOS耳机等音频输出设备切换的通知

    #import "ViewController.h" #import <AVFoundation/AVFoundation.h> @interface ViewCont ...

  4. Implicit declaration of function 'BMKCoordinateForMapPoint' is invalid in C99

    少一个头文件  #import <BaiduMapAPI_Utils/BMKGeometry.h> 加上这个就好了 <HPHalfScreenTopBar: 0x103570bb0; ...

  5. python 几个重要的概念

    转自:http://www.cnblogs.com/aylin/p/5601969.html

  6. Ubuntu16.04下写的Qt程序，调试时没问题，运行时偶现崩溃 （需要在运行时生成core dump文件，QMAKE_CC += -g）

    记录一下 Ubuntu16.04下写的Qt程序,调试时没问题,运行时偶现崩溃 需要在运行时生成core dump文件 首先在pro结尾里加入 QMAKE_CC += -g QMAKE_CXX += - ...

  7. redis连接池的标准用法：

    from .conf import HOST, PORT, POOL_NAME import redis redis_pool = redis.ConnectionPool(host=HOST, po ...

  8. 新版微信小程序即将上线 新增微信支付功能

    <经济参考报>消息,新版微信小程序正在测试中,有可能将在近期正式上线.新版小程序增加了“附近门店”功能的接口,微信用户可以通过定位功能,查看提供线下服务的各类门店,并直接利用小程序实现包括 ...

  9. 读取Linux上图片

    /** * 读取图片 * @param path * @param response * @throws Exception */ public static void showImg(String ...

  10. PAT 1017 Queueing at Bank[一般]

    1017 Queueing at Bank (25)(25 分)提问 Suppose a bank has K windows open for service. There is a yellow ...