执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds its limit [2048]

原因分析

shell脚本中一次提交的hql-mr作业量太大，其中包含的信息超过oozie launcher一次容许的最大值2K（2K是默认值）

解决办法

1）修改oozie-site.xml：
<property>
<name>oozie.action.max.output.data</name>
<value>204800</value>
</property>
2）CDH集群中修改 oozie-site.xml 的 Oozie Server 高级配置代码段（安全阀），如图：

参考：https://blog.csdn.net/abysscarry/article/details/82120855

异常日志

18/09/15 16:21:47 INFO CuratorFrameworkSingleton: Closing ZooKeeper client.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
18/09/15 16:21:58 INFO client.RMProxy: Connecting to ResourceManager at master.prodcdh.com/172.18.205.119:8032
18/09/15 16:21:58 INFO client.RMProxy: Connecting to ResourceManager at master.prodcdh.com/172.18.205.119:8032
Launcher AM execution failed
java.io.IOException: output.properties data exceeds its limit [2048]
at org.apache.oozie.action.hadoop.LocalFsOperations.getLocalFileContentAsString(LocalFsOperations.java:86)
at org.apache.oozie.action.hadoop.LauncherAM.processActionData(LauncherAM.java:521)
at org.apache.oozie.action.hadoop.LauncherAM.handleActionData(LauncherAM.java:501)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:229)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
Exception in thread "main" java.io.IOException: output.properties data exceeds its limit [2048]
at org.apache.oozie.action.hadoop.LocalFsOperations.getLocalFileContentAsString(LocalFsOperations.java:86)
at org.apache.oozie.action.hadoop.LauncherAM.processActionData(LauncherAM.java:521)
at org.apache.oozie.action.hadoop.LauncherAM.handleActionData(LauncherAM.java:501)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:229)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)

troubleshooting-执行Oozie调度Hive导数脚本抛java.io.IOException: output.properties data exceeds its limit [2048]的更多相关文章

  1. Oozie java.io.IOException: output.properties data exceeds its limit [2048]

    在使用oozie调用sqoop时,报了下边这个错 Launcher AM execution failed java.io.IOException: output.properties data ex ...

  2. hive使用python脚本导致java.io.IOException: Broken pipe异常退出

    反垃圾rd那边有一个hql,在执行过程中出现错误退出,报java.io.IOException: Broken pipe异常,hql中使用到了python脚本,hql和python脚本最近没有人改过, ...

  3. hive对于lzo文件处理异常Caused by: java.io.IOException: Compressed length 842086665 exceeds max block size 67108864 (probably corrupt file)

    hive查询lzo数据格式文件的表时,抛 Caused by: java.io.IOException: Compressed length 842086665 exceeds max block s ...

  4. hive报错：Failed with exception java.io.IOException: rename for src path:

    在hive中,会有这样一种情形: 1.创建一个分区外部表A(比如A表有5个字段),并且向A表里指定的分区(比如20160928这个分区)里插入数据 2.发现A表缺少一些字段,因为存在元数据不实时更新的 ...

  5. hive运行query语句时提示错误：org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException:

    hive> select product_id, track_time from trackinfo limit 5; Total MapReduce jobs = 1 Launching Jo ...

  6. 解决hiveserver2报错：java.io.IOException: Job status not available - Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

    用户使用的sql: select count( distinct patient_id ) from argus.table_aa000612_641cd8ce_ceff_4ea0_9b27_0a3a ...

  7. 解决方案--java执行cmd命令ProcessBuilder--出错Exception in thread "main" java.io.IOException: Cannot run program "dir d:\": CreateProcess error=2(xjl456852原创)

    当我尝试在java中通过ProcessBuilder运行window的cmd命令时出现错误: public static void main(String [] args) throws IOExce ...

  8. AIO 开始不定时的抛异常： java.io.IOException: 指定的网络名不再可用

    一天里会抛出几个这样的错误,但发现服务还在正常的运行. java.io.IOException: 指定的网络名不再可用. at sun.nio.ch.Iocp.translateErrorToIOEx ...

  9. ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

    sqoop从mysql导入到hive报错: 18/08/22 13:30:53 ERROR tool.ImportTool: Import failed: java.io.IOException: j ...

随机推荐

  1. Python装饰函数

    from time import ctime, sleep def tsfunc(func): def wrappedFunc(): print('[%s] %s() classed' % (ctim ...

  2. UVA 11776 - Oh Your Royal Greediness! - [贪心/模拟]

    题目链接:https://cn.vjudge.net/problem/UVA-11776 题意: 给出数字n(0<=n<=1000),代表有n个农民,接下来有n行,每行两个数字S和E代表这 ...

  3. Oracle安装部署之命令建库

    1.建目录: [oracle@wen ~]$ mkdir $ORACLE_BASE/admin/rezin/{a,b,c,dp}dump -p [oracle@wen ~]$ mkdir $ORACL ...

  4. 忽略Git仓库中已经存在的文件

    解决方案 使用Git bash进入到要忽略的文件所在的文件夹,执行以下命令即可 git update-index --assume-unchanged rebel.xml 参考 Ignore Git ...

  5. AOP与JAVA动态代理

    1.AOP的各种实现 AOP就是面向切面编程,我们可以从以下几个层面来实现AOP 在编译期修改源代码 在运行期字节码加载前修改字节码 在运行期字节码加载后动态创建代理类的字节码 2.AOP各种实现机制 ...

  6. Vue打开新页面的方法

    let routeData = this.$router.resolve({ name: "detail", query: {goodsId:'1111'} }); window. ...

  7. JS模块化编程（二）

    背景 我们常在页面引用js遇到下面情况 <script src="1.js"></script> <script src="2.js&quo ...

  8. 【查阅】mysql系统视图查看

    [1]查看表大小 SELECT CONCAT(table_schema,'.',table_name) AS 'Table Name', table_rows AS 'Number of Rows', ...

  9. android 6.0之后动态获取权限

    Android 6.0 动态权限申请   1. 概述 Android 6.0 (API 23) 之前应用的权限在安装时全部授予,运行时应用不再需要询问用户.在 Android 6.0 或更高版本对权限 ...

  10. Java-SpringMvc-@ResponseBody返回中文字符串乱码

    第一种.注解 @RequestMapping(value = "/test.do", method = {RequestMethod.GET},produces = "t ...