java执行spark查询hbase的jar包出现错误提示:ob aborted due to stage failure: Master removed our application: FAILED
执行java调用scala 打包后的jar时候出现异常
/14 23:57:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:23 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:38 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:39 INFO AppClient$ClientActor: Executor updated: app-20150414235011-0003/9 is now EXITED (Command exited with code 1)
15/04/14 23:57:39 INFO SparkDeploySchedulerBackend: Executor app-20150414235011-0003/9 removed: Command exited with code 1
15/04/14 23:57:39 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: Master removed our application: FAILED
15/04/14 23:57:39 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
15/04/14 23:57:39 INFO TaskSchedulerImpl: Cancelling stage 0
15/04/14 23:57:39 INFO DAGScheduler: Failed to run count at SparkSelect03.scala:55
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1049)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1033)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1031)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1031)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:635)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1234)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
问题1:
/14 23:57:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:23 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:38 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memor
分析:这个是内存不足?
我spark-env.sh的配置文件信息如下
export JAVA_HOME=/home/hadoop/jdk1.7.0_75
export SCALA_HOME=/home/hadoop/scala-2.11.6
export HADOOP_HOME=/home/hadoop/hadoop-2.3.0-cdh5.0.2
export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.3.0-cdh5.0.2/etc/hadoop
export SPARK_CLASSPATH=/home/hadoop/hbase-0.96.1.1-cdh5.0.2/lib/*
export SPARK_MASTER_IP=master
export SPARK_MASTER_PORT=17077
export SPARK_MASTER_WEBUI_PORT=18080 export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=1g
export SPARK_WORKER_WEBUI_PORT=18081
export SPARK_WORKER_INSTANCES=1
问题2:
15/04/14 23:57:39 INFO DAGScheduler: Failed to run count at SparkSelect03.scala:55
这句话的代码:
val count = hbaseRDD.count()
println("HBase RDD Count:" + count)
hbaseRDD.cache()
问题3:
in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
有遇到过类似的或者知道怎么解决的可以留言下
java执行spark查询hbase的jar包出现错误提示:ob aborted due to stage failure: Master removed our application: FAILED的更多相关文章
- 在服务器上执行hbase的jar包
hadoop命令执行hbase应用jar包时的环境变量加载问题 Apache HBase ™ Reference Guide HBase, MapReduce, and the CLASSPATH
- java调用scala 查询hbase数据
问题:将scala打成jar包,提供给java调用,但是java一直提示找不到类 实现功能:利用spark查询hbase数据,然后提供给外部接口调用 我的方式:spark查询Hbase用scala实现 ...
- Java SE 9 多版本兼容 JAR 包示例
Java SE 9 多版本兼容 JAR 包示例 作者:Grey 原文地址:Java SE 9 多版本兼容 JAR 包示例 说明 Java 9 版本中增强了Jar 包多版本字节码文件格式支持,也就是说在 ...
- eclipse java项目中明明引入了jar包 为什么项目启动的时候不能找到jar包 项目中已经 引入了 com.branchitech.app 包 ,但时tomcat启动的时候还是报错? java.lang.ClassNotFoundException: com.branchitech.app.startup.AppStartupContextListener java.lang.ClassN
eclipse java项目中明明引入了jar包 为什么项目启动的时候不能找到jar包 项目中已经 引入了 com.branchitech.app 包 ,但时tomcat启动的时候还是报错?java. ...
- 转!java web项目 build path 导入jar包,tomcat启动报错 找不到该类
在eclipse集成tomcat开发java web项目时,引入的外部jar包,编译通过,但启动tomcat运行web时提示找不到jar包内的类,需要作如下配置,将jar包在部署到集成的tomcat环 ...
- 不要轻易在java ext 目录放任何三方jar包
今天在编写一个简单spi 应用demo的时候,在编译时总有一个其他的错误,如下: ERROR Failed to execute goal org.apache.maven.plugins:maven ...
- 如果Android的jar包导入错误,怎么修改呢?
如果jar包导入错误,怎么修改呢? 右键工程---->properties---->Java Build Path --->Libraries-->选择android-supp ...
- java应用程序远程登录linux并执行其命令(ssh jar包)
http://www.ganymed.ethz.ch/ssh2/在这个网址下载一个调用ssh和scp命令的jar包. 然后,就可以写程序了.将上面的jar包导入MyEclipse,下面是一个类的实例代 ...
- java中如何根据函数查询引用的jar包
选中函数,按Ctrl+Shift+T,就可以弹出对应的jar包地址 例如:
随机推荐
- centos 下安装 shpinx2.1.7 记录
安装sphinx yum install -y mysql mysql-devel yum install automake autoconf cd /usr/local/src/ wget http ...
- mysql性能分析与故障分析
性能指标以及分析点:QPS,磁盘IO,cpu以及内存使用状况,连接数,慢查询,阻塞和锁等 qps的统计方法 ##使用方式 ,bash dba_qps.sh port(实例端口号) ## 脚本名 dba ...
- A. Srdce and Triangle--“今日头条杯”首届湖北省大学程序设计竞赛(网络同步赛)
如下图这是“今日头条杯”首届湖北省大学程序设计竞赛的第一题,作为赛后补题 题目描述:链接点此 这套题的github地址(里面包含了数据,题解,现场排名):点此 Let be a regualr tr ...
- php中如何实现多进程
php中如何实现多进程 一.总结 一句话总结: php多进程需要pcntl,posix扩展支持 可以通过 php - m 查看,没安装的话需要重新编译php,加上参数--enable-pcntl,po ...
- Python 中练习题涉及到的无穷大和无穷小问题。
首先来看一下所见的python联系题. inf = infinite 无限制的 float("inf")-1执行后的结果是:() A 1 B inf C -inf D 0 该考点考 ...
- unittest框架学习笔记一之testcase
# coding=utf-8案例一: 2 ''' 3 Created on 2017-7-22 4 @author: Jennifer 5 Project:登录百度测试用例 6 ''' 7 from ...
- springboot接口:CommandLineRunner
springBoot接口:CommandLineRunner 一.作用: 在使用SpringBoot构建项目时,我们通常有一些预先数据的加载.那么SpringBoot提供了一个简单的方式来实现–Com ...
- 在Python的列表中利用remove()方法删除元素的教程
在Python的列表中利用remove()方法删除元素的教程 这篇文章主要介绍了在Python的列表中利用remove()方法删除元素的教程,是Python入门中的基础知识,注意其和pop()方法的区 ...
- Linux的命名空间
1. 为什么提供命名空间 命名空间是一种轻量级的虚拟化手段. 传统的虚拟化软件,是虚拟化多个不同的操作系统,对共享资源的限制很大. 通过提供命名空间,可以让进程与进程之间,用户与用户之间彼此看不到对方 ...
- php ZipArchive 压缩整个文件夹
// Get real path for our folder $rootPath = realpath('folder-to-zip'); // Initialize archive object ...