Spark执行样例报警告:WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources
搭建Spark环境后,调测Spark样例时,出现下面的错误:
WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
[hadoop@gpmaster bin]$ ./run-example org.apache.spark.examples.SparkPi
15/10/01 08:59:33 INFO spark.SparkContext: Running Spark version 1.5.0
.......................
15/10/01 08:59:35 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.128:17514]
15/10/01 08:59:35 INFO util.Utils: Successfully started service 'sparkDriver' on port 17514.
.......................
15/10/01 08:59:36 INFO ui.SparkUI: Started SparkUI at http://192.168.1.128:4040
15/10/01 08:59:37 INFO spark.SparkContext: Added JAR file:/home/hadoop/spark/lib/spark-examples-1.5.0-hadoop2.6.0.jar at http://192.168.1.128:36471/jars/spark-examples-1.5.0-hadoop2.6.0.jar with timestamp 1443661177865
15/10/01 08:59:37 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
15/10/01 08:59:38 INFO client.AppClient$ClientEndpoint: Connecting to master spark://192.168.1.128:7077...
15/10/01 08:59:38 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20151001085938-0000
.................................
15/10/01 08:59:40 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
15/10/01 08:59:55 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:00:10 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:00:25 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:00:40 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:00:55 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:01:10 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:01:25 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:01:40 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:01:55 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:02:10 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:02:25 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
从警告信息大致可以知道:
初始化job时没有获取到任何资源;提示检查集群,确保workers可以被注册并有足够的内存资源。
可能的原因有几点,可以逐个排查:
1. 主机主机名和ip是否配置正确
先查看/etc/hosts文件配置是否正确
同时可以通过spark-shell查看SparkContext获取的上下文信息, 如下操作:
[hadoop@gpmaster bin]$ ./spark-shell
........
scala> sc.getConf.getAll.foreach(println)
(spark.fileserver.uri,http://192.168.1.128:34634)
(spark.app.name,Spark shell)
(spark.driver.port,25392)
(spark.app.id,app-20151001090322-0001)
(spark.repl.class.uri,http://192.168.1.128:24988)
(spark.externalBlockStore.folderName,spark-1254a794-fbfa-4b4c-9757-b5a94dc26ffc)
(spark.jars,)
(spark.executor.id,driver)
(spark.submit.deployMode,client)
(spark.driver.host,192.168.1.128)
(spark.master,spark://192.168.1.128:7077)
scala> sc.getConf.toDebugString
res8: String =
spark.app.id=app-20151001090322-0001
spark.app.name=Spark shell
spark.driver.host=192.168.1.128
spark.driver.port=25392
spark.executor.id=driver
spark.externalBlockStore.folderName=spark-1254a794-fbfa-4b4c-9757-b5a94dc26ffc
spark.fileserver.uri=http://192.168.1.128:34634
spark.jars=
spark.master=spark://192.168.1.128:7077
spark.repl.class.uri=http://192.168.1.128:24988
spark.submit.deployMode=client
2. 内存不足
我的环境就是因为内存的原因。
我集群环境中,spark-env.sh 文件配置如下:
export JAVA_HOME=/usr/java/jdk1.7.0_60
export SCALA_HOME=/usr/local/scala
export SPARK_MASTER_IP=192.168.1.128
export SPARK_WORKER_MEMORY=100m
export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.6.0/etc/hadoop
export MASTER=spark://192.168.1.128:7077
因为我的集群环境,每个节点只剩下500MB内存了,由于我没有配置SPARK_EXECUTOR_MEMORY参数,默认会使用1G内存,所以会出现内存不足,从而出现上面日志报的警告信息。
所以解决办法是添加如下参数:
export SPARK_EXECUTOR_MEMORY=512m
3.端口号被占用,之前的程序已运行。
Spark执行样例报警告:WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources的更多相关文章
- 18/03/18 04:53:44 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
1:遇到这个问题是在启动bin/spark-shell以后,然后呢,执行spark实现wordcount的例子的时候出现错误了,如: scala> sc.textFile()).reduceBy ...
- Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
Spark submit任务到Spark集群时,会出现如下异常: Exception 1:Initial job has not accepted any resources; check your ...
- azure iothub create-device-identity样例报错: unable to find valid certification path ,及iothub-explorer Error: CERT_UNTRUSTED
https://docs.microsoft.com/zh-cn/azure/iot-hub/iot-hub-java-java-getstarted 在IDEA中执行上述的代码,会出现下面的报错信息 ...
- Pytest执行用例报Hint: make sure your test modules/packages have valid Python names.
近日,使用Pytest+Appium 实现APP端UI自动化,遇到Pytest收集用例失败的情况. 报错信息如下: test_room.py:None (test_room.py) ImportErr ...
- Eureka 的 Application Service client的注冊以及执行演示样例
Eureka 服务器架起来了(关于架设步骤參考博客<Linux 下 Eureka 服务器的部署>),如今怎样把我们要负载均衡的服务器(也就是从 Application Cl ...
- spark mllib lda 中文分词、主题聚合基本样例
github https://github.com/cclient/spark-lda-example spark mllib lda example 官方示例较为精简 在官方lda示例的基础上,给合 ...
- Android OpenCV样例调试+报错处理
1.OpenCV样例调试:<OpenCV Sample - image-manipulations> blog+报错:E/CAMERA_ACTIVITY(17665): Cam ...
- Eureka 的 Application Client client的执行演示样例
上篇以一个 demo 演示样例介绍了 Eureka 的 Application Service 客户端角色.今天我们继续了解 Eureka 的 Application Client 客 ...
- 在Ubuntu下构建Bullet以及执行Bullet的样例程序
在Ubuntu下构建Bullet以及执行Bullet的样例程序 1.找到Bullet的下载页,地址是:https://code.google.com/p/bullet/downloads/list 2 ...
随机推荐
- JavaScript内存泄漏知多少?
垃圾回收解放了我们,它让我们可将精力集中在应用程序逻辑(而不是内存管理)上.但是,垃圾收集并不神奇.了解它的工作原理,以及如何使它保留本应在很久以前释放的内存,就可以实现更快更可靠的应用程序.在本文中 ...
- mysql 命令行导数据库
cd d: 然后应该会提示你已经进入D盘了,按照你数据库的地址,我的数据库是在D盘的wamp这个文件夹目录, 输入命令:cd ruanjian\mysql\bin 类似于这样,大家可以先在本 ...
- 如何使用POST 方法调用服务
一.WCF REST专用POST方法 1.1. 建立WCF REST 方法 [ServiceContract] public interface IBookingBizService { ...
- 转mysql横向扩展和纵向扩展
Scale-up(纵向扩展)和Scale-out(横向扩展)的解释 谈到系统的可伸缩性,Scale-up(纵向扩展)和Scale-out(横向扩展)是两个常见的术语,对于初学者来说,很容易搞迷糊这两个 ...
- centos网卡配置
DEVICE=物理设备名 IPADDR=IP地址 NETMASK=掩码值 NETWORK=网络地址 BROADCAST=广播地址 GATEWAY=网关地址 TYPE=Ethernet (网络类型)ON ...
- Python之NumPy中线性代数
参考博客:http://blog.csdn.net/u013930163/article/details/51839983 网站:https://docs.scipy.org/doc/numpy-de ...
- CodeForces 55D Beautiful numbers (SPOJ JZPEXT 数位DP)
题意 求[X,Y]区间内能被其各位数(除0)均整除的数的个数. CF 55D 有些时候因为问题的一些"整体性"而导致在按位统计的过程中不能顺便计算出某些量,所以只能在枚举到最后一位 ...
- SpringAnnotation之配置AnnotationXML文件
配置Annotation的环境:只需修改applicationContext.xml文件即可 1 2 3 4 5 6 7 8 9 10 11 <?xml version="1.0&qu ...
- php5.4.0以后加入trait实现代码复用【摘录】
在http://www.php.net/manual/zh/language.oop5.traits.php 查看了一下5.4.0提供的特性trait顺便做一个标记(总结): 首先要明确trait和类 ...
- Error:Cause: org/gradle/api/publication/maven/internal/DefaultMavenFactory 解决办法
当你使用的Gradle版本是2.4以上,Android插件版本是1.3.0以上的时候就会出现这个问题,这时候你只需将android-maven-gradle-plugin插件版本改为classpath ...