Hadoop:开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
问题:
windows开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary path,但是可以正常执行,并不影响结果。
// :: WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
// :: ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:)
at org.apache.hadoop.security.Groups.<init>(Groups.java:)
at org.apache.hadoop.security.Groups.<init>(Groups.java:)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$.apply(Utils.scala:)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$.apply(Utils.scala:)
at scala.Option.getOrElse(Option.scala:)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:)
at com.lm.sparkLearning.utils.SparkUtils.getJavaSparkContext(SparkUtils.java:)
at com.lm.sparkLearning.rdd.RddLearning.main(RddLearning.java:)
// :: WARN RddLearning: singleOperateRdd mapRdd->[, , , ]
// :: WARN RddLearning: singleOperateRdd flatMapRdd->[, , , , , , , ]
// :: WARN RddLearning: singleOperateRdd filterRdd->[, ]
// :: WARN RddLearning: singleOperateRdd distinctRdd->[, , ]
// :: WARN RddLearning: singleOperateRdd sampleRdd->[, ]
// :: WARN RddLearning: the program end
这里所执行的程序是:
package com.lm.sparkLearning.rdd; import java.util.Arrays;
import java.util.Iterator;
import java.util.List; import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.VoidFunction;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import com.lm.sparkLearning.utils.SparkUtils; public class RddLearning {
private static Logger logger = LoggerFactory.getLogger(RddLearning.class); public static void main(String[] args) { JavaSparkContext jsc = SparkUtils.getJavaSparkContext("RDDLearning", "local[2]", "WARN"); SparkUtils.createRddExternal(jsc, "D:/README.txt");
singleOperateRdd(jsc); jsc.stop(); logger.warn("the program end");
} public static void singleOperateRdd(JavaSparkContext jsc) {
List<Integer> nums = Arrays.asList(new Integer[] { 1, 2, 3, 3 });
JavaRDD<Integer> numsRdd = SparkUtils.createRddCollect(jsc, nums); // map
JavaRDD<Integer> mapRdd = numsRdd.map(new Function<Integer, Integer>() {
private static final long serialVersionUID = 1L; @Override
public Integer call(Integer v1) throws Exception {
return (v1 + 1);
}
}); logger.warn("singleOperateRdd mapRdd->" + mapRdd.collect().toString()); JavaRDD<Integer> flatMapRdd = numsRdd.flatMap(new FlatMapFunction<Integer, Integer>() {
private static final long serialVersionUID = 1L; @Override
public Iterable<Integer> call(Integer t) throws Exception {
return Arrays.asList(new Integer[] { 2, 3 });
}
}); logger.warn("singleOperateRdd flatMapRdd->" + flatMapRdd.collect().toString()); JavaRDD<Integer> filterRdd = numsRdd.filter(new Function<Integer, Boolean>() {
private static final long serialVersionUID = 1L; @Override
public Boolean call(Integer v1) throws Exception {
return v1 > 2;
}
}); logger.warn("singleOperateRdd filterRdd->" + filterRdd.collect().toString()); JavaRDD<Integer> distinctRdd = numsRdd.distinct(); logger.warn("singleOperateRdd distinctRdd->" + distinctRdd.collect().toString()); JavaRDD<Integer> sampleRdd = numsRdd.sample(false, 0.5); logger.warn("singleOperateRdd sampleRdd->" + sampleRdd.collect().toString());
}
}
解决方案:
1.下载winutils的windows版本
GitHub上,有人提供了winutils的windows的版本,项目地址是:https://github.com/srccodes/hadoop-common-2.2.0-bin,直接下载此项目的zip包,下载后是文件名是hadoop-common-2.2.0-bin-master.zip,随便解压到一个目录。
2.配置环境变量
增加用户变量HADOOP_HOME,值是下载的zip包解压的目录,然后在系统变量path里增加$HADOOP_HOME\bin 即可。
添加“%HADOOP%\bin”到path
再次运行程序,正常执行。
Hadoop:开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary path的更多相关文章
- ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
文章发自:http://www.cnblogs.com/hark0623/p/4170172.html 转发请注明 14/12/17 19:18:53 ERROR Shell: Failed to ...
- Spark- ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
运行 mport org.apache.log4j.{Level, Logger} import org.apache.spark.rdd.RDD import org.apache.spark.{S ...
- Windows本地运行调试Spark或Hadoop程序失败:ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
报错内容 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOExce ...
- idea 提示:ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException解决方法
Windows系统中的IDEA链接Linux里面的Hadoop的api时出现的问题 提示:ERROR util.Shell: Failed to locate the winutils binary ...
- windows本地调试安装hadoop(idea) : ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
1,本地安装hadoop https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/ 下载hadoop对应版本 (我本意是想下载hadoop ...
- ERROR [org.apache.hadoop.util.Shell] - Failed to locate the winutils binary in the hadoop binary path
错误日志如下: -- ::, DEBUG [org.apache.hadoop.metrics2.lib.MutableMetricsFactory] - field org.apache.hadoo ...
- WIN7下运行hadoop程序报:Failed to locate the winutils binary in the hadoop binary path
之前在mac上调试hadoop程序(mac之前配置过hadoop环境)一直都是正常的.因为工作需要,需要在windows上先调试该程序,然后再转到linux下.程序运行的过程中,报Failed to ...
- Windows7系统运行hadoop报Failed to locate the winutils binary in the hadoop binary path错误
程序运行的过程中,报Failed to locate the winutils binary in the hadoop binary path Java.io.IOException: Could ...
- Spark报错:Failed to locate the winutils binary in the hadoop binary path
之前在mac上调试hadoop程序(mac之前配置过hadoop环境)一直都是正常的.因为工作需要,需要在windows上先调试该程序,然后再转到linux下.程序运行的过程中,报 Failed to ...
随机推荐
- 怎么让链式调用setTimeout停止
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&quo ...
- 区别ES3ES5和ES6this的指向问题。区分普通函数和箭头函数中this的指向问题
ES3 ES5this的指向问题 this指的是该函数被调用的对象 var foo = function () { this.a = 'a', this.b = 'b', this.c = { a: ...
- JAVA泛型中的有界类型(extends super)(转)
JDK1.5中引入了泛型(Generic)机制.泛型的本质是参数化类型,也就是说所操作的数据类型被指定为一个参数.这种参数类型可以用在类.接口和方法的创建中,分别称为泛型类.泛型接口.泛型方法. Ja ...
- 下载8000首儿歌的python代码
下载8000首儿歌的python的代码: #-*- coding: UTF-8 -*- from pyquery import PyQuery as py from lxml import etree ...
- C#面向服务编程技术WCF从入门到实战演练
一.WCF课程介绍 1.1.Web Service会被WCF取代吗? 对于这个问题阿笨的回答是:两者在功能特性上却是有新旧之分,但是对于特定的系统,适合自己的就是最好的.不能哪一个技术框架和行业标准作 ...
- 基于TQ2440和Qemu的GDB+串口调试(1)
作者 彭东林 pengdonglin137@163.com 平台 TQ2440 + Linux-4.10.17 Qemu(vexpress-ca9) + Linux-4.10.17 概述 下面 ...
- jQuery统计上传文件的大小
对于现代浏览器(支持html5)来说,在客户端统计上传文件的大小,可以通过$(selector)[0].files[0].size来实现.但在老版本的IE浏览器中,比如IE7,IE8或IE9,却不支持 ...
- 使用RemObjects Pascal Script
摘自RemObjects Wiki 本文提供RemObjects Pascal Script的整体概要并演示如何创建一些简单的脚本. Pascal Script包括两个不同部分: 编译器 (uPSCo ...
- Windows Phone本地数据库(SQLCE):2、LINQ to SQL(翻译)(转)
首先.要说到的是,windows phone 7.1上基本的数据库功能是SQL Compact关于Mango的一个实现,使用linq to sql访问存储在数据库上的数据. 1.LINQ to S ...
- Visual Studio 2013 sqlce 配置(转)
Visual Studio 2013 把內建 SQL CE 的管理工具拿掉了 下载SQL Server Compact Toolbox by ErikEJ并安装 打开VS2013,新建一工程,在“视图 ...