之前在mac上调试hadoop程序(mac之前配置过hadoop环境)一直都是正常的。因为工作需要,需要在windows上先调试该程序,然后再转到linux下。程序运行的过程中,报 Failed to locate the winutils binary in the hadoop binary path   java.io.IOException: Could not locate executable null \bin\winutils.exe in the Hadoop binaries.

通过断点调试、查看源码发现程序需要根据HADOOP_HOME找到winutils.exe,由于win机器并没有配置该环境变量,所以程序报 null\bin\winutils.exe。

private static String checkHadoopHome() {
// first check the Dflag hadoop.home.dir with JVM scope
String home = System.getProperty("hadoop.home.dir"); // fall back to the system/user-global env variable
if (home == null) {
home = System.getenv("HADOOP_HOME");
}
try {
// couldn't find either setting for hadoop's home directory
if (home == null) {
throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
}
if (home.startsWith("\"") && home.endsWith("\"")) {
home = home.substring(1, home.length()-1);
}
// check that the home setting is actually a directory that exists
File homedir = new File(home);
if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
throw new IOException("Hadoop home directory " + homedir
+ " does not exist, is not a directory, or is not an absolute path.");
}
home = homedir.getCanonicalPath();
} catch (IOException ioe) {
if (LOG.isDebugEnabled()) {
LOG.debug("Failed to detect a valid hadoop home directory", ioe);
}
home = null;
}
return home;
}

private static String HADOOP_HOME_DIR = checkHadoopHome ();

public static final String getQualifiedBinPath(String executable)
throws IOException {
// construct hadoop bin path to the specified executable
String fullExeName = HADOOP_HOME_DIR + File.separator + "bin"
+ File.separator + executable; File exeFile = new File(fullExeName);
if (!exeFile.exists()) {
throw new IOException("Could not locate executable " + fullExeName
+ " in the Hadoop binaries.");
}
return exeFile.getCanonicalPath();
} /** a Windows utility to emulate Unix commands */
public static final String WINUTILS = getWinUtilsPath(); public static final String getWinUtilsPath() {
String winUtilsPath = null; try {
if (WINDOWS) {
winUtilsPath = getQualifiedBinPath("winutils.exe");
}
} catch (IOException ioe) {
LOG.error("Failed to locate the winutils binary in the hadoop binary path",
ioe);
} return winUtilsPath;
}

找到原因后就去网上问了度娘,找到了解决方案,很简单,如下:

1.下载winutils的windows版本

GitHub上,有人提供了winutils的windows的版本,项目地址是: https://github.com/srccodes/hadoop-common-2.2.0-bin ,直接下载此项目的zip包,下载后是文件名是hadoop-common-2.2.0-bin-master.zip,随便解压到一个目录

2.配置环境变量

增加用户变量HADOOP_HOME,值是下载的zip包解压的目录,然后在系统变量path里增加$HADOOP_HOME\bin 即可。

再次运行程序,正常执行。

Spark报错:Failed to locate the winutils binary in the hadoop binary path的更多相关文章

  1. WIN7下运行hadoop程序报:Failed to locate the winutils binary in the hadoop binary path

    之前在mac上调试hadoop程序(mac之前配置过hadoop环境)一直都是正常的.因为工作需要,需要在windows上先调试该程序,然后再转到linux下.程序运行的过程中,报Failed to ...

  2. 解决spark运行中failed to locate the winutils binary in the hadoop binary path的问题

    1.下载hadoop-common-2.2.0-bin并解压到某个目录 https://github.com/srccodes/hadoop-common-2.2.0-bin 2.设置hadoop.h ...

  3. Windows本地运行调试Spark或Hadoop程序失败:ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path

    报错内容 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOExce ...

  4. Hadoop:开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary path

    问题: windows开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary ...

  5. Windows7系统运行hadoop报Failed to locate the winutils binary in the hadoop binary path错误

    程序运行的过程中,报Failed to locate the winutils binary in the hadoop binary path  Java.io.IOException: Could ...

  6. Spark- ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

    运行 mport org.apache.log4j.{Level, Logger} import org.apache.spark.rdd.RDD import org.apache.spark.{S ...

  7. windows本地调试安装hadoop(idea) : ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path

    1,本地安装hadoop https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/ 下载hadoop对应版本 (我本意是想下载hadoop ...

  8. ERROR Shell: Failed to locate the winutils binary in the hadoop binary path

    文章发自:http://www.cnblogs.com/hark0623/p/4170172.html  转发请注明 14/12/17 19:18:53 ERROR Shell: Failed to ...

  9. spark报错处理

    Spark报错处理 1.问题:org.apache.spark.SparkException: Exception thrown in awaitResult 分析:出现这个情况的原因是spark启动 ...

随机推荐

  1. 2018HN多校

    http://acm.hi-54.com/contest_problemset.php?cid=1455 A : 摩斯密码 概览问题列表状态排名 Progress Bar 时间限制:1 Sec 内存限 ...

  2. Oracle linux安装Oracle 11G

    Oracle  linux安装Oracle 11G 系统环境 Oracle linux   5.8.19.6(64位) Oracle     11.2.0.1(64位) 本文档详细介绍Oracle  ...

  3. sqlplus环境设置

    1.0 --column 命令集                         改变列格式                {                column column_name1 f ...

  4. IIS网页GZIP压缩

    1.开GZIP有什么好处? 答:Gzip开启以后会将输出到用户浏览器的数据进行压缩的处理,这样就会减小通过网络传输的数据量,提高浏览的速度. 2.如何启用IIS的Gzip压缩功能: 答:首先,如果你需 ...

  5. Chrome即将封杀Google Earth、Google Talk等插件

    昨日,Chrome安全工程师Justin Schuh在官方博客中写道,到明年一月份,谷歌将封杀一系列基于NPAPI框架标准的浏览器插件.其中包括谷歌地球(Google Earth).Google Ta ...

  6. java基础第9天

    抽象 abstract 抽象类和抽象方法必须用abstract关键字修饰 抽象类格式 abstract class 类名{} 抽象方法定义,在返回值钱,或修饰符前加上abstract关键字 方法没有方 ...

  7. 【python】venv使用

    virtualenvwrapper 比 virualenv 好用一些. 准备 export WORKON_HOME=~/venv source /usr/bin/virtualenvwrapper.s ...

  8. I.MX6 CAAM

    /********************************************************************************* * I.MX6 CAAM * 说明 ...

  9. ssh命令的使用

    ssh命令是用来远程登录服务器的,默认端口号为22. 常用的命令是: ssh   用户名@服务器地址 然后回车,输入登录服务器的密码 登录服务器后要进行上传下载文件,可以用scp命令,命令格式为: s ...

  10. USB gadget学习笔记

    1.usb-OTG-ADP-HNP-SRP https://blog.csdn.net/xiongjiao0610/article/details/44150849