之前在mac上调试hadoop程序(mac之前配置过hadoop环境)一直都是正常的。因为工作需要,需要在windows上先调试该程序,然后再转到linux下。程序运行的过程中,报 Failed to locate the winutils binary in the hadoop binary path   java.io.IOException: Could not locate executable null \bin\winutils.exe in the Hadoop binaries.

通过断点调试、查看源码发现程序需要根据HADOOP_HOME找到winutils.exe,由于win机器并没有配置该环境变量,所以程序报 null\bin\winutils.exe。

private static String checkHadoopHome() {
// first check the Dflag hadoop.home.dir with JVM scope
String home = System.getProperty("hadoop.home.dir"); // fall back to the system/user-global env variable
if (home == null) {
home = System.getenv("HADOOP_HOME");
}
try {
// couldn't find either setting for hadoop's home directory
if (home == null) {
throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
}
if (home.startsWith("\"") && home.endsWith("\"")) {
home = home.substring(1, home.length()-1);
}
// check that the home setting is actually a directory that exists
File homedir = new File(home);
if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
throw new IOException("Hadoop home directory " + homedir
+ " does not exist, is not a directory, or is not an absolute path.");
}
home = homedir.getCanonicalPath();
} catch (IOException ioe) {
if (LOG.isDebugEnabled()) {
LOG.debug("Failed to detect a valid hadoop home directory", ioe);
}
home = null;
}
return home;
}

private static String HADOOP_HOME_DIR = checkHadoopHome ();

public static final String getQualifiedBinPath(String executable)
throws IOException {
// construct hadoop bin path to the specified executable
String fullExeName = HADOOP_HOME_DIR + File.separator + "bin"
+ File.separator + executable; File exeFile = new File(fullExeName);
if (!exeFile.exists()) {
throw new IOException("Could not locate executable " + fullExeName
+ " in the Hadoop binaries.");
}
return exeFile.getCanonicalPath();
} /** a Windows utility to emulate Unix commands */
public static final String WINUTILS = getWinUtilsPath(); public static final String getWinUtilsPath() {
String winUtilsPath = null; try {
if (WINDOWS) {
winUtilsPath = getQualifiedBinPath("winutils.exe");
}
} catch (IOException ioe) {
LOG.error("Failed to locate the winutils binary in the hadoop binary path",
ioe);
} return winUtilsPath;
}

找到原因后就去网上问了度娘,找到了解决方案,很简单,如下:

1.下载winutils的windows版本

GitHub上,有人提供了winutils的windows的版本,项目地址是: https://github.com/srccodes/hadoop-common-2.2.0-bin ,直接下载此项目的zip包,下载后是文件名是hadoop-common-2.2.0-bin-master.zip,随便解压到一个目录

2.配置环境变量

增加用户变量HADOOP_HOME,值是下载的zip包解压的目录,然后在系统变量path里增加$HADOOP_HOME\bin 即可。

再次运行程序,正常执行。

Spark报错:Failed to locate the winutils binary in the hadoop binary path的更多相关文章

  1. WIN7下运行hadoop程序报:Failed to locate the winutils binary in the hadoop binary path

    之前在mac上调试hadoop程序(mac之前配置过hadoop环境)一直都是正常的.因为工作需要,需要在windows上先调试该程序,然后再转到linux下.程序运行的过程中,报Failed to ...

  2. 解决spark运行中failed to locate the winutils binary in the hadoop binary path的问题

    1.下载hadoop-common-2.2.0-bin并解压到某个目录 https://github.com/srccodes/hadoop-common-2.2.0-bin 2.设置hadoop.h ...

  3. Windows本地运行调试Spark或Hadoop程序失败:ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path

    报错内容 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOExce ...

  4. Hadoop:开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary path

    问题: windows开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary ...

  5. Windows7系统运行hadoop报Failed to locate the winutils binary in the hadoop binary path错误

    程序运行的过程中,报Failed to locate the winutils binary in the hadoop binary path  Java.io.IOException: Could ...

  6. Spark- ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

    运行 mport org.apache.log4j.{Level, Logger} import org.apache.spark.rdd.RDD import org.apache.spark.{S ...

  7. windows本地调试安装hadoop(idea) : ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path

    1,本地安装hadoop https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/ 下载hadoop对应版本 (我本意是想下载hadoop ...

  8. ERROR Shell: Failed to locate the winutils binary in the hadoop binary path

    文章发自:http://www.cnblogs.com/hark0623/p/4170172.html  转发请注明 14/12/17 19:18:53 ERROR Shell: Failed to ...

  9. spark报错处理

    Spark报错处理 1.问题:org.apache.spark.SparkException: Exception thrown in awaitResult 分析:出现这个情况的原因是spark启动 ...

随机推荐

  1. cf812B 搜索

    B. Sagheer, the Hausmeister time limit per test 1 second memory limit per test 256 megabytes input s ...

  2. 出错的sql

    alter table m_account  AUTO_INCREMENT=200029860;

  3. 移动web开发适配方案之Rem

    移动端为什么要做适配 移动端相对PC端来说大部分浏览器内核都是基于Webkit的,所以大部分都支持CSS3的最新语法.但是由于手机的屏幕尺寸和分辨率都不太一样(尤其是安卓),所以不得不对不同分辨率的手 ...

  4. mysql基础运维

    1.创建用户并授权 一般新建数据库都需要新增一个用户,用于程序连接,这类用户只需要insert.update.delete.select权限. 新增一个用户,并授权如下: (1)grant selec ...

  5. new BigDecimal(0.01) 与 new BigDecimal(String.valueOf(0.01))的区别 (转)

    转自:http://blog.csdn.net/major1985/article/details/50210293 一般我们使用BigDecimal进行比较精密的计算,我这里计算金额.注意使用dou ...

  6. Java例子

    1. 本章学习总结 今天主要学习了三个知识点 封装 继承 多态 2. 书面作业 Q1. java HelloWorld命令中,HelloWorld这个参数是什么含义? 今天学了一个重要的命令javac ...

  7. Ubuntu系统安装,适用于14.04,16.04和17.10

    本文以14.04为案例进行安装,其他版本相关问题会做注解 1.选择要安装的系统语言 本界面建议选择English,之后再选择中文安装 注意: 安装服务器版时,对于14.x版本第一步选择中文没有问题,但 ...

  8. 20155234 2016-2017-2 《Java程序设计》第8周学习总结

    20155234 2016-2017-2 <Java程序设计>第8周学习总结 教材学习内容总结 java.util.loggging包提供了日志功能相关类与接口. 使用日志的起点是Logg ...

  9. ElasticSearch(一):windows下安装ElasticSearch6.3.2

    前言 最近开始学习ElasticSearch,从官网上下载的最新版的es,但是网上安装教程大部分是早期5.X版本的,因为6.x版本出现了不同,所以记录下. 正文 1. 下载ElasticSearch ...

  10. 6-3 Add Two Polynomials(20 分)

    Write a function to add two polynomials. Do not destroy the input. Use a linked list implementation ...