Spark报错:Failed to locate the winutils binary in the hadoop binary path
之前在mac上调试hadoop程序(mac之前配置过hadoop环境)一直都是正常的。因为工作需要,需要在windows上先调试该程序,然后再转到linux下。程序运行的过程中,报 Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable
null
\bin\winutils.exe in the Hadoop binaries.
通过断点调试、查看源码发现程序需要根据HADOOP_HOME找到winutils.exe,由于win机器并没有配置该环境变量,所以程序报 null\bin\winutils.exe。
private static String checkHadoopHome() {
// first check the Dflag hadoop.home.dir with JVM scope
String home = System.getProperty("hadoop.home.dir"); // fall back to the system/user-global env variable
if (home == null) {
home = System.getenv("HADOOP_HOME");
}
try {
// couldn't find either setting for hadoop's home directory
if (home == null) {
throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
}
if (home.startsWith("\"") && home.endsWith("\"")) {
home = home.substring(1, home.length()-1);
}
// check that the home setting is actually a directory that exists
File homedir = new File(home);
if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
throw new IOException("Hadoop home directory " + homedir
+ " does not exist, is not a directory, or is not an absolute path.");
}
home = homedir.getCanonicalPath();
} catch (IOException ioe) {
if (LOG.isDebugEnabled()) {
LOG.debug("Failed to detect a valid hadoop home directory", ioe);
}
home = null;
}
return home;
}
private static String HADOOP_HOME_DIR = checkHadoopHome ();
public static final String getQualifiedBinPath(String executable)
throws IOException {
// construct hadoop bin path to the specified executable
String fullExeName = HADOOP_HOME_DIR + File.separator + "bin"
+ File.separator + executable; File exeFile = new File(fullExeName);
if (!exeFile.exists()) {
throw new IOException("Could not locate executable " + fullExeName
+ " in the Hadoop binaries.");
}
return exeFile.getCanonicalPath();
} /** a Windows utility to emulate Unix commands */
public static final String WINUTILS = getWinUtilsPath(); public static final String getWinUtilsPath() {
String winUtilsPath = null; try {
if (WINDOWS) {
winUtilsPath = getQualifiedBinPath("winutils.exe");
}
} catch (IOException ioe) {
LOG.error("Failed to locate the winutils binary in the hadoop binary path",
ioe);
} return winUtilsPath;
}
找到原因后就去网上问了度娘,找到了解决方案,很简单,如下:
1.下载winutils的windows版本
GitHub上,有人提供了winutils的windows的版本,项目地址是: https://github.com/srccodes/hadoop-common-2.2.0-bin ,直接下载此项目的zip包,下载后是文件名是hadoop-common-2.2.0-bin-master.zip,随便解压到一个目录
2.配置环境变量
增加用户变量HADOOP_HOME,值是下载的zip包解压的目录,然后在系统变量path里增加$HADOOP_HOME\bin 即可。
再次运行程序,正常执行。
Spark报错:Failed to locate the winutils binary in the hadoop binary path的更多相关文章
- WIN7下运行hadoop程序报:Failed to locate the winutils binary in the hadoop binary path
之前在mac上调试hadoop程序(mac之前配置过hadoop环境)一直都是正常的.因为工作需要,需要在windows上先调试该程序,然后再转到linux下.程序运行的过程中,报Failed to ...
- 解决spark运行中failed to locate the winutils binary in the hadoop binary path的问题
1.下载hadoop-common-2.2.0-bin并解压到某个目录 https://github.com/srccodes/hadoop-common-2.2.0-bin 2.设置hadoop.h ...
- Windows本地运行调试Spark或Hadoop程序失败:ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
报错内容 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOExce ...
- Hadoop:开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
问题: windows开发机运行spark程序,抛出异常:ERROR Shell: Failed to locate the winutils binary in the hadoop binary ...
- Windows7系统运行hadoop报Failed to locate the winutils binary in the hadoop binary path错误
程序运行的过程中,报Failed to locate the winutils binary in the hadoop binary path Java.io.IOException: Could ...
- Spark- ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
运行 mport org.apache.log4j.{Level, Logger} import org.apache.spark.rdd.RDD import org.apache.spark.{S ...
- windows本地调试安装hadoop(idea) : ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
1,本地安装hadoop https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/ 下载hadoop对应版本 (我本意是想下载hadoop ...
- ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
文章发自:http://www.cnblogs.com/hark0623/p/4170172.html 转发请注明 14/12/17 19:18:53 ERROR Shell: Failed to ...
- spark报错处理
Spark报错处理 1.问题:org.apache.spark.SparkException: Exception thrown in awaitResult 分析:出现这个情况的原因是spark启动 ...
随机推荐
- 2018HN多校
http://acm.hi-54.com/contest_problemset.php?cid=1455 A : 摩斯密码 概览问题列表状态排名 Progress Bar 时间限制:1 Sec 内存限 ...
- Oracle linux安装Oracle 11G
Oracle linux安装Oracle 11G 系统环境 Oracle linux 5.8.19.6(64位) Oracle 11.2.0.1(64位) 本文档详细介绍Oracle ...
- sqlplus环境设置
1.0 --column 命令集 改变列格式 { column column_name1 f ...
- IIS网页GZIP压缩
1.开GZIP有什么好处? 答:Gzip开启以后会将输出到用户浏览器的数据进行压缩的处理,这样就会减小通过网络传输的数据量,提高浏览的速度. 2.如何启用IIS的Gzip压缩功能: 答:首先,如果你需 ...
- Chrome即将封杀Google Earth、Google Talk等插件
昨日,Chrome安全工程师Justin Schuh在官方博客中写道,到明年一月份,谷歌将封杀一系列基于NPAPI框架标准的浏览器插件.其中包括谷歌地球(Google Earth).Google Ta ...
- java基础第9天
抽象 abstract 抽象类和抽象方法必须用abstract关键字修饰 抽象类格式 abstract class 类名{} 抽象方法定义,在返回值钱,或修饰符前加上abstract关键字 方法没有方 ...
- 【python】venv使用
virtualenvwrapper 比 virualenv 好用一些. 准备 export WORKON_HOME=~/venv source /usr/bin/virtualenvwrapper.s ...
- I.MX6 CAAM
/********************************************************************************* * I.MX6 CAAM * 说明 ...
- ssh命令的使用
ssh命令是用来远程登录服务器的,默认端口号为22. 常用的命令是: ssh 用户名@服务器地址 然后回车,输入登录服务器的密码 登录服务器后要进行上传下载文件,可以用scp命令,命令格式为: s ...
- USB gadget学习笔记
1.usb-OTG-ADP-HNP-SRP https://blog.csdn.net/xiongjiao0610/article/details/44150849