关于java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.的问题
报错如下:
300 [main] DEBUG org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set
解决办法一:
根据 http://blog.csdn.net/baidu_19473529/article/details/54693523 配置hadoop_home变量
下载winutils地址https://github.com/srccodes/hadoop-common-2.2.0-bin下载解压
重新打开Eclipse,运行程序,可以解决内网的Hadoop连接问题。(亲测试通过)
解决办法二:
在java程序中加入
System.setProperty("hadoop.home.dir", "/usr/local/hadoop-2.6.0");
依然报错(未亲测试)
解决办法三:(不是同一问题?)
根据该提示:
依然报错
--------------------------------5.19.早晨------------------------------------------
今天早上再次执行该程序,将上传到HDFS部分的代码给注释掉后发现日志报错信息如下:
132 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
134 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
1)check Hadoop library:
root@Ubuntu-1:/usr/local/hadoop-2.6.0/lib/native# file libhadoop.so.1.0.
It's 64-bite library
2)Try adding the HADOOP_OPTS
environment variable:
root@Ubuntu-1:~# vi /etc/profile //environment variable
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native/"
It doesn't work, and reports the same error.
3)Try adding the HADOOP_OPTS
and HADOOP_COMMON_LIB_NATIVE_DIR
environment variable:
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/"
It still doesn't work, and reports the same error.
4)Adding the Hadoop
library into LD_LIBRARY_PATH
root@Ubuntu-1:~# vi .bashrc
export LD_LIBRARY_PATH=/usr/local/hadoop/lib/native/:$LD_LIBRARY_PATH
It doesn't work, and reports the same error.
5)append word native to HADOOP_OPTS
like this
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native"
It doesn't work, and reports the same error.
-----------------------------------5.20早晨-------------------------
win下依旧无法使用java代码上传文件到HDFS
----------------------------------5.24------------
copy了一份hadoop-2.6.0文件到本机,更改bin目录和path ,显示无效的path,遂更改回去
---------------------2017.5.26-------------------
今天早上开机 试了一下 发现它已经好了 其实上述的设置已经可以解决大部分遇到这类问题的人群
关于java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.的问题的更多相关文章
- windows 中使用hbase 异常:java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
平时一般是在windows环境下进行开发,在windows 环境下操作hbase可能会出现异常(java.io.IOException: Could not locate executable nul ...
- 排查Hive报错:org.apache.hadoop.hive.serde2.SerDeException: java.io.IOException: Start of Array expected
CREATE TABLE json_nested_test ( count string, usage string, pkg map<string,string>, languages ...
- java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries
在已经搭建好的集群环境Centos6.6+Hadoop2.7+Hbase0.98+Spark1.3.1下,在Win7系统Intellij开发工具中调试Spark读取Hbase.运行直接报错: ? 1 ...
- hive运行query语句时提示错误:org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException:
hive> select product_id, track_time from trackinfo limit 5; Total MapReduce jobs = 1 Launching Jo ...
- Spark- ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
运行 mport org.apache.log4j.{Level, Logger} import org.apache.spark.rdd.RDD import org.apache.spark.{S ...
- spark开发常见问题之一:java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
最近在学习研究pyspark机器学习算法,执行代码出现以下异常: 19/06/29 10:08:26 ERROR Shell: Failed to locate the winutils binary ...
- idea 提示:ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException解决方法
Windows系统中的IDEA链接Linux里面的Hadoop的api时出现的问题 提示:ERROR util.Shell: Failed to locate the winutils binary ...
- org.apache.hadoop.ipc.RemoteException(java.io.IOException)
昨晚突然之间mr跑步起来了 jps查看 进程都在的,但是在reduce任务跑了85%的时候会抛异常 异常情况如下: 2016-09-21 21:32:28,538 INFO [org.apache.h ...
- 运行基准测试hadoop集群中的问题:org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /benchmarks/TestDFSIO/io_data/test_
在master(即:host2)中执行 hadoop jar hadoop-test-1.1.2.jar DFSCIOTest -write -nrFiles 12 -fileSize 10240 - ...
随机推荐
- (转)关闭win10的Skype
https://blog.csdn.net/qq_38285661/article/details/86663849 使用win10的小伙伴们,有没有发现一个不用的功能Skype,假如你想卸载又怕卸不 ...
- unity 数学公式
Mathf.Abs绝对值 计算并返回指定参数 f 绝对值. Mathf.Acos反余弦 static function Acos (f : float) : float 以弧度为单位计算并返回参数 f ...
- 吴裕雄 oracle 存储过程
- js高级-闭包
function foo(x){ var tmp = 3; return function(y){ //把一个函数作为返回值,定义时候的作用域 console.log(x+y+(++tmp)) //+ ...
- com.android.dx.command.Main with arguments
Error:Execution failed for task ':jingyeyun:transformClassesWithDexForDebug'.> com.android.build. ...
- Python全栈开发 列表, 元组 数据类型知识运用及操作 range知识
一.列表 1.什么是列表? 列表是一个可变类型,由 [ ] 表示,每一项元素用逗号隔开.列表能够装大量的数据,可以装对象的对象. 2.列表的索引和切片. 列表和字符串一样,也有索引和切片.只不过列表 ...
- 十 suprocess模块
1 import subprocess 2 3 ''' 4 sh-3.2# ls /Users/egon/Desktop |grep txt$ 5 mysql.txt 6 tt.txt 7 事物.tx ...
- centos 7 redis-4.0.11 哨兵
redis-master:192.168.199.223 redis-slave_1: 192.168.199.224 redis-slave_2: 192.168.199.252 redis-mas ...
- css3修改滚动条样式
/*滚动条整体样式*/ /*高宽分别对应横竖滚动条的尺寸*/ .content-box::-webkit-scrollbar{ width: 4px; height: 4px; } /*滚动条里面小方 ...
- GitHub优秀项目
https://blog.csdn.net/javaxuexi123/article/details/79248124