hadoop操作hdfs错误
本文转自:http://www.aboutyun.com/blog-61-22.html
当我们对hdfs操作的时候,我们可能会碰到如下错误
错误1:权限问题
Exception in thread "main" org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=,
access=WRITE, inode="":root:supergroup:rwxr-xr-x
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1216)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:321)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1126)
at hdfs.hdoopapi.main(hdoopapi.java:19)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hyj, access=WRITE,
inode="":root:supergroup:rwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:199)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:180)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:128)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5214)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5188)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:2060)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2029)
at org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:817)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
at org.apache.hadoop.ipc.Client.call(Client.java:1070)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at sun.proxy.$Proxy1.mkdirs(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at sun.proxy.$Proxy1.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1214)
... 3 more
回答:产生这个错误的原因,是因为HDFS配置文件hdfs-site.xml中,配置了对权限的检查,我们需要把dfs.permissions,value值改为false。
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/work/hadoop_tmp</value>
</property>
<property>
<name>dfs.permissions</name>
<value>true</value>
</property>
</configuration>
还存在另外一种情况:如下,就是我们对其配置,这个也会产生这个错误。
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/work/hadoop_tmp</value>
</property>
</configuration>
错误2:端口问题
14/02/24 14:10:45 INFO ipc.Client: Retrying connect to server: aboutyun/192.168.159.10:9000. Already tried 0 time(s).
14/02/24 14:10:47 INFO ipc.Client: Retrying connect to server: aboutyun/192.168.159.10:9000. Already tried 1 time(s).
14/02/24 14:10:49 INFO ipc.Client: Retrying connect to server: aboutyun/192.168.159.10:9000. Already tried 2 time(s).
14/02/24 14:10:51 INFO ipc.Client: Retrying connect to server: aboutyun/192.168.159.10:9000. Already tried 3 time(s).
14/02/24 14:10:53 INFO ipc.Client: Retrying connect to server: aboutyun/192.168.159.10:9000. Already tried 4 time(s).
14/02/24 14:10:55 INFO ipc.Client: Retrying connect to server: aboutyun/192.168.159.10:9000. Already tried 5 time(s).
14/02/24 14:10:57 INFO ipc.Client: Retrying connect to server: aboutyun/192.168.159.10:9000. Already tried 6 time(s).
14/02/24 14:10:59 INFO ipc.Client: Retrying connect to server: aboutyun/192.168.159.10:9000. Already tried 7 time(s).
14/02/24 14:11:01 INFO ipc.Client: Retrying connect to server: aboutyun/192.168.159.10:9000. Already tried 8 time(s).
14/02/24 14:11:03 INFO ipc.Client: Retrying connect to server: aboutyun/192.168.159.10:9000. Already tried 9 time(s).
Exception in thread "main" java.net.ConnectException: Call to aboutyun/192.168.159.10:9000 failed on connection exception: java.net.ConnectException: Connection
refused: no further information
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1099)
at org.apache.hadoop.ipc.Client.call(Client.java:1075)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at hdfs.hdoopapi.getFileSystem(hdoopapi.java:29)
at hdfs.hdoopapi.main(hdoopapi.java:17)
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)
at org.apache.hadoop.ipc.Client.call(Client.java:1050)
... 14 more
上面是的也是错误,看到很多说是修改配置文件,其实发生这个问题的原因
1.你们之间通信确实产生了问题
2.看一下集群是否已经启动(这个很重要)
试一下start-all.sh或许这个就能解决你的问题。
错误3.Name node处于安全模式:
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /dl. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:2055)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2029)
at org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:817)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
at org.apache.hadoop.ipc.Client.call(Client.java:1070)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at sun.proxy.$Proxy1.mkdirs(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at sun.proxy.$Proxy1.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1214)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:321)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1126)
at hdfs.hdoopapi.main(hdoopapi.java:21)
hadoop操作hdfs错误的更多相关文章
- Hadoop操作hdfs的命令【转载】
本文系转载,原文地址被黑了,故无法贴出原始链接. Hadoop操作HDFS命令如下所示: hadoop fs 查看Hadoop HDFS支持的所有命令 hadoop fs –ls 列出目录及文件信息 ...
- hadoop学习(三)HDFS常用命令以及java操作HDFS
一.HDFS的常用命令 1.查看根目录下的信息:./hadoop dfs -ls 2.查看根目录下的in目录中的内容:./hadoop dfs -ls in或者./hadoop dfs -ls ./i ...
- Hadoop之HDFS文件操作常有两种方式(转载)
摘要:Hadoop之HDFS文件操作常有两种方式,命令行方式和JavaAPI方式.本文介绍如何利用这两种方式对HDFS文件进行操作. 关键词:HDFS文件 命令行 Java API HD ...
- hadoop执行hdfs文件到hbase表插入操作(xjl456852原创)
本例中需要将hdfs上的文本文件,解析后插入到hbase的表中. 本例用到的hadoop版本2.7.2 hbase版本1.2.2 hbase的表如下: create 'ns2:user', 'info ...
- Hadoop Java API操作HDFS文件系统(Mac)
1.下载Hadoop的压缩包 tar.gz https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable/ 2.关联jar包 在 ...
- Hadoop基础-通过IO流操作HDFS
Hadoop基础-通过IO流操作HDFS 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.上传文件 /* @author :yinzhengjie Blog:http://www ...
- hadoop的hdfs中的javaAPI操作
package cn.itcast.bigdata.hdfs; import java.net.URI; import java.util.Iterator; import java.util.Map ...
- Hadoop基础-HDFS的API常见操作
Hadoop基础-HDFS的API常见操作 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本文主要是记录一写我在学习HDFS时的一些琐碎的学习笔记, 方便自己以后查看.在调用API ...
- Hadoop JAVA HDFS客户端操作
JAVA HDFS客户端操作 通过API操作HDFS org.apache.logging.log4jlog4j-core2.8.2org.apache.hadoophadoop-common${ha ...
随机推荐
- Tomcat 中文乱码
问题描述 tomcat9启动后会有中文乱码,比如控制台乱码: startup.bat启动时乱码: 解决方法 打开"/apache-tomcat-9.0.20/conf/logging.pro ...
- GO学习-(7) Go语言基础之流程控制
流程控制是每种编程语言控制逻辑走向和执行次序的重要部分,流程控制可以说是一门语言的"经脉". Go语言中最常用的流程控制有if和for,而switch和goto主要是为了简化代码. ...
- GO语言面向对象01---封装属性与创建对象的方法与工厂模式
package main import "fmt" /* 面向过程编程:调度大量的变量.函数 ---------- 面向对象编程(OOP=Object Oriented Progr ...
- H.265 HD 和H.265 4K Video Encoder IP Core
H.265 HD 和H.265 4K Video Encoder IP Core H.265 HD Video Encoder IP Core(H.265 HD Video/Audio Encoder ...
- deepstream-开放式实时服务器
deepstream-开放式实时服务器 deepstream是一款开源服务器,其灵感来自金融交易技术背后的概念.它允许客户端和后端服务同步数据.发送消息并以非常高的速度和规模规划rpc. 参考:htt ...
- Thumb扩展
Thumb扩展 ARM处理器已在嵌入式系统,手持计算机和其系统中得到了最大的应用,在这些系统中,系统对使用有限资源进行大量工作的要求很高.Thumb扩展是为解决资源消耗中的某些方面而创建的,已成为当今 ...
- java并发编程工具类JUC第七篇:BlockingDeque双端阻塞队列
在之前的文章中已经为大家介绍了java并发编程的工具:BlockingQueue接口.ArrayBlockingQueue.DelayQueue.LinkedBlockingQueue.Priorit ...
- vscode使用版本控制git commit unstaged时提示对话框的设置
使用 vscode 版本控制提交代码时,如果有 unstaged file,会有一个弹出框: 选择 always 或者 never ,这个框下次就不再弹出了. 如果你想让他再次出现,请去setting ...
- Spring Cloud09: Config 配置中心
一.概述 什么是配置中心呢,在基于微服务的分布式系统中,每个业务模块都可以拆分成独立自主的服务,由多个请求来协助完成某个需求,那么在某一具体的业务场景中,某一个请求需要调用多个服务来完成,那么就存在一 ...
- Java协程实践指南(一)
一. 协程产生的背景 说起协程,大多数人的第一印象可能就是GoLang,这也是Go语言非常吸引人的地方之一,它内建的并发支持.Go语言并发体系的理论是C.A.R Hoare在1978年提出的CSP(C ...