因磁盘空间不足导致HDFS的NameNode进入安全模式问题记录
因磁盘空间不足导致HDFS的NameNode进入安全模式问题记录,调用API上传及下载文件时报如下错误信息:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot delete /null/daxt/0265803074a54d54b8ec0b91df359bdf.log. Name node is in safe mode.
Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE: If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3713)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:953)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:611)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy51.delete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:540)
at sun.reflect.GeneratedMethodAccessor5314.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy52.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2044)
at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707)
at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:703)
at com.pub.file.business.HDFSManager.delete(HDFSManager.java:295)
at com.pub.file.business.PubFileManager.deleteHdfsFile(PubFileManager.java:1284)
at com.pub.file.business.PubFileManager$$FastClassByCGLIB$$6d05010e.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:692)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:625)
at com.framework.impl.PubFileServiceImpl$$EnhancerByCGLIB$$c63330d3.deleteHdfsFile(<generated>)
at com.pub.file.action.UniversalPubFileAction.deleteHdfsFile(UniversalPubFileAction.java:249)
at sun.reflect.GeneratedMethodAccessor5319.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.common.crud.EntityAction.execute(EntityAction.java:104)
at sun.reflect.GeneratedMethodAccessor5291.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:892)
at ognl.OgnlRuntime.callAppropriateMethod(OgnlRuntime.java:1294)
at ognl.ObjectMethodAccessor.callMethod(ObjectMethodAccessor.java:68)
at com.opensymphony.xwork2.ognl.accessor.XWorkMethodAccessor.callMethodWithDebugInfo(XWorkMethodAccessor.java:117)
at com.opensymphony.xwork2.ognl.accessor.XWorkMethodAccessor.callMethod(XWorkMethodAccessor.java:108)
at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1370)
at ognl.ASTMethod.getValueBody(ASTMethod.java:91)
at ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:212)
at ognl.SimpleNode.getValue(SimpleNode.java:258)
at ognl.Ognl.getValue(Ognl.java:467)
at ognl.Ognl.getValue(Ognl.java:431)
at com.opensymphony.xwork2.ognl.OgnlUtil$3.execute(OgnlUtil.java:352)
at com.opensymphony.xwork2.ognl.OgnlUtil.compileAndExecuteMethod(OgnlUtil.java:404)
at com.opensymphony.xwork2.ognl.OgnlUtil.callMethod(OgnlUtil.java:350)
at com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:430)
at com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:290)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:251)
at org.apache.struts2.interceptor.DeprecationInterceptor.intercept(DeprecationInterceptor.java:41)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:256)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:168)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:265)
at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:76)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:191)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:125)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:253)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:100)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:141)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:145)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:140)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:193)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:54)
at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:575)
at org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:81)
at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:99)
at com.common.SysFilterDispatcher.doFilter(SysFilterDispatcher.java:162)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.common.util.CodeTransferFilter.doFilter(CodeTransferFilter.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1137)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
遇到此种情况通过df -h命令查看磁盘空间使用情况,如果确实空间紧缺,清理相关文件释放磁盘空间。
清理完成之后通过如下命令恢复服务:
hdfs dfsadmin -safemode leave
因磁盘空间不足导致HDFS的NameNode进入安全模式问题记录的更多相关文章
- oracle所在磁盘空间不足导致了数据库异常
oracle所在磁盘空间不足导致了数据库异常.需要减小数据文件的大小来解决. 1.检查数据文件的名称和编号 select file#,name from v$datafile; 2.看哪个数据文件所占 ...
- Oracle 不小心删除undo数据文件以及磁盘空间不足导致不能登录的解决办法
在一次测试中,由于导入的数据量过大导致事务一直提交失败因为磁盘空间不够用了,一检查发现是undo表空间不够用,于是重新创建了一个表空间,准备把之前的undo表空间删除,删除时却发现一直删不掉,因为它一 ...
- 阿里云rds 磁盘空间满导致实例锁定
1.RDS 数据日志已经快满了, 导致数据库不能写入,只读. 2. Binlog日志的保存及清理规则 MySQL实例的空间内默认清理binlog日志的规则如下: 实例空间内默认会保存最近18个小时内的 ...
- tokudb引擎磁盘空间不足导致写入失败的调查
故障现象 2016.1.1号早上4点左右,zabbi数据库服务器报警,写入数据失败.登陆机器后检查发现磁盘空间使用95%没有用满,进去zabbix数据库,执行insert命令提示错误“errir 10 ...
- DG备库磁盘空间满导致无法创建归档
上周五去某客户那里做数据库巡检.是window 2008系统上10g的一套NC系统的库,已经配置了DG,可是巡检时发现数据库报错: Tue Nov 11 10:13:57 2014 LNS: Stan ...
- 解决因block的损坏而导致hdfs启动后进入安全模式
问题描述: 以单机伪分布式方式安装了hadoop2.7.1,并在该机器上安装了hive1.2.1.首先执行 sbin/start-dfs.sh 启动hdfs服务,然后执行hive,这时hive脚本启动 ...
- Hadoop源码学习笔记之NameNode启动场景流程五:磁盘空间检查及安全模式检查
本篇内容关注NameNode启动之前,active状态和standby状态的一些后台服务及准备工作,即源码里的CommonServices.主要包括磁盘空间检查. 可用资源检查.安全模式等.依然分为三 ...
- Linux下几个与磁盘空间和文件尺寸相关的命令
大家好,我是良许. 硬盘是计算机非常重要的一个部件,不管是代码,还是 UI .声音.文档,抑或是没人时偷偷看的小视频,都需要保存在硬盘里. 对于很多 Linux 服务器,会进行很多的编译操作.而编译操 ...
- 查找Linux系统中的占用磁盘空间
目录的来查看空间占用情况 du -sh /* 先看看根目录下面 让文件夹下的文件让文件按大小排序 方法一:# ls -lhSl 长格式显示,h human readable模式,大小单位为M,G等易读 ...
随机推荐
- SQL 和 NoSQL 比较
定义: SQL (Structured Query Language) 数据库,指关系型数据库.主要代表:SQL Server,Oracle,MySQL(开源),PostgreSQL(开源). NoS ...
- HDU 2222 Keywords Search(瞎搞)
Keywords Search Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 131072/131072 K (Java/Others ...
- python大法好 vijos1375 大整数
一个k(1<=k<=80)位的十进制正整数N,就是所谓的大整数.请你设计程序,对于给出的某一个大整数N,找到满足p^3+p^2+3p<=n的p的最大值. 输入格式 输入数据只有一行, ...
- 常见的页面效果,相关的js代码
1.焦点图 $(document).ready(function(){ var i=0; var autoChange= setInterval(function(){ if(i<$(" ...
- 关于树论【动态树问题(LCT)】
搬运:看一道caioj1439 题目描述 一开始给你一棵n个点n-1条边的树,每个点有一个权值wi. 三种操作: op=1 u v :在点u和点v之间建一条边. op=2 u v:摧毁点u到点v之间的 ...
- HDU4283 You Are the One —— 区间DP
题目链接:https://vjudge.net/problem/HDU-4283 You Are the One Time Limit: 2000/1000 MS (Java/Others) M ...
- codeforces A. Kitahara Haruki's Gift 解题报告
题目链接:http://codeforces.com/problemset/problem/433/A 题目意思:给定 n 个只由100和200组成的数,问能不能分成均等的两份. 题目其实不难,要考虑 ...
- RPi 2B DDNS 动态域名
/**************************************************************************** * RPi 2B DDNS 动态域名 * 说 ...
- dos窗口出现error:could not open ...jvm.cfg解决方法
在cmd程序中,运行javac -version查看jdk是多少位时出现错误 error:could not open ...jvm.cfg解决方法 出现这种情况大多是因为电脑上之前安装过JDK,卸载 ...
- [yii2]Module的Namespace和控制器位置
namespace和目录对应,否则无法找到控制器类,module文件在根路径 使用gii生成Module为\app\admin,那么 namespace app; class admin extend ...