Hadoop 从节点的 NodeManager 无法启动
一、问题描述
日志文件信息如下：
-- ::, INFO nodemanager.NodeManager (LogAdapter.java:info()) - registered UNIX signal handlers for [TERM, HUP, INT]
-- ::, INFO recovery.NMLeveldbStateStoreService (NMLeveldbStateStoreService.java:openDatabase()) - Using state database at /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state for recovery
-- ::, INFO service.AbstractService (AbstractService.java:noteFailure()) - Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-: No such file or directory
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-: No such file or directory
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:)
-- ::, INFO service.AbstractService (AbstractService.java:noteFailure()) - Service NodeManager failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-: No such file or directory
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-: No such file or directory
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:) .......
二、解决办法一
i. 删除 yarn-nm-state 目录的所有文件再次启动
[root@zwlbs3 ~]# cd /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/
[root@zwlbs3 yarn-nm-state]# ls
.log CURRENT LOCK MANIFEST-
[root@zwlbs3 yarn-nm-state]# rm -rf *
ii. 再次启动 NodeManager 组件
我这里使用的是 Ambari 来管理的，直接就Web界面操作，也可以使用命令操作启动。
iii. 查看是否启动成功，已启动成功
[root@zwlbs3 ~]# jps
DataNode
Jps
HRegionServer
NodeManager
三、解决办法二（如果办法一无效，试试办法二）
i. 如果过一会变成 stop 状态，报错日志跟上面相同，如下：
-- ::, INFO nodemanager.NodeManager (LogAdapter.java:info()) - registered UNIX signal handlers for [TERM, HUP, INT]
-- ::, INFO recovery.NMLeveldbStateStoreService (NMLeveldbStateStoreService.java:openDatabase()) - Using state database at /var/log/hadoop-yarn/nodema
nager/recovery-state/yarn-nm-state for recovery
-- ::, INFO service.AbstractService (AbstractService.java:noteFailure()) - Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateSt
oreService failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/
MANIFEST-: No such file or directory
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-: No such file or directory
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:)
-- ::, INFO service.AbstractService (AbstractService.java:noteFailure()) - Service NodeManager failed in state INITED; cause: org.apache.hadoop.servic
e.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-:
No such file or directory
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm
-state/MANIFEST-: No such file or directory
.......
ii. 删除 yarn-nm-state 目录的所有文件
[root@zwlbs3 ~]# cd /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/
[root@zwlbs3 yarn-nm-state]# ls
.log CURRENT LOCK MANIFEST-
[root@zwlbs3 yarn-nm-state]# rm -rf *
iii. 重启该服务器的整个Hadoop
命令行操作请自行到 Hadoop 目录下重启所有组件即可。
Hadoop 从节点的 NodeManager 无法启动的更多相关文章
- Hadoop 2.x 安装常见问题FAQ（一） NodeManager 无法启动问题解决
一.问题描述 在搭建 Hadoop hadoop-2.4.1 集群的最后一步启动集群,在命令窗口并没有报任何错误,但是Slave 节点的 NodeManager进程始终启动不起来.随后查看了后台启动日 ...
- Hadoop的多节点集群详细启动步骤（3或5节点）
版本1 利用自己写的脚本来启动,见如下博客 hadoop-2.6.0-cdh5.4.5.tar.gz(CDH)的3节点集群搭建 hadoop-2.6.0.tar.gz的集群搭建(3节点) hadoop ...
- Hadoop Datanode节点无法启动（All directories in dfs.data.dir are invalid）
Hadoop Datanode节点无法启动(All directories in dfs.data.dir are invalid) java.io.IOException: All director ...
- nodemanager 无法启动报错“doesn't satisfy minimum allocations”
在启动nodemanager节点nodemanager时候报错 2019-03-29 14:15:44,648 INFO org.apache.hadoop.yarn.server.nodemanag ...
- HBase的多节点集群详细启动步骤（3或5节点）（分为Zookeeper自带还是外装）
HBase的多节点集群详细启动步骤(3或5节点)分为: 1.HBASE_MANAGES_ZK的默认值是false(zookeeper外装)(推荐) 2.HBASE_MANAGES_ZK的默认值是tru ...
- Hadoop多节点Cluster
Hadoop多节点集群规划 服务起名称 内网IP HDFS YARN master 192.168.1.155 NameNode ResourceManager slave1 192.168.1.11 ...
- 实验-hadoop新增节点
关于hadoop新增节点网上的说法都有些差别,自己来实践一把 1.建立一个namenode一个datanode的集群 master:192.168.126.130 slave1:192.168.126 ...
- hadoop集群之Datenode无法启动解决办法
hadoop集群之Datenode无法启动解决办法 我们在启动hadoop集群的时候,通过jps查看进程,发现namenode RM和Secondary NameNode都有,但datanode没有启 ...
- Hadoop添加节点datanode(生产环境)
Hadoop添加节点datanode 博客分类: hadoop HadoopSSHJDKXML工作 1.部署hadoop 和普通的datanode一样.安装jdk,ssh 2.修改host ...
随机推荐
- 在MVC视图中将数字转换为string类型后保留两位小数
<td>@item.recharge_reward_rate.ToString("F2")%</td> @*保留小数两位*@ <td>@item ...
- [Go] gocron源码阅读-go语言的结构体
结构体类型 type 名字 struct{},下面这段是github.com/urfave/cli包里的代码,声明了一个App的结构体类型 type App struct { // The name ...
- 简单的shell脚本
1.1每隔一秒向屏幕输出一个数字,并且每次加1. #/bin/bashfor((i=1;i<=100;i++)) do echo -en " $i\n"; sleep 1;d ...
- centos 的系统管理命令 service systemctl
centos 的 systemctl 命令 systemctl is-enabled *.service #查询服务是否开机启动 systemctl enable *.service # ...
- drf扩展知识点总结视图
- [C2W2] Improving Deep Neural Networks : Optimization algorithms
第二周:优化算法(Optimization algorithms) Mini-batch 梯度下降(Mini-batch gradient descent) 本周将学习优化算法,这能让你的神经网络运行 ...
- leetcode279. 完全平方数
给定正整数 n,找到若干个完全平方数(比如 1, 4, 9, 16, ...)使得它们的和等于 n.你需要让组成和的完全平方数的个数最少. 示例 1: 输入: n = 12输出: 3 解释: 12 = ...
- 使用Python拆分数据量大的CSV文件（亲测有效）
转载:https://www.cnblogs.com/FYZHANG/p/11629075.html 一次就运行成功了,感谢博主分享 #!/usr/bin/env python3 # -*- # @F ...
- ESA2GJK1DH1K升级篇: 阿里云物联网平台 OTA: 关于阿里云物联网平台 OTA 的升级流程
前言 鉴于有些用户直接想使用现成的物联网平台实现 OTA 远程升级 我就写一写这系列的文章 注意:首先大家必须把我自建服务器是如何实现的看明白! 我看了下阿里云提供的,实际上流程和咱自建实现的差别不大 ...
- stm32串口烧写程序到开发板
前言:我们如果要断电后重启板子还能使用程序的话就需要通过串口把程序烧写到开发板中. 1.设置Keil编译完之后,生成Hex文件.在工程选项[output]的标签,勾选“Create HEX File” ...