Hadoop 从节点的 NodeManager 无法启动

一、问题描述

日志文件信息如下：

-- ::, INFO  nodemanager.NodeManager (LogAdapter.java:info()) - registered UNIX signal handlers for [TERM, HUP, INT]
-- ::, INFO  recovery.NMLeveldbStateStoreService (NMLeveldbStateStoreService.java:openDatabase()) - Using state database at /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state for recovery
-- ::, INFO  service.AbstractService (AbstractService.java:noteFailure()) - Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-: No such file or directory
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-: No such file or directory
    at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:)
    at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:)
    at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:)
    at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:)
    at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:)
    at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:)
-- ::, INFO  service.AbstractService (AbstractService.java:noteFailure()) - Service NodeManager failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-: No such file or directory
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-: No such file or directory
    at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:)

.......

二、解决办法一

i. 删除 yarn-nm-state 目录的所有文件再次启动

[root@zwlbs3 ~]# cd /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/
[root@zwlbs3 yarn-nm-state]# ls
.log  CURRENT  LOCK  MANIFEST-
[root@zwlbs3 yarn-nm-state]# rm -rf *

ii. 再次启动 NodeManager 组件

我这里使用的是 Ambari 来管理的，直接就Web界面操作，也可以使用命令操作启动。

iii. 查看是否启动成功，已启动成功

[root@zwlbs3 ~]# jps
 DataNode
 Jps
 HRegionServer
 NodeManager

三、解决办法二（如果办法一无效，试试办法二）

i. 如果过一会变成 stop 状态，报错日志跟上面相同，如下：

-- ::, INFO  nodemanager.NodeManager (LogAdapter.java:info()) - registered UNIX signal handlers for [TERM, HUP, INT]
-- ::, INFO  recovery.NMLeveldbStateStoreService (NMLeveldbStateStoreService.java:openDatabase()) - Using state database at /var/log/hadoop-yarn/nodema
nager/recovery-state/yarn-nm-state for recovery
-- ::, INFO  service.AbstractService (AbstractService.java:noteFailure()) - Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateSt
oreService failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/
MANIFEST-: No such file or directory
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-: No such file or directory
        at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:)
        at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:)
        at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:)
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:)
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:)
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:)
-- ::, INFO  service.AbstractService (AbstractService.java:noteFailure()) - Service NodeManager failed in state INITED; cause: org.apache.hadoop.servic
e.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/MANIFEST-:
No such file or directory
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm
-state/MANIFEST-: No such file or directory
.......

ii. 删除 yarn-nm-state 目录的所有文件

[root@zwlbs3 ~]# cd /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/
[root@zwlbs3 yarn-nm-state]# ls
.log  CURRENT  LOCK  MANIFEST-
[root@zwlbs3 yarn-nm-state]# rm -rf *

iii. 重启该服务器的整个Hadoop

命令行操作请自行到 Hadoop 目录下重启所有组件即可。