Namenode写Journalnode超时，导致Namenode挂掉的问题

昨天还好好的集群，今天早上来看又挂掉了，还好是家里的测试服务器集群。。。

首先，查看了Namenode的状态，发现两台Namenode只剩下一台了，赶紧到挂了的那台去查看了logs下的日志：

-- ::, INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after  milliseconds
-- ::, INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited  ms (timeout= ms) for a response for sendEdits. Succeeded so far: [10.80.248.17:]
-- ::, FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: flush failed for required journal (JournalAndStream(mgr=QJM to [10.80.248.17:, 10.80.248.18:, 10.80.248.19:], stream=QuorumOutputStream starting at txid ))
java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond.
    at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:)
    at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:)
    at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:)
    at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$.apply(JournalSet.java:)
    at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:)
    at org.apache.hadoop.hdfs.server.namenode.JournalSet.access$(JournalSet.java:)
    at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:)
    at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:)
    at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$.callBlockingMethod(NamenodeProtocolProtos.java:)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)
    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)
-- ::, WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Aborting QuorumOutputStream starting at txid
-- ::, INFO org.apache.hadoop.util.ExitUtil: Exiting with status
-- ::, INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ut07/10.80.248.17
************************************************************/

以上是hadoop-hadooptest-namenode-ut07.log在Namenode退出时候的关键日志，可以从中发现，Namenode在写Journalnode发生了超时，默认的超时时间为20秒，而在超时发生后，Namenode会触发ExitUtil类的terminate 方法，导致进程的System.exit()

至于为什么好好的集群，会发生写入超时的问题呢，这个问题现在还不太确定，有人说是因为执行了较大文件的HDFS操作，导致Namenode的FULL GC时间较长，所以导致写Journalnode超时。但是我昨天到今天基本上没有去动HDFS，这个根本原因还得再深挖一下。。。

无论如何，先把集群恢复吧，还要用呢。

其实在实际的生产环境中，也很容易发生类似的这种超时情况，所以我们需要把默认的20s超时改成更大的值，比如60s。

我们可以在hadoop/etc/hadoop下的hdfs-site.xml中，加入一组配置:

<property>
        <name>dfs.qjournal.write-txns.timeout.ms</name>
        <value>60000</value>
</property>

这也是我从别人博客中看到的配置方法，神奇的是，我在hadoop的官网中的关于hdfs-site.xml介绍中，居然找不到关于这个配置的说明。。。

http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

最后记得重启整个集群，这样配置才能生效。

友情提示：使用了Flume的同学，记得也要重启Flume集群哦~