Fault Tolerance —— Storm的故障容错性

——本文讲解了Storm故障容忍性(Fault-Tolerance)的设计细节：当Worker、节点、Nimbus或者Supervisor出现故障时是如何实现故障容忍性，以及Nimbus是否存在单点故障问题。

当一个Worker挂了会怎样？

When a worker dies, the supervisor will restart it. If it continuously fails on startup and is unable to heartbeat to Nimbus, Nimbus will reassign the worker to another machine.

当一个Worker挂了，Supervisor会重启它。如果这个Worker连续在启动时失败，并且无法让Nimbus观察到它的心跳，Nimbus将这个Worker重新分配到另一台机器上。

当一个节点挂了会怎样？

The tasks assigned to that machine will time-out and Nimbus will reassign those tasks to other machines.

分配给这台机器的任务将会超时，并且Nimbus将这些任务重新分配给其它机器。

当Nimbus或者Supervisor daemon进程挂了会怎样？

The Nimbus and Supervisor daemons are designed to be fail-fast (process self-destructs whenever any unexpected situation is encountered) and stateless (all state is kept in Zookeeper or on disk). As described in Setting up a Storm cluster, the Nimbus and Supervisor daemons must be run under supervision using a tool like daemontools or monit. So if the Nimbus or Supervisor daemons die, they restart like nothing happened.

Nimbus和Supervisor daemon进程设计成快速失败(无论何时当遇到任何异常情况，将会执行自毁)和无状态（所有的状态都保存在Zookeeper或者磁盘上）。正如Setting up a Storm cluster中描述的，Nimbus和Supervior daemon进程必须在监控下运行，如使用daemontools或者monit工具。所以如果Nimbus或者Supervisor daemon进程挂了，它可以像什么异常也没有发生似的重新启动。

Most notably, no worker processes are affected by the death of Nimbus or the Supervisors. This is in contrast to Hadoop, where if the JobTracker dies, all the running jobs are lost.

非常重要的是，没有任何Worker进程会因Nimbus或者Supervisor的挂掉而受到影响。这个和Hadoop相反。在Hadoop中如果JobTracker挂了，所有运行的Job将会丢失。

Nimbus是否有单点故障？

If you lose the Nimbus node, the workers will still continue to function. Additionally, supervisors will continue to restart workers if they die. However, without Nimbus, workers won't be reassigned to other machines when necessary (like if you lose a worker machine).

当Nimbus节点出现故障，Worker将依然可以继续工作。此外，Supervisor将可以继续重启挂掉的Worker。然而，没有了Nimbus节点，Worker不能在需要的时候被重新分配到其它的机器。（就像你丢失了一台Woker机器)。

So the answer is that Nimbus is "sort of" a SPOF. In practice, it's not a big deal since nothing catastrophic happens when the Nimbus daemon dies. There are plans to make Nimbus highly available in the future.

所以答案是Nimbus是会有单点故障的问题。在实践中，这个不是大问题。Nimbus deamon进程挂掉不会引起任何灾难发生。在将来，计划将Nimbus设计成高可用。

Storm如何保证数据处理?

Storm提供了一些机制来保证即使在节点挂了或者消息被丢失的情况下也能正确的进行数据处理。可以参考 Guaranteeing message processing 与可靠的消息处理译文。

参考链接：

Storm官方文档：Fault Tolerance

博文：storm的故障容错分析