Akka源码分析-Persistence

　　在学习akka过程中，我们了解了它的监督机制，会发现actor非常可靠，可以自动的恢复。但akka框架只会简单的创建新的actor，然后调用对应的生命周期函数，如果actor有状态需要回复，我们需要hook对应的生命周期函数，自己恢复状态。但此时恢复的只是初始状态，如果actor在接收消息过程中，状态发生了变化，为了保证可恢复就需要我们自行把状态保存在第三方组件了。考虑到通用性以及Actor模型的特点，akka提供了持久化机制，可以帮助我们做状态恢复。

　　其实，actor的状态千差万别，怎么来统一恢复模型呢？比如有些场景下，actor的状态是一个整数，有些则是负责的类型。akka Persistence并没有直接保存actor的状态，而是另辟蹊径，保存了actor接收到的消息，在actor重启时，把接收到的所有消息重新发送给actor，由actor自己决定如何构建最新的状态。这其实就是事件溯源的典型应用，结合Actor模型，简直是完美。

　　在分析akka Persistence源码之前，我们先来介绍一下这个框架涉及到的几个重要概念。

　　PersistentActor。是一个持久化、有状态actor的trait。它可以持久化消息，并在actor重启时重放消息，以达到状态恢复的目的。

　　AtLeastOnceDelivery。至少一次机制。也就是至少发送一次消息给目标，没发成功咋办？当然是重发了啊。

　　AsyncWriteJournal。异步写日志。journal把发送给持久化actor的消息按照顺序存储。当然了，持久化actor是可以自定义需要存储的消息范围的。

　　Snapshot store。快照存储。如果actor收到的消息太多，在恢复的时候从头开始回放，会导致恢复时间过长。快照其实就是当前状态的截面，在恢复的时候，我们可以从最新的快照开始恢复，从而大大减少恢复的时间。

　　Event sourcing。事件溯源。akka对事件溯源应用开发提供了一个抽象，事件溯源提供了一种物化领域模型数据的持久化方式，它不直接存储最终状态，而是存储导致状态变化的事件，通过这些事件构建最终的状态。比如不直接记录账户的最终金额，而是记录账户的借贷记录（也就是+、-记录），通过这些借贷记录，计算最终的账户余额。

case class ExampleState(events: List[String] = Nil) {
  def updated(evt: Evt): ExampleState = copy(evt.data :: events)
  def size: Int = events.length
  override def toString: String = events.reverse.toString
}

class ExamplePersistentActor extends PersistentActor {
  override def persistenceId = "sample-id-1"

  var state = ExampleState()

  def updateState(event: Evt): Unit =
    state = state.updated(event)

  def numEvents =
    state.size

  val receiveRecover: Receive = {
    case evt: Evt                                 ⇒ updateState(evt)
    case SnapshotOffer(_, snapshot: ExampleState) ⇒ state = snapshot
  }

  val snapShotInterval = 1000
  val receiveCommand: Receive = {
    case Cmd(data) ⇒
      persist(Evt(s"${data}-${numEvents}")) { event ⇒
        updateState(event)
        context.system.eventStream.publish(event)
        if (lastSequenceNr % snapShotInterval == 0 && lastSequenceNr != 0)
          saveSnapshot(state)
      }
    case "print" ⇒ println(state)
  }

}

　　上面是官网提供的一个持久化actor的demo。可以看到，非常简单，只需要继承PersistentActor，然后就可以调用persist持久化事件了，持久化成功之后可以进行响应的业务操作。actor重启的时候，就会把持久化的事件，发送给receiveRecover函数，进行状态恢复。那PersistentActor具体提供了哪些功能呢？又是如何实现的呢？

　　上面是PersistentActor这个trait的定义，大概有三类功能：恢复、持久化（同步、异步、同步批量、异步批量）、推迟（同步、异步）。当然这里说的同步、异步是针对持久化actor本身的，在调用persist时候，actor还是可以接收消息的，只不过会暂时被stash（可以理解成缓存），等最后一条消息持久化之后再开始处理后面的消息。

/**
   * Asynchronously persists `event`. On successful persistence, `handler` is called with the
   * persisted event. It is guaranteed that no new commands will be received by a persistent actor
   * between a call to `persist` and the execution of its `handler`. This also holds for
   * multiple `persist` calls per received command. Internally, this is achieved by stashing new
   * commands and unstashing them when the `event` has been persisted and handled. The stash used
   * for that is an internal stash which doesn't interfere with the inherited user stash.
   *
   * An event `handler` may close over persistent actor state and modify it. The `sender` of a persisted
   * event is the sender of the corresponding command. This means that one can reply to a command
   * sender within an event `handler`.
   *
   * Within an event handler, applications usually update persistent actor state using persisted event
   * data, notify listeners and reply to command senders.
   *
   * If persistence of an event fails, [[#onPersistFailure]] will be invoked and the actor will
   * unconditionally be stopped. The reason that it cannot resume when persist fails is that it
   * is unknown if the event was actually persisted or not, and therefore it is in an inconsistent
   * state. Restarting on persistent failures will most likely fail anyway, since the journal
   * is probably unavailable. It is better to stop the actor and after a back-off timeout start
   * it again.
   *
   * @param event event to be persisted
   * @param handler handler for each persisted `event`
   */
  def persist[A](event: A)(handler: A ⇒ Unit): Unit = {
    internalPersist(event)(handler)
  }

　　官方注释对persist方法解释的特别清楚，这个方法会异步的持久化消息，持久化成功之后会调用handler这个函数。在persist执行与handler调用之间，可以保证没有新的命令接收。持久化机制内部有一个stash机制，当然了，这不是继承的stash。在handler这个代码块中，sender还是原来命令的发送者，并不会因为异步的原因导致失效。为啥？接收到的所有命令都被stash了，你说为啥。

@InternalApi
  final private[akka] def internalPersist[A](event: A)(handler: A ⇒ Unit): Unit = {
    if (recoveryRunning) throw new IllegalStateException("Cannot persist during replay. Events can be persisted when receiving RecoveryCompleted or later.")
    pendingStashingPersistInvocations += 1
    pendingInvocations addLast StashingHandlerInvocation(event, handler.asInstanceOf[Any ⇒ Unit])
    eventBatch ::= AtomicWrite(PersistentRepr(event, persistenceId = persistenceId,
      sequenceNr = nextSequenceNr(), writerUuid = writerUuid, sender = sender()))
  }

　　internalPersist创建StashingHandlerInvocation把它加入到pendingInvocations这列表尾部，然后又创建AtomicWrite添加到eventBatch这个list头部。从StashingHandlerInvocation这个命令来看，这是一个挂起的handler调用，也就是说会在合适的时间调用。AtomicWrite代表原子写入，表示事件要么全写入成功、要么全都写入失败，不存在中间状态，当然了，这在批量时比较有用，有点事务的概念，但并不是所有的持久化机制都支持。persist好像只是操作了几个list，那么handler什么时候会被调用呢？

@InternalApi
  final private[akka] def internalPersistAll[A](events: immutable.Seq[A])(handler: A ⇒ Unit): Unit = {
    if (recoveryRunning) throw new IllegalStateException("Cannot persist during replay. Events can be persisted when receiving RecoveryCompleted or later.")
    if (events.nonEmpty) {
      events.foreach { event ⇒
        pendingStashingPersistInvocations += 1
        pendingInvocations addLast StashingHandlerInvocation(event, handler.asInstanceOf[Any ⇒ Unit])
      }
      eventBatch ::= AtomicWrite(events.map(PersistentRepr.apply(_, persistenceId = persistenceId,
        sequenceNr = nextSequenceNr(), writerUuid = writerUuid, sender = sender())))
    }
  }　　

　　internalPersistAll与internalPersist比较类似，就放一块分析了，internalPersistAll只不过一次性加入了多个元素而已。但我们还是不知道handler啥时候会被调用啊。其实这一点确实有点绕，需要我们仔细认真的通读源码才能找到蛛丝马迹。

override protected[akka] def aroundReceive(receive: Receive, message: Any): Unit =
    currentState.stateReceive(receive, message)

　　在Eventsourced这个trait的方法列表中，有上面这段代码，我们知道aroundReceive是actor的一个钩子，在receive方法调用之前调用。

/**
   * INTERNAL API.
   *
   * Can be overridden to intercept calls to this actor's current behavior.
   *
   * @param receive current behavior.
   * @param msg current message.
   */
  @InternalApi
  protected[akka] def aroundReceive(receive: Actor.Receive, msg: Any): Unit = {
    // optimization: avoid allocation of lambda
    if (receive.applyOrElse(msg, Actor.notHandledFun).asInstanceOf[AnyRef] eq Actor.NotHandled) {
      unhandled(msg)
    }
  }

　　默认实现就是调用receive方法，如果receive无法匹配对应的消息，则调用unhandled来处理。既然Eventsourced覆盖了aroundReceive，那就意味着aroundReceive在持久化过程中很重要，但是又简单的调用了currentState.stateReceive方法，说明currentState就很重要了。

 // safely null because we initialize it with a proper `waitingRecoveryPermit` state in aroundPreStart before any real action happens
  private var currentState: State = null

override protected[akka] def aroundPreStart(): Unit = {
    require(persistenceId ne null, s"persistenceId is [null] for PersistentActor [${self.path}]")
    require(persistenceId.trim.nonEmpty, s"persistenceId cannot be empty for PersistentActor [${self.path}]")

    // Fail fast on missing plugins.
    val j = journal;
    val s = snapshotStore
    requestRecoveryPermit()
    super.aroundPreStart()
  }

　　注释中说的也比较清楚，这个变量是在aroundPreStart中最先赋值的，而aroundPreStart又是Actor的一个生命周期hook。其实吧这个j和s变量的赋值，我是不喜欢的，居然用简单的赋值来初始化一个lazy val变量，进而判断对应的插件是否存在，真是恶心。

private def requestRecoveryPermit(): Unit = {
    extension.recoveryPermitter.tell(RecoveryPermitter.RequestRecoveryPermit, self)
    changeState(waitingRecoveryPermit(recovery))
  }

　　这个方法给recoveryPermitter发送了一个RequestRecoveryPermit消息，然后修改状态为waitingRecoveryPermit。通过分析源码得知recoveryPermitter是一个RecoveryPermitter，这个actor在收到RequestRecoveryPermit消息后，会返回一个RecoveryPermitGranted消息。

/**
   * Initial state. Before starting the actual recovery it must get a permit from the
   * `RecoveryPermitter`. When starting many persistent actors at the same time
   * the journal and its data store is protected from being overloaded by limiting number
   * of recoveries that can be in progress at the same time. When receiving
   * `RecoveryPermitGranted` it switches to `recoveryStarted` state
   * All incoming messages are stashed.
   */
  private def waitingRecoveryPermit(recovery: Recovery) = new State {

    override def toString: String = s"waiting for recovery permit"

    override def recoveryRunning: Boolean = true

    override def stateReceive(receive: Receive, message: Any) = message match {
      case RecoveryPermitter.RecoveryPermitGranted ⇒
        startRecovery(recovery)

      case other ⇒
        stashInternally(other)
    }
  }

　　waitingRecoveryPermit返回了一个新的State，这个state的stateReceive有重新定义，我们知道当前持久化actor收到消息后会首先调用state的stateReceive，此时也就是调用waitingRecoveryPermit的stateReceive。所以RecoveryPermitGranted会按照上面源码的逻辑进行处理。此时recovery是默认的Recovery

/**
 * Recovery mode configuration object to be returned in [[PersistentActor#recovery]].
 *
 * By default recovers from latest snapshot replays through to the last available event (last sequenceId).
 *
 * Recovery will start from a snapshot if the persistent actor has previously saved one or more snapshots
 * and at least one of these snapshots matches the specified `fromSnapshot` criteria.
 * Otherwise, recovery will start from scratch by replaying all stored events.
 *
 * If recovery starts from a snapshot, the persistent actor is offered that snapshot with a [[SnapshotOffer]]
 * message, followed by replayed messages, if any, that are younger than the snapshot, up to the
 * specified upper sequence number bound (`toSequenceNr`).
 *
 * @param fromSnapshot criteria for selecting a saved snapshot from which recovery should start. Default
 *                     is latest (= youngest) snapshot.
 * @param toSequenceNr upper sequence number bound (inclusive) for recovery. Default is no upper bound.
 * @param replayMax maximum number of messages to replay. Default is no limit.
 */
@SerialVersionUID(1L)
final case class Recovery(
  fromSnapshot: SnapshotSelectionCriteria = SnapshotSelectionCriteria.Latest,
  toSequenceNr: Long                      = Long.MaxValue,
  replayMax:    Long                      = Long.MaxValue)

　　Recovery就是给状态恢复提供状态，其实吧，这应该改一个名字，叫RecoveryConfig比较合适。

private def startRecovery(recovery: Recovery): Unit = {
    val timeout = {
      val journalPluginConfig = this match {
        case c: RuntimePluginConfig ⇒ c.journalPluginConfig
        case _                      ⇒ ConfigFactory.empty
      }
      extension.journalConfigFor(journalPluginId, journalPluginConfig).getMillisDuration("recovery-event-timeout")
    }
    changeState(recoveryStarted(recovery.replayMax, timeout))
    loadSnapshot(snapshotterId, recovery.fromSnapshot, recovery.toSequenceNr)
  }

　　startRecovery大概做了两件事，修改状态为recoveryStarted，然后调用loadSnapshot。recoveryStarted不分析，先来看看loadSnapshot。

 /**
   * Instructs the snapshot store to load the specified snapshot and send it via an [[SnapshotOffer]]
   * to the running [[PersistentActor]].
   */
  def loadSnapshot(persistenceId: String, criteria: SnapshotSelectionCriteria, toSequenceNr: Long): Unit =
    snapshotStore ! LoadSnapshot(persistenceId, criteria, toSequenceNr)

　　就是给snapshotStore发送了一个LoadSnapshot消息，LoadSnapshot又是啥呢？

/** Snapshot store plugin actor. */
  private[persistence] def snapshotStore: ActorRef

　　居然是一个ActorRef，上面只是snapshotStore的定义，那具体在哪里实现的呢？这个，，，你猜一下？哈哈。

private[persistence] lazy val snapshotStore = {
    val snapshotPluginConfig = this match {
      case c: RuntimePluginConfig ⇒ c.snapshotPluginConfig
      case _                      ⇒ ConfigFactory.empty
    }
    extension.snapshotStoreFor(snapshotPluginId, snapshotPluginConfig)
  }

　　我们可以在Eventsourced找到一个同名的定义，嗯，没错就是在这里实现并赋值的。从这段代码来看，snapshotStoreFor肯定就是根据配置创建了一个Actor，并返回了它的ActorRef。那么snapshotPlugin是如何配置的呢？下面是官方demo的一个配置。

# Absolute path to the default journal plugin configuration entry.
akka.persistence.journal.plugin = "akka.persistence.journal.inmem"
# Absolute path to the default snapshot store plugin configuration entry.
akka.persistence.snapshot-store.plugin = "akka.persistence.snapshot-store.local"

　　而akka.persistence.snapshot-store.local的配置如下：

akka.persistence.snapshot-store.local {
    # Class name of the plugin.
    class = "akka.persistence.snapshot.local.LocalSnapshotStore"
    # Dispatcher for the plugin actor.
    plugin-dispatcher = "akka.persistence.dispatchers.default-plugin-dispatcher"
    # Dispatcher for streaming snapshot IO.
    stream-dispatcher = "akka.persistence.dispatchers.default-stream-dispatcher"
    # Storage location of snapshot files.
    dir = "snapshots"
    # Number load attempts when recovering from the latest snapshot fails
    # yet older snapshot files are available. Each recovery attempt will try
    # to recover using an older than previously failed-on snapshot file
    # (if any are present). If all attempts fail the recovery will fail and
    # the persistent actor will be stopped.
    max-load-attempts = 3
}

　　很显然对应的class是akka.persistence.snapshot.local.LocalSnapshotStore，LocalSnapshotStore收到LoadSnapshot之后如何应答的呢？

　　上面代码显示，LocalSnapshotStore会先调用loadAsync进行加载快照，然后返回加载结果给sender。

/**
   * Response message to a [[LoadSnapshot]] message.
   *
   * @param snapshot loaded snapshot, if any.
   */
  final case class LoadSnapshotResult(snapshot: Option[SelectedSnapshot], toSequenceNr: Long)
    extends Response

　　具体如何加载的，这里先略过，反正是返回了LoadSnapshotResult消息。recoveryStarted是如何对这个消息响应的呢？

　　上面代码显示，如果快照加载成功会调用recoveryBehavior来处理，

private val recoveryBehavior: Receive = {
      val _receiveRecover = try receiveRecover catch {
        case NonFatal(e) ⇒
          try onRecoveryFailure(e, Some(e))
          finally context.stop(self)
          returnRecoveryPermit()
          Actor.emptyBehavior
      }

      {
        case PersistentRepr(payload, _) if recoveryRunning && _receiveRecover.isDefinedAt(payload) ⇒
          _receiveRecover(payload)
        case s: SnapshotOffer if _receiveRecover.isDefinedAt(s) ⇒
          _receiveRecover(s)
        case RecoveryCompleted if _receiveRecover.isDefinedAt(RecoveryCompleted) ⇒
          _receiveRecover(RecoveryCompleted)

      }
    }

　　而recoveryBehavior又把消息转发给了receiveRecover，receiveRecover是用户自定义的，状态恢复的函数。状态恢复之后又修改了当前状态为recovering，修改之后给journal发送ReplayMessages消息，请读者注意这个消息的几个参数的实际值。

　　其实吧，根据快照的加载过程来分析上面的代码，就简单了，我们可以先猜一下，它估计就是给journal发送状态恢复的消息，包括从哪里恢复，恢复后的消息发送给谁（当然是self了），journal会把反序列化后的消息以ReplayedMessage的形式返回给self，self去调用recoveryBehavior函数。这个函数哪里定义的呢？从源码上下文来看，居然是从上一个状态（recoveryStarted）定义然后传过来的，这实现，真尼玛醉了。

# In-memory journal plugin.
akka.persistence.journal.inmem {
    # Class name of the plugin.
    class = "akka.persistence.journal.inmem.InmemJournal"
    # Dispatcher for the plugin actor.
    plugin-dispatcher = "akka.actor.default-dispatcher"
}

　　与查找快照相关actor的方法一样，我们找到了默认配置，InmemJournal是不是像我们猜测的那样实现的呢？

　　上面是InmemJournal收到ReplayMessages消息之后的逻辑，其实概括来说就是asyncReplayMessages从某个序号开始，读取并构造恢复后的数据，然后发送给replyTo。不过需要提醒读者adaptFromJournal，这个在官网也说过，是用来做消息适配的，可以对历史消息进行适配，当然了这主要是在版本升级过程中特别有用。有时候，随着系统的升级，存储的消息跟现在的消息不一定能很好的兼容，而adaptFromJournal作为一个可配置的插件机制，来动态的混入特定的类，对旧消息进行适配，转换成新版本的消息类型，再进行状态恢复。请记住这种神一样的操作，我还是非常喜欢这种设计的。

　　还要一点需要注意，那就是RecoverySuccess，这只有在所有的消息都发送完之后发送，也就是说，历史消息处理完之后，会发送给一个RecoverySuccess标志结束状态。ReplayedMessage这个消息忽略，反正它就是调用我们自定义的receiveRecover函数恢复状态。我们先来看如何处理RecoverySuccess，就是调用transitToProcessingState修改了当前状态，然后调用returnRecoveryPermit消息。

private def returnRecoveryPermit(): Unit =
        extension.recoveryPermitter.tell(RecoveryPermitter.ReturnRecoveryPermit, self)

　　returnRecoveryPermit又给recoveryPermitter发送了ReturnRecoveryPermit，真是麻烦。而recoveryPermitter还记得是干啥的么？其实就是控制并发读取jounal的数量，如果太多，就先把需要状态恢复的actor推迟一段时间，因为需要恢复状态的actor可能很多，若不加限制则会导致IO飙升。这样做还是很有好处的，比如leveldb是不太适合高并发的读取的，jdbc也是不太适合大量链接的，反正限制一下并行总是有好处的嘛，毕竟这需要跟存储打交道，不能把存储搞挂了。

　　recoveryPermitter是如何处理ReturnRecoveryPermit消息的，这里就不贴源码分析了，反正就是释放并发度的，感兴趣的可以自行阅读。

private def transitToProcessingState(): Unit = {
        if (eventBatch.nonEmpty) flushBatch()

        if (pendingStashingPersistInvocations > 0) changeState(persistingEvents)
        else {
          changeState(processingCommands)
          internalStash.unstashAll()
        }

      }

　　transitToProcessingState就显得比较重要了，它又改变了当前的行为。我们假设actor刚启动，还没有收到任何command，上面这段代码会走第二个if语句的else部分。又修改了状态，然后调用了internalStash.unstashAll()

/**
   * Command processing state. If event persistence is pending after processing a
   * command, event persistence is triggered and state changes to `persistingEvents`.
   */
  private val processingCommands: State = new ProcessingState {
    override def toString: String = "processing commands"

    override def stateReceive(receive: Receive, message: Any) =
      if (common.isDefinedAt(message)) common(message)
      else try {
        Eventsourced.super.aroundReceive(receive, message)
        aroundReceiveComplete(err = false)
      } catch {
        case NonFatal(e) ⇒ aroundReceiveComplete(err = true); throw e
      }

    private def aroundReceiveComplete(err: Boolean): Unit = {
      if (eventBatch.nonEmpty) flushBatch()

      if (pendingStashingPersistInvocations > 0) changeState(persistingEvents)
      else unstashInternally(all = err)
    }

    override def onWriteMessageComplete(err: Boolean): Unit = {
      pendingInvocations.pop()
      unstashInternally(all = err)
    }
  }

　　processingCommands很重要，这里贴出了完整的源码。

　　它继承了ProcessingState这个抽象类，收到消息后，先判断common这个Receive能否处理，不能处理则调用用户自定义的receive方法，其实就是receiveCommand。简单来说就是，判断是不是系统消息，如果是系统消息就拦截处理一下，不是就直接调用用户定义的方法。aroundReceiveComplete这方法非常重要，因为它是在用户自定义处理函数之后调用的，为啥在后面调用就重要？因为调用receiveCommand之后，eventBatch和pendingStashingPersistInvocations就有值了啊。

private def aroundReceiveComplete(err: Boolean): Unit = {
      if (eventBatch.nonEmpty) flushBatch()

      if (pendingStashingPersistInvocations > 0) changeState(persistingEvents)
      else unstashInternally(all = err)
    }

　　根据之前persist的分析，eventBatch此时已经有值了，所以会调用flushBatch

private def flushBatch() {
    if (eventBatch.nonEmpty) {
      journalBatch ++= eventBatch.reverse
      eventBatch = Nil
    }

    flushJournalBatch()
  }

　　请注意journalBatch的赋值逻辑，它居然把用户的eventBatch反转了。

private def flushJournalBatch(): Unit =
    if (!writeInProgress && journalBatch.nonEmpty) {
      journal ! WriteMessages(journalBatch, self, instanceId)
      journalBatch = Vector.empty
      writeInProgress = true
    }

　　很显然，flushJournalBatch把用户的eventBatch的值，用WriteMessages封装一下发送给了journal，journal收到WriteMessages消息肯定是把消息序列换然后存起来了啊，根据common的定义，存成功之后肯定还会返回WriteMessagesSuccess消息的喽。参照journal状态恢复的机制来看，全都保存成功后，还应该发送WriteMessagesSuccessful消息喽，收到WriteMessagesSuccessful消息后就会把writeInProgress设置成false。

 case WriteMessageSuccess(p, id) ⇒
        // instanceId mismatch can happen for persistAsync and defer in case of actor restart
        // while message is in flight, in that case we ignore the call to the handler
        if (id == instanceId) {
          updateLastSequenceNr(p)
          try {
            peekApplyHandler(p.payload)
            onWriteMessageComplete(err = false)
          } catch {
            case NonFatal(e) ⇒ onWriteMessageComplete(err = true); throw e
          }
        }

　　上面是对写入成功后的处理逻辑。注意peekApplyHandler，根据它的定义以及参数来看，这个就是在调用之前的handler。

 private def peekApplyHandler(payload: Any): Unit =
    try pendingInvocations.peek().handler(payload)
    finally flushBatch()

　　确实，它从pendingInvocations中peek，然后调用了handler，之后调用了onWriteMessageComplete。请注意在flushBatch之后，又调用了changeState(persistingEvents)，此时的状态是persistingEvents，所以onWriteMessageComplete源码如下。

override def onWriteMessageComplete(err: Boolean): Unit = {
      pendingInvocations.pop() match {
        case _: StashingHandlerInvocation ⇒
          // enables an early return to `processingCommands`, because if this counter hits `0`,
          // we know the remaining pendingInvocations are all `persistAsync` created, which
          // means we can go back to processing commands also - and these callbacks will be called as soon as possible
          pendingStashingPersistInvocations -= 1
        case _ ⇒ // do nothing
      }

      if (pendingStashingPersistInvocations == 0) {
        changeState(processingCommands)
        unstashInternally(all = err)
      }
    }

　　可以看到它从pendingInvocations中pop了一个元素，如果pendingInvocations为空，则重新进入processingCommands状态。

override def stateReceive(receive: Receive, message: Any) =
      if (common.isDefinedAt(message)) common(message)
      else stashInternally(message)

　　而在persistingEvents状态下，对于用户的普通消息会调用stashInternally进行缓存的，也就是说在pendingInvocations调用完之后，用户的消息是直接缓存而不处理的。当重新进入processingCommands状态时，才会把原来缓存的命令unstash，进行后续的处理。这就是同步的概念，同步其实是针对持久化actor的，而不针对存数据等一系列内部操作。这种设计我还是很喜欢的。

　　其实分析到这里，我们对持久化actor的persist就分析完了。概括来说就是，PersistentActor会创建两个Actor：journal、snapshotStore。分别用来持久化、恢复用户消息和当前状态快照。PersistentActor重写aroundReceive，先调用用户的自定义事件逻辑，之后把需要持久化的消息发送给journal，这期间收到的所有命令先缓存，持久化成功后，调用对应的handler，然后把缓存的消息在发出来，进行下一步处理。当然了，这期间使用changeState维护当前状态，这一点我非常不喜欢，既然是维护状态那为啥不用FSM来实现，非要自己写呢？！！

　　有一点我觉得设计的非常好，也非常巧妙，当然也非常自然。那就是journal和snapshotStore都是两个普通的actor，读者可能会问了，这有啥好值得提的呢？其实你仔细想想就知道为啥了，好像如果这两个组件不用actor来封装，就是单纯的读写存储，也是很正常的一件事呢。至于并发嘛，通过线程池也是可以控制的，用个actor就值得吹嘘了？值得，很值得。为啥呢？因为灵活和自构建（akka的很多高级特性都是基于最基本的actor实现的），而且试想在remote和cluster模式下，这有啥好处呢？如果这就是简单的actor，你是不是可以把journal、snapshotStore放到远程呢？而这个远程节点是不是可以使用一个SSD硬盘的节点呢？这两个actor既然有并发的限制，那它能不能是一个router呢？能不能有一组journal、snapshotStore来处理某些actor（persistId的hash值相同）的持久化呢？我实在不敢想下去了，有了actor简直就是完美啊，简直就是牛逼出了天际啊。^_^

　　最后啰嗦一点，刚才你一定注意到journalBatch是逆序的了。下面是PersistentActor内部的stash的逻辑，加上官方的注释你一定就知道为啥了，我就不再具体分析啦。

/**
   * Enqueues `envelope` at the first position in the mailbox. If the message contained in
   * the envelope is a `Terminated` message, it will be ensured that it can be re-received
   * by the actor.
   */
  private def enqueueFirst(envelope: Envelope): Unit = {
    mailbox.enqueueFirst(self, envelope)
    envelope.message match {
      case Terminated(ref) ⇒ actorCell.terminatedQueuedFor(ref)
      case _               ⇒
    }
  }

akka Persistence

事件溯源