TF_RNNCell

参考：链接。

RNNCell

• BasicRNNCell
• GRUCell
• BasicLSTMCell
• LSTMCell
• MultiRNNCell

抽象类RNNCell

所有的rnncell均继承于RNNCell, RNNCell主要定义了几个抽象方法：

 def __call__(self, inputs, state, scope=None):
     raise NotImplementedError("Abstract method")

 @property
 def state_size(self):
     raise NotImplementedError("Abstract method")

 @property
 def output_size(self):
     raise NotImplementedError("Abstract method")

上述方法，__call__在对象被使用时调用，其他可以看做属性方法，主要用作获取状态state的大小，cell的输出大小。既然对象使用时会调用__call__，那么各类RNN的操作都定义在这个方法中。接下来，我们就针对各个不同的cell来详细介绍各类RNN。

BasicRNNCell

这个cell是最基础的一个RNNCell，可以看做是对一般全连接层的拓展，除了在水平方向加入时序关系，可以用下图表示：

而BasicRNNCell的初始化方法可如代码所示：

 def __init__(self, num_units, input_size=None, activation=tanh):
     if input_size is not None:
       logging.warn("%s: The input_size parameter is deprecated.", self)
     self._num_units = num_units
     self._activation = activation

初始化只需要给出num_units,用来指有多少个隐藏层单元；而activation指使用哪种激活函数用作激活输出。而对应的RNN操作定义在__call__方法中：

 def __call__(self, inputs, state, scope=None):
     """Most basic RNN: output = new_state = activation(W * input + U * state + B)."""
     with vs.variable_scope(scope or type(self).__name__):  # "BasicRNNCell"
       output = self._activation(_linear([inputs, state], self._num_units, True))
     return output, output

很清晰，inputs表示隐藏层的输入，state表示上个时间的隐藏层状态，也可以说是上一次隐藏层向自身的输出，对于第一次输入，则需要初始化state，对应初始化方法有很多种，可以使用tensorflow提供的各种初始化函数。在__call__中，对输入inputs和state进行activation(wx+b)，用作下次的输入。

GRUCell

GRU是对RNN的一种改进，相比LSTM来说，也可以看做是对LSTM的一种简化，是Bengio在14年提出来的，用作机器翻译。先看一下GRU的基本结构：

这里我们结合代码来看原理：

def __call__(self, inputs, state, scope=None):
  """Gated recurrent unit (GRU) with nunits cells."""
  with vs.variable_scope(scope or type(self).__name__):  # "GRUCell"
    with vs.variable_scope("Gates"):  # Reset gate and update gate.
      # We start with bias of 1.0 to not reset and not update.
      r, u = array_ops.split(1, 2, _linear([inputs, state],
                                           2 * self._num_units, True, 1.0))
      r, u = sigmoid(r), sigmoid(u)
    with vs.variable_scope("Candidate"):
      c = self._activation(_linear([inputs, r * state],
                                   self._num_units, True))
    new_h = u * state + (1 - u) * c
  return new_h, new_h

GRUCell的初始化与RNN一样，给出输入和初始化的state，在使用对象时，利用输入和前一个时间的隐藏层状态，得到对应的Gates: r, u, 然后利用r更新cell状态，最后利用u得到新的隐藏层状态。对于RNN的改进，最厉害的莫过于下面的，而且有很多变种，这里tensorflow中只有几个简单常见的cell。接下来，我们开始看看LSTM。

BasicLSTMCell

这个cell可以看做是最简单的LSTM，在每个连接中没有额外的连接，即其他变种在连接中加入各种改进。对于BasicLSTMCell,可以如下图所示：

同样的，我们结合代码来看它的原理：

 def __call__(self, inputs, state, scope=None):
   """Long short-term memory cell (LSTM)."""
   with vs.variable_scope(scope or type(self).__name__):  # "BasicLSTMCell"
     # Parameters of gates are concatenated into one multiply for efficiency.
     if self._state_is_tuple:
       c, h = state
     else:
       c, h = array_ops.split(1, 2, state)
     concat = _linear([inputs, h], 4 * self._num_units, True)

     # i = input_gate, j = new_input, f = forget_gate, o = output_gate
     i, j, f, o = array_ops.split(1, 4, concat)

     new_c = (c * sigmoid(f + self._forget_bias) + sigmoid(i) *
              self._activation(j))
     new_h = self._activation(new_c) * sigmoid(o)

     if self._state_is_tuple:
       new_state = LSTMStateTuple(new_c, new_h)
     else:
       new_state = array_ops.concat(1, [new_c, new_h])
     return new_h, new_state

lstm有三个门，inputs, forget, output, 而中间cell用来管理结合他们生产需要的输出。在初始化结束之后，利用输入分别得到对应的门的输出，然后利用这三个门的信息分别更新cell和当前隐藏层状态。f 用来控制遗忘之前的信息和记忆当前信息的比例，进而更新cell，lstm可以看做是一种复杂的激活函数，它的存在依赖RNN的递归性。BasicLSTMCell只是个最基本的LSTM，而完整的LSTM可能比这个复杂，可以参看blog。

MultiRNNCell

对于MultiRNNCell，只能贴出完整代码来分析了：

 class MultiRNNCell(RNNCell):
   """RNN cell composed sequentially of multiple simple cells."""

   def __init__(self, cells, state_is_tuple=False):
     """Create a RNN cell composed sequentially of a number of RNNCells.

     Args:
       cells: list of RNNCells that will be composed in this order.
       state_is_tuple: If True, accepted and returned states are n-tuples, where
         `n = len(cells)`.  By default (False), the states are all
         concatenated along the column axis.

     Raises:
       ValueError: if cells is empty (not allowed), or at least one of the cells
         returns a state tuple but the flag `state_is_tuple` is `False`.
     """
     if not cells:
       raise ValueError("Must specify at least one cell for MultiRNNCell.")
     self._cells = cells
     self._state_is_tuple = state_is_tuple
     if not state_is_tuple:
       if any(nest.is_sequence(c.state_size) for c in self._cells):
         raise ValueError("Some cells return tuples of states, but the flag "
                          "state_is_tuple is not set.  State sizes are: %s"
                          % str([c.state_size for c in self._cells]))

   @property
   def state_size(self):
     if self._state_is_tuple:
       return tuple(cell.state_size for cell in self._cells)
     else:
       return sum([cell.state_size for cell in self._cells])

   @property
   def output_size(self):
     return self._cells[-1].output_size

   def __call__(self, inputs, state, scope=None):
     """Run this multi-layer cell on inputs, starting from state."""
     with vs.variable_scope(scope or type(self).__name__):  # "MultiRNNCell"
       cur_state_pos = 0
       cur_inp = inputs
       new_states = []
       for i, cell in enumerate(self._cells):
         with vs.variable_scope("Cell%d" % i):
           if self._state_is_tuple:
             if not nest.is_sequence(state):
               raise ValueError(
                   "Expected state to be a tuple of length %d, but received: %s"
                   % (len(self.state_size), state))
             cur_state = state[i]
           else:
             cur_state = array_ops.slice(
                 state, [0, cur_state_pos], [-1, cell.state_size])
             cur_state_pos += cell.state_size
           cur_inp, new_state = cell(cur_inp, cur_state)
           new_states.append(new_state)
     new_states = (tuple(new_states) if self._state_is_tuple
                   else array_ops.concat(1, new_states))
     return cur_inp, new_states

创建对象时，可以看到初始化函数中不再是输入，而是变成了cells，,即一个cell是一层，多个cell便有多层RNNcell。而在使用对象时，单层可以看做多层的特例，对于输入inputs和state，同时得到多个cell的当前隐藏层状态，用作下个时间步。看似麻烦，其实很简洁，就是加入了对多个cell的计算，最后得到的新的隐藏层状态即每个cell的上个时间步的输出。