强化学习（八）：Eligibility Trace

Eligibility Traces

Eligibility Traces是强化学习中很基本很重要的一个概念。几乎所有的TD算法可以结合eligibility traces获得更一般化的算法，并且通常会更有效率。

Eligibility traces可以将TD和Monte Carlo算法统一起来。之前我们见过n-step方法将二者统一起来，eligibility traces 优于n-step的主要地方在于计算非常有效率，其只需要一个trace向量，而不需要储存n个特征向量。另外，eligibility trace的学习是逐步的，而不需要等到n-steps之后。

Eligibility traces机制是通过一个短期的向量得到的，这个向量称为eligibility trace $\pmb{z_t} \in R^d$。正好呼应算法中的权重向量$\pmb{w_t}\in R^d$.　大致的思想是这样的：当权重向量的某些分量参与到估计价值当中，那么与之相对应的eligibility trace分量也会立即变化，然后变化会慢慢的消失。

强化学习的很多算法都是‘向前看’的算法，即当前状态或动作的价值更新是依赖于未来状态或动作的，这样的　算法称为‘前向视角’。前向视角实施起来多少有些不方便，毕竟未来的事，当前并知道。还有另外一种思路是所谓的‘后向视角’，通过eligibility trace来考虑最近刚刚访问过的状态或动作。

The $\lambda$-return

之前定义过n-steps return为最初n rewards加上n步之后的状态的价值估计的总和：

\[G_{t:t+n} \dot = R_{t+1} + \gamma R_{t+2} + ... + \gamma^{n-1}R_{t+n} + \gamma^n \hat v(S_{t+n}, w_{t+n-1})
\]

其实一个有效的更新不一定只依赖于n－steps return，可能依赖于多个n-step return的平均会更好，比如：

\[\frac{1}{2}G_{t:t+2} + \frac{1}{2}G_{t:t+4}
\]

$\lambda$-return 可以理解为一种特殊的平均多个n-step return，它平均所有n-step return：

\[G_{t}^{\lambda}\dot = (1- \lambda)\sum_{n=1}^{\infty}\gamma^{n-1}G_{t:t+n} = (1-\lambda)\sum_{n=1}^{T-t-1}\lambda^{n-1}G_{t:t+n} + \lambda^{T-t-1}G_{t}
\]

基于$\lambda$ return 的学习算法比如off-line $\lambda$－return algorithm:

\[w_{t+1}\dot = w_t + \alpha [ G_t^{\lambda} - \hat v(S_t, w_t)]\nabla \hat v(S_t,w_t), t = 0,..., T-1.
\]

以上其实就是一种前向视角：

TD($\lambda$)

TD($\lambda$)是上面off-line $\lambda$-return算法的改进，主要有三点：一是它每步更新，二是因为每步更新，它的计算过程分布在每一时间步上，三是它可以应用于连续过程而不仅限于episodic problems。

如上所述，eligibility trace　通过一个向量$\pmb{z_t}\in R^d$ 实现，其与$\pmb{w_t}$维度相同，$\pmb{w_t}$可看作是长时记忆，而eligibility trace则是短时记忆。

\[z_{-1}\dot = 0, \qquad z_t\dot = \gamma \lambda z_{t-1} +\nabla\hat v(S_t,w_t),\qquad 0 \le t\le T
\]

其中$\gamma$是discount rate，$\lambda$ 是trace decay参数。Eligibility trace记录权重向量各成分在近期状态价值评估中的贡献。

TD error:

\[\delta_t \dot = R_{t+1} +\gamma\hat v(S_{t+1},w_t)- \hat v(S_t,w_t)
\]

那么权重向量的更新：

\[w_{t+1} \dot = w_t+\alpha \delta_t z_t
\]

# semi-gradient TD(lambda) for estimating v_hat = v_pi

Input: the policy pi to be evaluated

Input: a differentiable function v_hat: S_plus x R^d --> R such that v_hat(terminal,.) = 0

Algorithm parameters: step size alpha > 0, trace decay rate lambda in [0,1]

Initialize value-function weights w arbitrarily (e.g. w = 0)

Loop for each episode:

    Initialize S

    z = 0

    Loop for each step of episode:

        Choose action A based on pi(.|S)

        Take A, observe R,S'

        z = gamma * lambda *z + grad V(S,w)

        delta = R + gamma v_hat(S',w) - v_hat(S,w)

        w = w + alpha delta z

        S = S'

    until S' is terminal

TD($\lambda$)是时间上反向的，从当前状态，向后考虑已经经历过的状态。

易知，当$\lambda = 0$　$TD(\lambda)$退化成为one step semi-gradient也就是TD(0)；当$　＝　１\lambda　＝　１$时，TD(1)则表现出MC的特征。

n-step Truncated $\lambda$-return Methods

线下的$\lambda-return$算法具有很大的局限性，因为$\lambda$-return要在最后才知道，在连续状态情况下，甚至永远得不到。我们知道，一般$\gamma\lambda<1$，因此太长时间的状态价值的影响就会消失掉，那么可以想到一种近似$\lambda$-return的方法称为Truncated $\lambda$-return方法：

\[G_{t:h}^{\lambda}\dot = (1-\lambda)\sum_{n =1}^{h-t-1} \lambda^{n-1}G_{t:t+n} + \lambda^{h-t-1}G_{t:h},\qquad 0 \le t<h\le T
\]

同理可以推广出truncated TD($\lambda$)(TTD($\lambda$))：

\[w_{t+n}\dot = W_{t+n-1} + \alpha [G_{t:t+n}^\lambda - \hat v(S_t,w_{t+n-1})]\nabla \hat v(S_t,w_{t+n-1})
\]

此算法可以以一种比较有效率的方式来改写，使其不依赖于n（当然，背后的理论关系还是依赖n的）：

\[G_{t:t+k}^{\lambda} = \hat v(S_t,w_{t-1}) + \sum_{i = t}^{t+k-1} (\gamma \lambda)^{i-t}\delta_i'
\]

其中$\delta_t' \dot = R_{t+1} + \gamma\hat v(S_{t+1},w_t) -\hat v(S_t,w_{t-1})$.

Redoing Updates: The On-line $\lambda$-return Algorithm

TTD($\lambda$)中n的选取是要有取舍的，n可以很大，这样更接近off-line $\lambda$-return algorithm；或者很小这样更新参数不用等很久。这两种好处增加一些计算复杂度可以都得到。

其思想是这样的：在每一步时，都会得到新的数据，然后从头重新更新。

\[w_{t+1}^h \dot = w_t^h + \alpha [G_{t:h}^h - \hat v(S_t,w_t^h)]\nabla \hat v(S_t,w_t^h),\qquad 0\le t\le h\le T.
\]

$w_t \dot = w_t^t$此即为on-line $\lambda$-return algorithm.

True On-line TD($\lambda$)

借助eligibility trace，将on-line $\lambda$ -return 的前向视角转变成后向视角即成为true on-line TD($\lambda$).

\[w_{t+1}\dot = w_t + \alpha \delta_t z_t + \alpha(w_t^Tx_t - w_{t-1}^Tx_t)(z_t - x_t)
\]

其中$\hat v(s,w) = w^T x(s),\ x_t\dot = x(S_t),\ \delta_t \dot = R_{t+1} +\gamma\hat v(S_{t+1},w_t)- \hat v(S_t,w_t),\ z_t = \dot = \gamma\lambda z_{t-1}+(1-\alpha \gamma\lambda z_{t-1}^Tx_t)x_t $.

# True On-line TD(lambda) for estimating w^T x = v_pi

Input: the policy pi to be evaluated

Input: a feature function x: S_plus --> R such that x(terminal,.) = 0

Algorithm parameters: step size alpha > 0, trace decay rate lambda in [0,1]

Initialize value-function weights w in R

Loop for each episode:

    Initialize state and obtain initial feature vector x

    z = 0

    V_old = 0

    Loop for each step of episode:

        choose A from pi.

        take action A, observe R, x'(feature vector of the next state)

        V = w^T x

        V' = w^T x'

        delta = R + gamma V' - V

        z = gamma lambda z + (1- alpha gamma z^T x)x

        w = w + alpha(delta + V- V_old)z - alpha (V-V_old) x

        V_old = V'

        x = x'

    until x' = 0(signaling arrival at a terminal state)

这里的eligibility trace 称为 dutch trace, 还有一种称为replacing trace:

\[z_{i,t} \dot =\left\{\begin{array}\\ 1\qquad if\quad x_{i,t} = 1,\\
\gamma\lambda z_{i,t-1} \qquad otherwise.\end{array}\right.
\]

Dutch Traces in Monte Carlo Learning

Eligibility traces也可应用于Monte Carlo 相产算法之中。

在梯度Monte Carlo算法的线性版本中：

\[w_{t+1} \dot = w_t +\alpha [G - w_t^Tx_t]x_t,\qquad 0 \le t \le T.
\]

其中Ｇ是在episode结束后得到的奖励，因此没有下标，也没有折扣因子。因为MC方法只能在episode结束后更新参数，这个特点我们不试图改进，而是在尽力将计算分布在各个时间步上。

\[\begin{array}\\
w_T& =& w_{T-1} + \alpha(G - w_{T-1}^Tx_{T-1})x_{T-1}\\
&=& w_{T-1} +\alpha x_{T-1}(-x_{T-1}^Tw_{T-1})+\alpha Gx_{T-1}\\
&=& (I - \alpha x_{T-1}x_{T-1}^T)w_{T-1}+\alpha Gx_{T-1}\\
&=& F_{T-1}w_{T-1}+\alpha Gx_{T-1}
\end{array}
\]

其中$F_t \dot = I - \alpha x_t x_t^T$是一个遗忘矩阵或称衰退矩阵（forgetting or fading matrix), 可以递归得到。

Ｓarsa($\lambda$)

Sarsa($\lambda$)是一个TD方法，与TD($\lambda$)类似，只不过，Sarsa($\lambda$)是针对state-action pair的：

\[w_{t+1}\dot = w_t +\alpha \delta_t z_t
\]

其中：

\[\delta \dot = R_{t+1}+\gamma \hat q(S_{t+1},A_{t+1},w_t) - \hat q(S_t,A_t,w_t)
\]

而：

\[\begin{array}\\
z_{-1} &=& 0,\\
z_t&=& \gamma \lambda z_{t-1}+\nabla\hat q(S_t,A_t,w_t),\qquad 0\le t\le T.
\end{array}
\]

# Sarsa(lambda) with binary features and linear function approximation for estimating # w^T x q_{pi} or q*

Input: a function F(s,a) returning teh set of (indices of ) active features for s,a

Input: a policy pi (if estimating q_pi)

Algorithm parameteres: step size alpha > 0, trace decay rate lambda in [0,1]

Initialize: w= (w1,...,wd)^T in R^d z = (z1,...,z_d)^T in R^d

Loop for each episode:

    Initialize S

    Choose A from pi(.|S) or e-greedy according to q_hat(S,.,w)

    z = 0

    Loop for each step of episode:

       take action A, observe R,S'

       delta =  R

       Loop for i in F(S,A):

           delta = delta - w_i

           z_i = z_i + 1  (accumulating traces)

           or z_i = 1 (replacing traces)

       If S' is terminal then:

            w = w + alpha delta z

            Go to next episode

       Choose A' in pi(.|S) or near greedily q_hat(S',.,w)

       Loop for i in F(S',A'): dleta = delta + gamma w_i

       w = w + alpha delta z

       z = gamma lambda z

       S = S', A = A'

# True Online Sarsa(lambda) for estimating w^Tx = q_pi or q*

Input: a feature function x: s_plus x A = R^d such that x(terminal,.) = 0

Input: a policy pi (if estimating q_pi)

Algorithm parameters: step size alpha >0, trace decay rate lambda in [0,1]

Initialize: w in R^d

Loop for each episode:

    Initialize S

    Choose A from pi(.|S) or nearly greedily from S using w

    x = x(S,A)

    z = 0

    Q_old = 0

    Loop for each step of episode:

        Take action A, observe R,S'

        Choose A' from pi(.|S') or near greedily from S' using w

        x' = x(S',A')

        Q = w^Tx

        Q' = w^Tx'

        delta = R + gamma Q' - Q

        z = gamma lambda z + (1 - alpha gamma lambda z^T x) x

        w = w + alpha(delta + Q - Q_old )z - alpha(Q - Q_old) x

        Q_old = Q'

        x = x'

        A = A'

    until S' is terminal

Off-policy Eligibility Traces with Control Variates

Watkins's Q($\lambda$) to Tree-Backup($\lambda$)

Stable Off-policy Methods with Traces