02 Transformer 中 Add&Norm （残差和标准化）代码实现

python/pytorch 基础

https://www.cnblogs.com/nickchen121

培训机构（Django 类似于 Transformers）

首先由一个 norm 函数

norm 里面做残差，会输入（ x 和 淡粉色z1，残差值），输出一个值紫粉色的 z1

标准化

\[y = \frac{x-E(x)}{\sqrt{Var(x)+\epsilon}}*\gamma+\beta
\]

\(E(x)\) 对 x 求均值

\(Var(x)\) 对 x 求方差

\(\epsilon\) 加在方差上的数字，避免分母为0；

\(\gamma\)和\(\beta\) 为学习参数，二者均可学习随着训练过程而变化；

class LayerNorm(nn.Module):

    def __init__(self, feature, eps=1e-6):
        """
        :param feature: self-attention 的 x 的大小
        :param eps:
        """
        super(LayerNorm, self).__init__()
        self.a_2 = nn.Parameter(torch.ones(feature))
        self.b_2 = nn.Parameter(torch.zeros(feature))
        self.eps = eps

    def forward(self, x):
        mean = x.mean(-1, keepdim=True)
        std = x.std(-1, keepdim=True)
        return self.a_2 * (x - mean) / (std + self.eps) + self.b_2

残差+标准化

class SublayerConnection(nn.Module):
    """
    这不仅仅做了残差，这是把残差和 layernorm 一起给做了

    """
    def __init__(self, size, dropout=0.1):
        super(SublayerConnection, self).__init__()
        # 第一步做 layernorm
        self.layer_norm = LayerNorm(size)
        # 第二步做 dropout
        self.dropout = nn.Dropout(p=dropout)

    def forward(self, x, sublayer):
        """
        :param x: 就是self-attention的输入
        :param sublayer: self-attention层
        :return:
        """
        return self.dropout(self.layer_norm(x + sublayer(x)))