理解自动梯度计算autograd

理解自动求导

• 例子

def f(x):
    a = x * x
    b = x * a
    c = a + b
    return c

• 基于图理解
  
  

• 代码实现

def df(x):
   # forward pass
    a = x * x
    b = x * a
    c = a + b

    # backward pass
    g_c = 1
    # c = a + b
    g_a = g_c * 1
    g_b = g_c * 1
    # b = x * a
    g_x = g_b * a
    g_a += g_b * x
    # a = x * x
    g_x += g_a * 2x
    return g_x

• 进一步, 如果前后两节点都是向量, 相应的导数改为Jacobi矩阵, 导数改为梯度.

其他资料

• 参考 https://ai.googleblog.com/2017/11/tangent-source-to-source-debuggable.html
• google http://learningsys.org/nips17/assets/papers/paper_28.pdf
• 自动求导的定理证明 https://page.mi.fu-berlin.de/rojas/neural/chapter/K7.pdf