[Xavier] Understanding the difficulty of training deep feedforward neural networks

目录

• 概
• 主要内容

Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks[C]. international conference on artificial intelligence and statistics, 2010: 249-256.

@article{glorot2010understanding,

title={Understanding the difficulty of training deep feedforward neural networks},

author={Glorot, Xavier and Bengio, Yoshua},

pages={249--256},

year={2010}}

概

本文提出了Xavier参数初始化方法.

主要内容

在第\(i=1, \ldots, d\)层:

\[\mathbf{s}^i=\mathbf{z}^i W^i+\mathbf{b}^i \\
\mathbf{z}^{i+1}= f(\mathbf{s}^i),
\]

其中\(\mathbf{z}^i\)是第\(i\)层的输入, \(\mathbf{s}^i\)是激活前的值, \(f(\cdot)\)是激活函数(假设其在0点对称, 且\(f'(0)=1\) 如tanh).

则

\[\mathrm{Var}(z^i) = n_l\mathrm{Var}(w^iz^i),
\]

在\(0\)附近近似成立(既然\(f'(0)=1\)), 其中\(z^i, w^i,\)分别是\(\mathbf{z}^i, W^i\)的某个元素, 且假设这些\(\{w^i\}\)之间是独立同分布的, \(w^i, z^i\)是相互独立的, 进一步假设\(\mathbb{E}(w^i)=0,\mathbb{E}(x)=0\)(\(x\)是输入的样本), 则

\[\mathrm{Var}(z^i) = n_l\mathrm{Var}(w^i)\mathrm{Var}(z^i),
\]

在\(0\)点附近近似成立.

故

\[\mathrm{Var}(z^i) = \mathrm{Var}(x) \prod_{i'=0}^{i-1} n_{i'} \mathrm{Var}(w_{i'})
\]

其中\(n_i\)表示第\(i\)层输入的节点个数.

根据梯度反向传播可知:

\[\tag{2}
\frac{\partial Cost}{\partial s_k^i} = f'(s_k^i) W_{k, \cdot}^{i+1} \frac{\partial Cost}{\partial \mathbf{s}^{i+1}}
\]

\[\tag{3}
\frac{\partial Cost}{\partial w_{l,k}^i} = z_l^i \frac{\partial Cost}{\partial s_k^i}.
\]

于是

\[\tag{6}
\mathrm{Var}[\frac{\partial Cost}{\partial s_k^i}] = \mathrm{Var}[\frac{\partial Cost}{\partial s^d}] \prod_{i'=i}^d n_{i'+1} \mathrm{Var} [w^{i'}],
\]

\[\mathrm{Var}[\frac{\partial Cost}{\partial w^i}] = \prod_{i'=0}^{i-1} n_{i'} \mathrm{Var}[w^{i'}] \prod_{i'=i}^d n_{i'+1} \mathrm{Var} [w^{i'}] \times \mathrm{Var}(x) \mathrm{Var}[\frac{\partial Cost}{\partial s^d}],
\]

当我们要求前向进程中关于\(z^i\)的方差一致, 则

\[\tag{10}
\forall i, \quad n_i \mathrm{Var} [w^i]=1.
\]

当我们要求反向进程中梯度的方差\(\frac{\partial Cost}{\partial s^i}\)一致, 则

\[\tag{11}
\forall i \quad n_{i+1} \mathrm{Var} [w^i]=1.
\]

本文选了一个折中的方案

\[\mathrm{Var} [w^i] = \frac{2}{n_{i+1}+n_{i}},
\]

并构造了一个均匀分布, \(w^i\)从其中采样

\[w^i \sim U[-\frac{\sqrt{6}}{\sqrt{n_{i+1}+n_{i}}},\frac{\sqrt{6}}{\sqrt{n_{i+1}+n_{i}}}].
\]

文章还有许多关于不同的激活函数的分析, 如sigmoid, tanh, softsign... 这些不是重点, 就不记录了.