Deep Learning 资料总结

http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ : 理解神经网络

 

ELU：

 

梯度下降优化方式：

http://sebastianruder.com/optimizing-gradient-descent/:各种优化算法详解

 

1. GradientDescentOptimizer
   This one is sensitive to the problem and you can face lots of problems using it, from getting stuck in saddle points to oscillating around the minimum and slow convergence. I found it useful for Word2Vec, CBOW and feed-forward architectures in general, but Momentum is also good.
2. AdadeltaOptimizer 
   Adadelta addresses the issues of using constant of linearly decaying learning rate. In case of recurrent networks it’s among the fastest.
3. MomentumOptimizer
   If you learn a regression and find your loss function oscillating, switching from SGD to Momentum may be the right solution.
4. AdamOptimizer
   Adaptive momentum in addition to the Adadelta features.
5. FtrlOptimizer
   I haven’t used it myself, but from the paper I see that it’s better suited for online learning on large sparse datasets, like recommendation systems.
6. RMSPropOptimizer
   This is a variant Adadelta that serves the same purpose - dynamic decay of a learning rate multiplier.

 

CNN神经网络一些tricky的地方：

http://lamda.nju.edu.cn/weixs/project/CNNTricks/CNNTricks.html

摘要：

1、适合Relu的参数初始化：w = np.random.randn(n) * sqrt(2.0/n) # current recommendation

2、LR： In practice, if you see that you stopped making progress on the validation set, divide the LR by 2 (or by 5), and keep going, which might give you a surprise.亲测有效

3、关于learning rate：

RNN学习：

中文教程：http://blog.csdn.net/heyongluoyao8/article/details/48636251

 

FCN：http://blog.csdn.net/happyer88/article/details/47205839：Fully Convolutional Networks for Semantic Segmentation笔记

优点：

1，训练一个end-to-end的FCN模型，利用卷积神经网络的很强的学习能力，得到较准确的结果，以前的基于CNN的方法都是要对输入或者输出做一些处理，才能得到最终结果。

 

2，直接使用现有的CNN网络，如AlexNet, VGG16, GoogLeNet，只需在末尾加上upsampling，参数的学习还是利用CNN本身的反向传播原理，"whole image training is effective and efficient."

 

3，不限制输入图片的尺寸，不要求图片集中所有图片都是同样尺寸，只需在最后upsampling时按原图被subsampling的比例缩放回来，最后都会输出一张与原图大小一致的dense prediction map

 

理解DL细节的不错的文章：

http://colah.github.io/posts/2014-07-Conv-Nets-Modular/ ：CNN细节

http://colah.github.io/posts/2015-08-Understanding-LSTMs/：RNN细节

 

http://blog.csdn.net/mao_xiao_feng/article/details/53444333 tensorflow 计算CNN的细节

 

如果遇到了最后的输出值都一样的情况，可能的解决办法如下：

Hey, I had a similar issue with my own (hand-coded) CNN trying to get some results with the CIFAR-10 dataset. What I found was that I had forgotten to normalize the input images to some range that made sense with my weight scales. Try something like X = X / max(abs(X)) to put values between -1 and 1.

Another possibility is your weight initialization is causing many ReLU units to die. I usually initialize all weights with a small number times a normal Gaussian distribution. For wx+ b, b being the biases, you can try that + a small positive constant. I.e. b = weight_scale*random.randn(num, 1) + 0.1

Another idea — your sigmoid unit might be squashing your responses too much. They’re fairly uncommon in CNNs from what I understand, maybe just stick to ReLUs.

Last point — try testing on a small training batch (say 10–20 images) and just train until you overfit with 100% accuracy. That’s one way of knowing that your network is capable of doing something. I think these smaller tests are very important before investing hours or days into proper training, which is what these networks often require.

我最后的解决办法是：加了batch normalization，不过具体原因也没有确定

 

 

GAN的资料：

http://it.sohu.com/20161003/n469570219.shtml

http://www.wtoutiao.com/p/172tUtn.html