深度学习有不少的trick,而且这些trick有时还挺管用的,所以,了解一些trick还是必要的。上篇说的normalization、initialization就是trick的一种,下面再总结一下自己看Deep Learning Summer School, Montreal 2016 总结的一些trick。请路过大牛指正~~~

early stop

“早停止”很好理解,就是在validation的error开始上升之前,就把网络的训练停止了。说到这里,把数据集分成train、validation和test也算是一种trick吧。看一张图就明白了:

L1、L2 regularization

第一次接触正则化是在学习PRML的回归模型,那时还不太明白,现在可以详细讲讲其中的原理了。额~~~~(I will write the surplus of this article in English)

At the very outset, we take a insight on L1, L2 regularization. Assume the loss function of a linear regression model as . In fact, L1, L2 regularization can be seen as introducing prior distribution for the parameters. Here, L1 regularization can be interpreted as Laplace prior, and Guass prior for L2 regularization.Such tricks are used to reduce the complexity of a model.

L2 regularization

As before, we make a hypothesis that the target variable  is determined by the function  with an additive guass noise ,  resulting in, where is a zero mean Guassian random variable with precision . Thus we know  and its loglikelihood function form can be wrote as

Finding the optimal solution for this problem is equal to minimize the least squre function .   If a zero mean  variance Guass prior is introduced for ,  we get the posterior distribution for  from the function (need to know prior + likelihood = posterior)

rewrite the equaltion in loglikelihood form, we will get

at the present,  you see . This derivation process accounts for why L2 regularization could be interpreted as Guass Prior.

L1 regularization

Generally, solving the the L1 regularization is called LASSO problem, which will force some coefficients to zero thus create a sparse model. You can get reference from here.

  

From the above images(two-dimensionality), something can be discovered that for the L1 regularization, the interaction points always locate on the axis, and this will forces some of the variables to be zero. Addtionally, we can learn that L2 regularization doesn't produce a sparse model(the right side image).

Fine tuning

It's easy to understand fine tuning, which could be interpreted as a way to initailizes the weight parameters then proceed a supervised learning. Parameters value can be migrated from a well-trained network, such as ImageNet. However, sometimes you may need an autoencoder, which is an unsupervised learning method and seems not very popular now. An autoencoder will learn the latent structure of the data in hidden layers, whose input and output should be same.

In fact, supervised learning of fine tune just performs regular feed forward process as in a feed-forwad network.

Then, I want to show you an image about autoencoder, also you can get more information in detail through this website.

Data argumentation

If training data is not large enough, it's a neccessary to expand the data by fliping, transferring or rotating to avoid overfit. However, according to the paper "Visualizing and Understanding Convolutional Networks",  CNN may not be as stable when fliping for image as transferring and rotating, except a symmetrical image.

Pooling

I have introduced why pooling works well in the ealier article in this blog, if interested, you may need patience to look through the catalogue. In the paper "Visualizing and Understanding Convolutional Networks", you will see pooling sometimes a good way to control the invariance, I want to write down my notes about this classical paper in next article.

Dropout and Drop layer

Dropout now has become a prevalent technique to prevent overfitting, proposed by Hinton, etc. The dropout program will inactivate some nodes according to a probability p, which has been fixed before training. It makes the network more thinner. But why does it works ?  There may exist two main reasons, (1) for a batch of input data, nodes can be trained better in a thinner network, for they trained with more data in average.  (2) dropout inject noise into network which will do help.

Inspired by dropout and ResNet, Huang, etc propose drop layer, which make use of the shortcut connection in ResNet. It will enforce the information flow through the shortcut connection directly and obstruct the normal way according to a probability just like what dropout does. This technique makes the network  shorter in training, and can be used to trained a deeper network even more than 1000 layers.

Depth is important

The chart below interprets the reason why depth is important,

as the number of layer increasing, the classification accuracy arises. While visualizing the feature extracted from layers using decovnet("Visualizing and Understanding Convolutional Networks"), the deeper layer the more abstract feature extracted. And abstract feature will strengthen the  robustness, i.e. the invariance to transformation.

【CV知识学习】early stop、regularation、fine-tuning and some other trick to be known的更多相关文章

  1. 【CV知识学习】神经网络梯度与归一化问题总结+highway network、ResNet的思考

    这是一篇水货写的笔记,希望路过的大牛可以指出其中的错误,带蒟蒻飞啊~ 一.    梯度消失/梯度爆炸的问题 首先来说说梯度消失问题产生的原因吧,虽然是已经被各大牛说烂的东西.不如先看一个简单的网络结构 ...

  2. 【CV知识学习】Fisher Vector

    在论文<action recognition with improved trajectories>中看到fisher vector,所以学习一下.但网上很多的资料我觉得都写的不好,查了一 ...

  3. 【CV知识学习】【转】beyond Bags of features for rec scenen categories。基于词袋模型改进的自然场景识别方法

    原博文地址:http://www.cnblogs.com/nobadfish/articles/5244637.html 原论文名叫Byeond bags of features:Spatial Py ...

  4. L23模型微调fine tuning

    resnet185352 链接:https://pan.baidu.com/s/1EZs9XVUjUf1MzaKYbJlcSA 提取码:axd1 9.2 微调 在前面的一些章节中,我们介绍了如何在只有 ...

  5. 网络知识学习2---(IP地址、子网掩码)(学习还不深入,待完善)

    紧接着:网络知识学习1 1.IP地址    IP包头的结构如图 A.B.C网络类别的IP地址范围(图表) A.B.C不同的分配网络数和主机的方式(A是前8个IP地址代表网络,后24个代表主机:B是16 ...

  6. (原)caffe中fine tuning及使用snapshot时的sh命令

    转载请注明出处: http://www.cnblogs.com/darkknightzh/p/5946041.html 参考网址: http://caffe.berkeleyvision.org/tu ...

  7. HTML5标签汇总及知识学习线路总结

    HTML5标签汇总,以及知识学习线路总结.

  8. 安全测试3_Web后端知识学习

    其实中间还应该学习下web服务和数据库的基础,对于web服务大家可以回家玩下tomcat或者wamp等东西,数据库的话大家掌握基本的增删该查就好了,另外最好掌握下数据库的内置函数,如:concat() ...

  9. GCC基础知识学习

    GCC基础知识学习 一.GCC编译选项解析 常用编译选项 命令格式:gcc [选项] [文件名] -E:仅执行编译预处理: -S:将C代码转换为汇编代码: -c:仅执行编译操作,不进行连接操作: -o ...

随机推荐

  1. bat 时间 的运算与提取

    比如在系统中date这个环境变量的值为 -- 星期六 年------%date:~,% 表示从左向右指针向右偏0位,然后从指针偏移到的位置开始提取4位字符,结果是2011 月------%date:~ ...

  2. vue2.0学习——使用webstorm创建一个vue项目

    https://blog.csdn.net/weixin_40877388/article/details/80911934

  3. JSP的九大对象和四大作用域

    1.JSP中九大内置对象为: request            请求对象          类型 javax.servlet.ServletRequest        作用域   Request ...

  4. hibernate5.3版本出现hibernate中The server time zone value“乱码”问题的解决办法。

    <!-- 配置关于数据库连接的四个项 driverClass url username password --> <property name="hibernate.con ...

  5. 大项目之网上书城(五)——主页(End)

    目录 大项目之网上书城(五)--主页(End) 主要改动 1.主页(终于完成啦) 完整代码 效果图 2.head.jsp的小改动 代码 3.login.jsp ###代码 效果图 4.login.js ...

  6. 零基础入门学习Python(10)--列表:一个打了激素的数组

    前言 有时候我们需要把一些东西暂时保存起来,因为他们有着一些直接或间接的联系,我们需要把它们放在某个组或者集合中,未来可能用得上. 很多接触过编程的朋友都知道,都接触过数组这个概念,那么数组这个概念事 ...

  7. Go:slice

    一.切片创建方式 func main() { // 创建切片方式1 // 让切片引用一个数组 array := [...]int{1, 2, 3, 4} slice1 := array[1:3] fm ...

  8. tomcat时间与系统时间不一致问题

    我在部署应用到centos系统上的tomcat服务器中运行,发现操作系统的时间和tomcat中的访问日志的时间与系统时间不一致,但是查看当前操作系统的时区也是CST时区(中国标准时区). 查看系统的时 ...

  9. 九度oj 题目1516:调整数组顺序使奇数位于偶数前面

    题目1516:调整数组顺序使奇数位于偶数前面 时间限制:1 秒 内存限制:128 兆 特殊判题:否 提交:3416 解决:1091 题目描述: 输入一个整数数组,实现一个函数来调整该数组中数字的顺序, ...

  10. Tyvj 1176 火焰巨魔的惆怅

    Tyvj 1176 火焰巨魔的惆怅 背景 TYVJ2月月赛第一道 巨魔家族在某天受到了其他种族的屠杀,作为一个英雄,他主动担任了断后的任务,但是,在巨魔家族整体转移过后,火焰巨魔却被困住了,他出逃的方 ...