caffe中的Local Response Normalization (LRN)有什么用,和激活函数区别
http://stats.stackexchange.com/questions/145768/importance-of-local-response-normalization-in-cnn
caffe 解释:
The local response normalization layer performs a kind of “lateral inhibition” by normalizing over local input regions.双边抑制。看起来就像是激活函数
几种解释以上链接的几个答复,引用附上如上链接:
(1)是优化的计算更加快
Here is my suggested answer, though I don't claim to be knowledgeable. When performing gradient descent on a linear model, the error surface is quadratic, with the curvature determined by XXTXXT, where XX is your input. Now the ideal error surface for or gradient descent has the same curvature in all directions (otherwise the step size is too small in some directions and too big in others). Normalising your inputs by rescaling the inputs to mean zero, variance 1 helps and is fast:now the directions along each dimension all have the same curvature, which in turn bounds the curvature in other directions.
The optimal solution would be to sphere/whiten the inputs to each neuron, however this is computationally too expensive. LCN can be justified as an approximate whitening based on the assumption of a high degree of correlation between neighbouring pixels (or channels) So I would claim the benefit is that the error surface is more benign for SGD... A single Learning rate works well across the input dimensions (of each neuron)
(2)在NIPS 2012这篇文章中,提到ReLU可以不需要这个归一化,但为了一般化,仍加上这个使之generalization。效果提高2%(没说明是什么激活函数)
Indeed, there seems no good explanation in a single place. The best is to read the articles from where it comes:
The original AlexNet article explains a bit in Section 3.3:
Krizhevsky, Sutskever, and Hinton, ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012. www.cs.toronto.edu/~fritz/absps/imagenet.pdf
The exact way of doing this was proposed in (but not much extra info here):
Kevin Jarrett, Koray Kavukcuoglu, Marc’Aurelio Ranzato and Yann LeCun, What is the best Multi-Stage Architecture for Object Recognition?, ICCV 2009. yann.lecun.com/exdb/publis/pdf/jarrett-iccv-09.pdf
It was inspired by computational neuroscience:
S. Lyu and E. Simoncelli. Nonlinear image representation using divisive normalization. CVPR 2008. www.cns.nyu.edu/pub/lcv/lyu08b.pdf . This paper goes deeper into the math, and is in accordance with the answer of seanv507.
[24] N. Pinto, D. D. Cox, and J. J. DiCarlo. Why is real-world vi- sual object recognition hard? PLoS Computational Biology, 2008. http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.0040027
caffe中的Local Response Normalization (LRN)有什么用,和激活函数区别的更多相关文章
- 局部响应归一化(Local Response Normalization,LRN)
版权声明:本文为博主原创文章,欢迎转载,注明地址. https://blog.csdn.net/program_developer/article/details/79430119 一.LRN技术介 ...
- Local Response Normalization 60 million parameters and 500,000 neurons
CNN是工具,在图像识别中是发现图像中待识别对象的特征的工具,是剔除对识别结果无用信息的工具. ImageNet Classification with Deep Convolutional Neur ...
- Local Response Normalization作用——对局部神经元的活动创建竞争机制,使得其中响应比较大的值变得相对更大,并抑制其他反馈较小的神经元,增强了模型的泛化能力
AlexNet将LeNet的思想发扬光大,把CNN的基本原理应用到了很深很宽的网络中.AlexNet主要使用到的新技术点如下. (1)成功使用ReLU作为CNN的激活函数,并验证其效果在较深的网络超过 ...
- caffe中添加local层
下载caffe-local,解压缩; 修改makefile.config:我是将cuudn注释掉,去掉cpu_only的注释; make all make test(其中local_test出错,将文 ...
- caffe 中base_lr、weight_decay、lr_mult、decay_mult代表什么意思?
在机器学习或者模式识别中,会出现overfitting,而当网络逐渐overfitting时网络权值逐渐变大,因此,为了避免出现overfitting,会给误差函数添加一个惩罚项,常用的惩罚项是所有权 ...
- 【神经网络与深度学习】如何在Caffe中配置每一个层的结构
如何在Caffe中配置每一个层的结构 最近刚在电脑上装好Caffe,由于神经网络中有不同的层结构,不同类型的层又有不同的参数,所有就根据Caffe官网的说明文档做了一个简单的总结. 1. Vision ...
- LRN(local response normalization--局部响应标准化)
LRN全称为Local Response Normalization,即局部响应归一化层,LRN函数类似DROPOUT和数据增强作为relu激励之后防止数据过拟合而提出的一种处理方法.这个函数很少使用 ...
- 浅谈caffe中train_val.prototxt和deploy.prototxt文件的区别
本文以CaffeNet为例: 1. train_val.prototxt 首先,train_val.prototxt文件是网络配置文件.该文件是在训练的时候用的. 2.deploy.prototxt ...
- Caffe中deploy.prototxt 和 train_val.prototxt 区别
之前用deploy.prototxt 还原train_val.prototxt过程中,遇到了坑,所以打算总结一下 本人以熟悉的LeNet网络结构为例子 不同点主要在一前一后,相同点都在中间 train ...
随机推荐
- ubuntu18.04 64bit如何安装docker
注:参考自https://docs.docker.com/install/linux/docker-ce/ubuntu/ 1.卸载旧版本docker(如果之前安装了) sudo apt-get rem ...
- LeetCode——largest-rectangle-in-histogram1
Question Given n non-negative integers representing the histogram's bar height where the width of ea ...
- 解题报告:codeforce 7C Line
codeforce 7C C. Line time limit per test1 second memory limit per test256 megabytes A line on the pl ...
- 转载- ACM常见的各种说法
from : http://blog.csdn.net/qq_15015129/article/details/52738184 1.答案错误 —— wrong answer 就是最常见的.这个没办法 ...
- [小问题笔记(五)] 用SQL加密字符串(MD5、SHA1),顺便解决读取数据加密后不一样的问题
这里用到SQL Server内置的函数 HashBytes(). select HashBytes('MD5','bubu') select HashBytes('SHA1','bubu') 以MD5 ...
- spring boot2.1读取 apollo 配置中心3
上篇记录了springboot读取apollo的配置信息,以及如何获取服务端的推送更新配置. 接下来记录一下,如何获取公共namespace的配置. 上文中使用如下代码共聚公共命名空间的配置: @Ap ...
- JAVA初学者(一)
2015-12-15 21:26:17 刚学的java 做个总结: 1.构造函数没有返回值. 2.A对象调用Q的方法,Q方法里的变量就是A的变量 Fraction add(Fraction f) 在 ...
- 什么是ZooKeeper(一)(通俗易懂)
以前在做别的项目时用过zk,但没有过多深入的学习,本着通俗易懂.简单方便学习成本低的方式,建议大家耐心看完,如果文章中有不清楚的地方,可发私信进步探讨! 学习zk共分为二部分,第一部分主要以理论为主. ...
- 分布式系统中的幂等性-zookeeper与dubbo
现如今我们的系统大多拆分为分布式SOA,或者微服务,一套系统中包含了多个子系统服务,而一个子系统服务往往会去调用另一个服务,而服务调用服务无非就是使用RPC通信或者restful,既然是通信,那么就有 ...
- IE 中的 button type默认值问题
今天遇到一个问题. 将项目页面的渲染模式从 IE7 改为 IE10 后(<meta http-equiv="X-UA-Compatible" content="IE ...