这篇论文思路简单、易实现、效果好,是一篇难得的佳作。从实现的角度理解,就是做了以下两个替换:

  1. 将线性卷积替换为多层感知机(某种程度上,线性卷积可以认为识一层感知机)。
  2. 将全连接层用global average pooling layer替换。

下面我们就来分析引入上述两个替换的妙处。首先分析第一个替换的妙处,替换的效果(图示如下)

论文中提到“The linear convolution is sufficient for abstraction when the instances of the latent concepts are linearly separable.”,然而现实太复杂,the instances of the latent concepts通常不是线性可分的。在这种情况下,通常有两种做法:一是,引入大量的linear convolution(以体量应对复杂现实);二是,干脆寻找一个能够模拟任意复杂情形的“参数化函数”(以灵活性应对复杂现实)。

可以预见,如果你计算、存储资源充裕的话,你可以采取简单暴力的第一种情形;通常情况下,计算、存储资源受限,因此第二种做法更加接近现实一点(也更容易将算法植入到计算、存储资源有限的移动设备上,如手机)。下面的问题就是寻找所需的“参数化函数”。庆幸的是,多层感知机在某种程度上能够满足我们的需求,此外它能够与BP算法完美兼容(这篇论文选择的就是多层感知机)。这样的Mlpconv layer就可以作为深度网络的几个基本block,用以构建深度网络。

在CNN当中,随着层数的加深,我们得到的特征越来越抽象。这种抽象是以组合较低一层抽象特征得到的。从这个角度理解,如果在较低层就能够比之前对应层更抽象的特征,然后整个网络的输出抽象程度将会变得更高,这样高度抽象的特征对于分类、任务迁移都是有极大帮助的。

下面分析第二个替换的妙处

传统的CNN是将最后一层的卷积输出向量化,然后输入到全连接层,全连接层之后是常用的分类损失函数,如softmax。如果最后一层卷积输出特征维度过高、类别较多,那么这一块引入的参数量是很大的,这会造成网络过拟合(还好,目前有一些防止过拟合的手段,如dropout)。

“The idea is to generate one feature map for each corresponding category of the classification task in the last mlpconv layer. Instead of adding fully connected layers on top od the feature maps, we take the advantage of each feature map, and the resulting vector is fed directly into the softmax layer”,这样做的好处是,直接在类别与feature maps之间建立了联系,“The features maps can be easily interpreted as categories confidence maps”。此外,这里没有引入要学习的参数,也间接起到了防止过拟合的效果。

在Caffe框架下实现上述网络是一个很简单的事情,以在cifar10上的网络结果为例

layers {
name: "conv1"
type: CONVOLUTION
bottom: "data"
top: "conv1"
blobs_lr:
blobs_lr:
weight_decay: .
weight_decay: .
convolution_param {
num_output:
pad:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
}
}
}
layers {
name: "relu1"
type: RELU
bottom: "conv1"
top: "conv1"
}
layers {
name: "cccp1"
type: CONVOLUTION
bottom: "conv1"
top: "cccp1"
blobs_lr:
blobs_lr:
weight_decay:
weight_decay:
convolution_param {
num_output:
group:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
value:
}
}
}
layers {
name: "relu_cccp1"
type: RELU
bottom: "cccp1"
top: "cccp1"
}
layers {
name: "cccp2"
type: CONVOLUTION
bottom: "cccp1"
top: "cccp2"
blobs_lr:
blobs_lr:
weight_decay:
weight_decay:
convolution_param {
num_output:
group:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
value:
}
}
}
layers {
name: "relu_cccp2"
type: RELU
bottom: "cccp2"
top: "cccp2"
}

两个kernel_size为1的卷积核实现的就是多层感知机的功能,全部的网络结果代码如下

name: "CIFAR10_full"
layers {
name: "cifar"
type: DATA
top: "data"
top: "label"
data_param {
source: "cifar-train-leveldb"
batch_size:
}
include: { phase: TRAIN }
}
layers {
name: "cifar"
type: DATA
top: "data"
top: "label"
data_param {
source: "cifar-test-leveldb"
batch_size:
}
include: { phase: TEST }
}
layers {
name: "conv1"
type: CONVOLUTION
bottom: "data"
top: "conv1"
blobs_lr:
blobs_lr:
weight_decay: .
weight_decay: .
convolution_param {
num_output:
pad:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
}
}
}
layers {
name: "relu1"
type: RELU
bottom: "conv1"
top: "conv1"
}
layers {
name: "cccp1"
type: CONVOLUTION
bottom: "conv1"
top: "cccp1"
blobs_lr:
blobs_lr:
weight_decay:
weight_decay:
convolution_param {
num_output:
group:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
value:
}
}
}
layers {
name: "relu_cccp1"
type: RELU
bottom: "cccp1"
top: "cccp1"
}
layers {
name: "cccp2"
type: CONVOLUTION
bottom: "cccp1"
top: "cccp2"
blobs_lr:
blobs_lr:
weight_decay:
weight_decay:
convolution_param {
num_output:
group:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
value:
}
}
}
layers {
name: "relu_cccp2"
type: RELU
bottom: "cccp2"
top: "cccp2"
}
layers {
name: "pool1"
type: POOLING
bottom: "cccp2"
top: "pool1"
pooling_param {
pool: MAX
kernel_size:
stride:
}
}
layers {
name: "drop3"
type: DROPOUT
bottom: "pool1"
top: "pool1"
dropout_param {
dropout_ratio: 0.5
}
}
layers {
name: "conv2"
type: CONVOLUTION
bottom: "pool1"
top: "conv2"
blobs_lr:
blobs_lr:
weight_decay: .
weight_decay: .
convolution_param {
num_output:
pad:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
}
}
}
layers {
name: "relu2"
type: RELU
bottom: "conv2"
top: "conv2"
}
layers {
name: "cccp3"
type: CONVOLUTION
bottom: "conv2"
top: "cccp3"
blobs_lr:
blobs_lr:
weight_decay:
weight_decay:
convolution_param {
num_output:
group:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
value:
}
}
}
layers {
name: "relu_cccp3"
type: RELU
bottom: "cccp3"
top: "cccp3"
}
layers {
name: "cccp4"
type: CONVOLUTION
bottom: "cccp3"
top: "cccp4"
blobs_lr:
blobs_lr:
weight_decay:
weight_decay:
convolution_param {
num_output:
group:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
value:
}
}
}
layers {
name: "relu_cccp4"
type: RELU
bottom: "cccp4"
top: "cccp4"
}
layers {
name: "pool2"
type: POOLING
bottom: "cccp4"
top: "pool2"
pooling_param {
pool: AVE
kernel_size:
stride:
}
}
layers {
name: "drop6"
type: DROPOUT
bottom: "pool2"
top: "pool2"
dropout_param {
dropout_ratio: 0.5
}
}
layers {
name: "conv3"
type: CONVOLUTION
bottom: "pool2"
top: "conv3"
blobs_lr: .
blobs_lr: .
weight_decay: .
weight_decay: .
convolution_param {
num_output:
pad:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
}
}
}
layers {
name: "relu3"
type: RELU
bottom: "conv3"
top: "conv3"
}
layers {
name: "cccp5"
type: CONVOLUTION
bottom: "conv3"
top: "cccp5"
blobs_lr:
blobs_lr:
weight_decay:
weight_decay:
convolution_param {
num_output:
group:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
value:
}
}
}
layers {
name: "relu_cccp5"
type: RELU
bottom: "cccp5"
top: "cccp5"
}
layers {
name: "cccp6"
type: CONVOLUTION
bottom: "cccp5"
top: "cccp6"
blobs_lr: 0.1
blobs_lr: 0.1
weight_decay:
weight_decay:
convolution_param {
num_output:
group:
kernel_size:
weight_filler {
type: "gaussian"
std: 0.05
}
bias_filler {
type: "constant"
value:
}
}
}
layers {
name: "relu_cccp6"
type: RELU
bottom: "cccp6"
top: "cccp6"
}
layers {
name: "pool3"
type: POOLING
bottom: "cccp6"
top: "pool3"
pooling_param {
pool: AVE
kernel_size:
stride:
}
}
layers {
name: "accuracy"
type: ACCURACY
bottom: "pool3"
bottom: "label"
top: "accuracy"
include: { phase: TEST }
}
layers {
name: "loss"
type: SOFTMAX_LOSS
bottom: "pool3"
bottom: "label"
top: "loss"
}

总结:这篇文章引入的改进网络结构的方式、global average pooling启发了后续很多算法,以后有时间再慢慢分析。

论文笔记 Network In Network的更多相关文章

  1. 论文笔记系列-Neural Network Search :A Survey

    论文笔记系列-Neural Network Search :A Survey 论文 笔记 NAS automl survey review reinforcement learning Bayesia ...

  2. 论文笔记-Deep Affinity Network for Multiple Object Tracking

    作者: ShijieSun, Naveed Akhtar, HuanShengSong, Ajmal Mian, Mubarak Shah 来源: arXiv:1810.11780v1 项目:http ...

  3. 论文笔记——N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning

    论文地址:https://arxiv.org/abs/1709.06030 1. 论文思想 利用强化学习,对网络进行裁剪,从Layer Removal和Layer Shrinkage两个维度进行裁剪. ...

  4. 【论文笔记】Malware Detection with Deep Neural Network Using Process Behavior

    [论文笔记]Malware Detection with Deep Neural Network Using Process Behavior 论文基本信息 会议: IEEE(2016 IEEE 40 ...

  5. 论文笔记: Dual Deep Network for Visual Tracking

    论文笔记: Dual Deep Network for Visual Tracking  2017-10-17 21:57:08  先来看文章的流程吧 ... 可以看到,作者所总结的三个点在于: 1. ...

  6. Face Aging with Conditional Generative Adversarial Network 论文笔记

    Face Aging with Conditional Generative Adversarial Network 论文笔记 2017.02.28  Motivation: 本文是要根据最新的条件产 ...

  7. 论文《Network in Network》笔记

    论文:Lin M, Chen Q, Yan S. Network In Network[J]. Computer Science, 2013. 参考:关于CNN中1×1卷积核和Network in N ...

  8. 论文笔记 《Maxout Networks》 && 《Network In Network》

    论文笔记 <Maxout Networks> && <Network In Network> 发表于 2014-09-22   |   1条评论 出处 maxo ...

  9. [论文阅读笔记] Structural Deep Network Embedding

    [论文阅读笔记] Structural Deep Network Embedding 本文结构 解决问题 主要贡献 算法原理 参考文献 (1) 解决问题 现有的表示学习方法大多采用浅层模型,这可能不能 ...

  10. [论文阅读笔记] Unsupervised Attributed Network Embedding via Cross Fusion

    [论文阅读笔记] Unsupervised Attributed Network Embedding via Cross Fusion 本文结构 解决问题 主要贡献 算法原理 实验结果 参考文献 (1 ...

随机推荐

  1. windows平台把UliPad添加到右键菜单

    对.py文件支持右键用UliPad打开方式支持: 1.打开注册表(win+R,运行框输入regedit) 2.先对*.py文件进行设置.找到注册表目录HKEY_CLASSES_ROOT\Python. ...

  2. Linux - 进程调度算法

    进程调度: 无论是在批处理系统还是分时系统中,用户进程数一般都多于处理机数.这将导致它们互相争夺处理机.另外,系统进程也同样需要使用处理机. 这就要求进程调度程序按一定的策略,动态地把处理机分配给处于 ...

  3. 浅谈聚类算法(K-means)

    聚类算法(K-means)目的是将n个对象根据它们各自属性分成k个不同的簇,使得簇内各个对象的相似度尽可能高,而各簇之间的相似度尽量小. 而如何评测相似度呢,采用的准则函数是误差平方和(因此也叫K-均 ...

  4. App启动页倒计时功能

    转载请注明出处:http://www.cnblogs.com/cnwutianhao/p/6753418.html 示例代码采用 RxJava + RxLifecycle + Data-Binding ...

  5. .NET枚举类型转为List类型

    如图所示这个竞卖状态,原先是在前端界面通过html代码写死的几个状态,现在需要改为动态加载.这个几个状态是定义的枚举类型. 1:定义一个枚举类型 /// <summary>    /// ...

  6. YARN资源调度策略之Capacity Scheduler

    背景 yarn默认使用的是最简单的FIFO调度器,即一个default队列,所有用户共享,分配资源也是先到先得,没有优先级之分.有时一两个任务就把资源全占了,其他任务吃不到资源造成饥饿,显然这样的资源 ...

  7. CODEVS上一道很有趣的题(2145 判断奇偶性)

    判断函数y=x^n次方的奇偶性若是奇函数就输出ji,偶函数输出ou 233333 用到了long long 还是爆了,于是就上了char a[1000000] =.= #include<stdi ...

  8. phpmyadmin的初始账号密码是多少

    问这个问题,是不是有点弱智,但是这个问题确实阻塞了我那么几分钟. 实际上问题很简单,初始账号是root,密码为空

  9. 给自己的QQ群开启腾讯官方的群聊机器人

    看到腾讯有个机器人还不错 能聊天 能唱歌 方法:1.点击链接填写https://wj.qq.com/s/946969/64ac 群聊机器人内测邀请.按照提示填就行了 2.然后就是开启机器人了 http ...

  10. Angular vs React 最全面深入对比

    如今,Angular和React这两个JavaScript框架可谓红的发紫,同时针对这两个框架的选择变成了当下最容易被问及或者被架构设计者考虑的问题,本文或许无法告诉你哪个框架更优秀,但尽量从更多的角 ...