一、Abstract

从近期对unsupervised learning 的研究得到启发,在large-scale setting 上,本文把unsupervised learning 与supervised learning结合起来,提高了supervised learning的性能。主要是把autoencoder与CNN结合起来

二、Key words:

SAE;SWWAE; reconstruction;encoder;decoder;VGG-16;Alex-Net

三、 Motivation

  1. reconstruction loss 很有用,reconstruction loss可以看作一个regularizer(SWWAE文中提到).
  2. unsupervised learning会对model起一定的限定作用,即相当于一个regularizer,这个regularizer使得encoder阶段提取得到的特征具有可解释性

四、Main contributions

  1. 本文实验表明了,high-capacity neural networks(采用了known switches)的 intermediate activations 可以保存input的大量信息,除了部分

    2.通过结合decoder pathway 的loss,提升了supervised learning model的分类正确率

    3.做了几个 autoencoder模型的对比实验,发现: the pooling switches and the layer-wise reconstruction loss 非常重要!

五、Inspired by

  1. Zhao, J., Mathieu, M., Goroshin, R., and Lecun, Y. Stacked what-where auto-encoders. ArXiv:1506.02351, 2015.
  2. Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In ICLR,2015.
  3. Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks.In NIPS, 2012.

    Rasmus, A., Valpola, H., Honkala, M., Berglund, M., and Raiko, T. Semi-supervised learning with ladder network.In NIPS, 2015.
  4. Adaptive deconvolutional networks for mid and high level feature learning
  5. Zeiler, M. D., Krishnan, D., Taylor, G. W., and Fergus, R. Deconvolutional networks. CVPR, 2010.
  6. Zeiler, M., Taylor, G., and Fergus, R. Adaptive deconvolu-tional networks for mid and high level feature learning.In ICCV, 2011.

key word:SWWAE;VGG-16;Alex-Net;ladder-Net;Deconvolutional network

六、文献具体实验及结果

1.SAE-all模型的训练:

第一步,采用VGG-16(训练好的VGG-16)初始化encoder,采用gaussian初始化decoder

第二步,固定encoder部分,用layerwise的方法训练decoder

第三步,用数据整体的训练更新decoder和encoder的参数

SAE-first模型的训练同SAE-all

SAE-layerwise一般只是拿来初始化 SAE-first SAE-all

SWWAE-all 提升了 1.66 % and 1.18% for single-crop and convolution schemes.

(top-1)

七、 感悟

  1. 2006~2010年期间, unsupervised learning 盛行是以为当时有标签数据不够大,所以需要用unsupervised leanring 的方法来初始化网络,可以取得较好效果,而 类似imagenet这样的大量标签数据的出现, 用autoencoder来初始化网络的优势已经没有。从这里也可以知道,当数据量较小时,可以考虑用unsupervised learning 的方法来初始化网络,从而提升分类准确率
  2. reconstruction loss 可以看作 regularization , 即是对enconder的weights做了一些限制,限制其获得的activations要能recon出input,是的提取得到的特征具有可解释性

【文献阅读】Augmenting Supervised Neural Networks with Unsupervised Objectives-ICML-2016的更多相关文章

  1. 【文献阅读】Self-Normalizing Neural Networks

    Self-Normalizing Neural Networks ,长达93页的附录足以成为吸睛的地方(给人感觉很厉害), 此paper提出了新的激活函数,称之为 SELUs ,其具有normaliz ...

  2. 论文阅读 Streaming Graph Neural Networks

    3 Streaming Graph Neural Networks link:https://dl.acm.org/doi/10.1145/3397271.3401092 Abstract 本文提出了 ...

  3. [ufldl]Supervised Neural Networks

    要实现的部分为:forward prop, softmax函数的cost function,每一层的gradient,以及penalty cost和gradient. forwad prop forw ...

  4. [C3] Andrew Ng - Neural Networks and Deep Learning

    About this Course If you want to break into cutting-edge AI, this course will help you do so. Deep l ...

  5. [Converge] Training Neural Networks

    CS231n Winter 2016: Lecture 5: Neural Networks Part 2 CS231n Winter 2016: Lecture 6: Neural Networks ...

  6. An Intuitive Explanation of Convolutional Neural Networks

    https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/ An Intuitive Explanation of Convolu ...

  7. 论文笔记之:Learning Multi-Domain Convolutional Neural Networks for Visual Tracking

    Learning Multi-Domain Convolutional Neural Networks for Visual Tracking CVPR 2016 本文提出了一种新的CNN 框架来处理 ...

  8. How to Use Convolutional Neural Networks for Time Series Classification

    How to Use Convolutional Neural Networks for Time Series Classification 2019-10-08 12:09:35 This blo ...

  9. 《Graph Neural Networks: A Review of Methods and Applications》阅读笔记

    本文是对文献 <Graph Neural Networks: A Review of Methods and Applications> 的内容总结,详细内容请参照原文. 引言 大量的学习 ...

随机推荐

  1. Understand:高效代码静态分析神器详解(一)【转】

    转自:http://www.codemx.cn/2016/04/30/Understand01/ 之前用Windows系统,一直用source insight查看代码非常方便,但是年前换到mac下面, ...

  2. xammp无法启动apache 由于80端口引起的问题 摘自百度经验

    启动过程提示: 15:33:05 [Apache] Problem detected!15:33:05 [Apache] Port 80 in use by "Unable to open ...

  3. DB2 数据库中字段特定字符替换为空

    Update RM_CarInfo set ImportTitle = Replace(ImportTitle,'ZD','') WHERE ImportTitle LIKE'%ZD%';

  4. C# .NET4.0 改为 到.NET2.0 时 TypedTableBase 报错解决方法

    .NET 4.0 降版本 到.NET 2.0.不出意外,问题必然来了. 编译错误一: 错误 1 命名空间“System”中不存在类型或命名空间名称“Linq”(是缺少程序集引用吗?)解决: 删掉该引用 ...

  5. centos 7 mysql 离线安装教程

    1. 解压下载的zip包,会发现有以下几个rpm包: MySQL-client-advanced-5.6.22-1.el7.x86_64.rpm MySQL-devel-advanced-5.6.22 ...

  6. python正则表达式从路径中取文件名出来不加后缀(txt)

    正则表达式[^\\/:*?"<>|\r\n]+$ ---->取文件名包括后缀 e.g. >>>D:\PyCharm 2018.2.4\pythonWork ...

  7. 各语言最原始数据库访问组件封装DBHelper

    源码:https://github.com/easonjim/DBHelper bug提交:https://github.com/easonjim/DBHelper/issues 每个语言放在不同的分 ...

  8. 比JSONKit还要快的第三方JSON解析器NextiveJson

    这款比JSONKit还好用,效率跟iOS5原生的差不多,不过解析后对内存的释放比原生的要多.所以推荐 https://github.com/nextive/NextiveJson 顺便提一下解析XML ...

  9. Python操作sqlite数据库小节

    学习了Python操作sqlite数据库,做一个小结,以备后用. import sqlite3import os# 进行数据库操作时,主要是参数如何传输try:# 链接数据库conn=sqlite3. ...

  10. 常用 linux操作

    查看libreoffice进程 ps -ef | grep libreoffice