【文献阅读】Augmenting Supervised Neural Networks with Unsupervised Objectives-ICML-2016
一、Abstract
从近期对unsupervised learning 的研究得到启发,在large-scale setting 上,本文把unsupervised learning 与supervised learning结合起来,提高了supervised learning的性能。主要是把autoencoder与CNN结合起来
二、Key words:
SAE;SWWAE; reconstruction;encoder;decoder;VGG-16;Alex-Net
三、 Motivation
- reconstruction loss 很有用,reconstruction loss可以看作一个regularizer(SWWAE文中提到).
- unsupervised learning会对model起一定的限定作用,即相当于一个regularizer,这个regularizer使得encoder阶段提取得到的特征具有可解释性
四、Main contributions
- 本文实验表明了,high-capacity neural networks(采用了known switches)的 intermediate activations 可以保存input的大量信息,除了部分
2.通过结合decoder pathway 的loss,提升了supervised learning model的分类正确率
3.做了几个 autoencoder模型的对比实验,发现: the pooling switches and the layer-wise reconstruction loss 非常重要!
五、Inspired by
- Zhao, J., Mathieu, M., Goroshin, R., and Lecun, Y. Stacked what-where auto-encoders. ArXiv:1506.02351, 2015.
- Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In ICLR,2015.
- Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks.In NIPS, 2012.
Rasmus, A., Valpola, H., Honkala, M., Berglund, M., and Raiko, T. Semi-supervised learning with ladder network.In NIPS, 2015. - Adaptive deconvolutional networks for mid and high level feature learning
- Zeiler, M. D., Krishnan, D., Taylor, G. W., and Fergus, R. Deconvolutional networks. CVPR, 2010.
- Zeiler, M., Taylor, G., and Fergus, R. Adaptive deconvolu-tional networks for mid and high level feature learning.In ICCV, 2011.
key word:SWWAE;VGG-16;Alex-Net;ladder-Net;Deconvolutional network
六、文献具体实验及结果
1.SAE-all模型的训练:
第一步,采用VGG-16(训练好的VGG-16)初始化encoder,采用gaussian初始化decoder
第二步,固定encoder部分,用layerwise的方法训练decoder
第三步,用数据整体的训练更新decoder和encoder的参数
SAE-first模型的训练同SAE-all
SAE-layerwise一般只是拿来初始化 SAE-first SAE-all
SWWAE-all 提升了 1.66 % and 1.18% for single-crop and convolution schemes.
(top-1)
七、 感悟
- 2006~2010年期间, unsupervised learning 盛行是以为当时有标签数据不够大,所以需要用unsupervised leanring 的方法来初始化网络,可以取得较好效果,而 类似imagenet这样的大量标签数据的出现, 用autoencoder来初始化网络的优势已经没有。从这里也可以知道,当数据量较小时,可以考虑用unsupervised learning 的方法来初始化网络,从而提升分类准确率
- reconstruction loss 可以看作 regularization , 即是对enconder的weights做了一些限制,限制其获得的activations要能recon出input,是的提取得到的特征具有可解释性
【文献阅读】Augmenting Supervised Neural Networks with Unsupervised Objectives-ICML-2016的更多相关文章
- 【文献阅读】Self-Normalizing Neural Networks
Self-Normalizing Neural Networks ,长达93页的附录足以成为吸睛的地方(给人感觉很厉害), 此paper提出了新的激活函数,称之为 SELUs ,其具有normaliz ...
- 论文阅读 Streaming Graph Neural Networks
3 Streaming Graph Neural Networks link:https://dl.acm.org/doi/10.1145/3397271.3401092 Abstract 本文提出了 ...
- [ufldl]Supervised Neural Networks
要实现的部分为:forward prop, softmax函数的cost function,每一层的gradient,以及penalty cost和gradient. forwad prop forw ...
- [C3] Andrew Ng - Neural Networks and Deep Learning
About this Course If you want to break into cutting-edge AI, this course will help you do so. Deep l ...
- [Converge] Training Neural Networks
CS231n Winter 2016: Lecture 5: Neural Networks Part 2 CS231n Winter 2016: Lecture 6: Neural Networks ...
- An Intuitive Explanation of Convolutional Neural Networks
https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/ An Intuitive Explanation of Convolu ...
- 论文笔记之:Learning Multi-Domain Convolutional Neural Networks for Visual Tracking
Learning Multi-Domain Convolutional Neural Networks for Visual Tracking CVPR 2016 本文提出了一种新的CNN 框架来处理 ...
- How to Use Convolutional Neural Networks for Time Series Classification
How to Use Convolutional Neural Networks for Time Series Classification 2019-10-08 12:09:35 This blo ...
- 《Graph Neural Networks: A Review of Methods and Applications》阅读笔记
本文是对文献 <Graph Neural Networks: A Review of Methods and Applications> 的内容总结,详细内容请参照原文. 引言 大量的学习 ...
随机推荐
- Understand:高效代码静态分析神器详解(一)【转】
转自:http://www.codemx.cn/2016/04/30/Understand01/ 之前用Windows系统,一直用source insight查看代码非常方便,但是年前换到mac下面, ...
- xammp无法启动apache 由于80端口引起的问题 摘自百度经验
启动过程提示: 15:33:05 [Apache] Problem detected!15:33:05 [Apache] Port 80 in use by "Unable to open ...
- DB2 数据库中字段特定字符替换为空
Update RM_CarInfo set ImportTitle = Replace(ImportTitle,'ZD','') WHERE ImportTitle LIKE'%ZD%';
- C# .NET4.0 改为 到.NET2.0 时 TypedTableBase 报错解决方法
.NET 4.0 降版本 到.NET 2.0.不出意外,问题必然来了. 编译错误一: 错误 1 命名空间“System”中不存在类型或命名空间名称“Linq”(是缺少程序集引用吗?)解决: 删掉该引用 ...
- centos 7 mysql 离线安装教程
1. 解压下载的zip包,会发现有以下几个rpm包: MySQL-client-advanced-5.6.22-1.el7.x86_64.rpm MySQL-devel-advanced-5.6.22 ...
- python正则表达式从路径中取文件名出来不加后缀(txt)
正则表达式[^\\/:*?"<>|\r\n]+$ ---->取文件名包括后缀 e.g. >>>D:\PyCharm 2018.2.4\pythonWork ...
- 各语言最原始数据库访问组件封装DBHelper
源码:https://github.com/easonjim/DBHelper bug提交:https://github.com/easonjim/DBHelper/issues 每个语言放在不同的分 ...
- 比JSONKit还要快的第三方JSON解析器NextiveJson
这款比JSONKit还好用,效率跟iOS5原生的差不多,不过解析后对内存的释放比原生的要多.所以推荐 https://github.com/nextive/NextiveJson 顺便提一下解析XML ...
- Python操作sqlite数据库小节
学习了Python操作sqlite数据库,做一个小结,以备后用. import sqlite3import os# 进行数据库操作时,主要是参数如何传输try:# 链接数据库conn=sqlite3. ...
- 常用 linux操作
查看libreoffice进程 ps -ef | grep libreoffice