Training Neural Networks: Q&A with Ian Goodfellow, Google
Training Neural Networks: Q&A with Ian Goodfellow, Google
Neural networks require considerable time and computational firepower to train. Previously, researchers believed that neural networks were costly to train because gradient descent slows down near local minima or saddle points. At the RE.WORK Deep Learning Summit in San Francisco, Ian Goodfellow, Research Scientist at Google, will challenge that view and look deeper to find the true bottlenecks in neural network training.
Before joining the Google team, Ian earned a PhD in machine learning from Université de Montréal, under his advisors Yoshua Bengio and Aaron Courville. During his studies, which were funded by the Google PhD Fellowship in Deep Learning, he wrote Pylearn2, the open source deep learning research library, and introduced a variety of new deep learning algorithms. Previously, he obtained a BSc and MSc in Computer Science from Stanford University, where he was one of the earliest members of Andrew Ng's deep learning research group.
We caught up with Ian ahead of the summit in January 2016 to hear more about his current work and thoughts on the future of deep learning.
What are you currently working on in deep networks?
I am interested in developing generic methods that make any neural network train faster and generalize better. To improve generalization, I study the way neural networks respond to “adversarial examples” that are intentionally constructed to confuse the network. To improve optimization, I study the structure of neural network optimization problems and determine which factors cause learning to be slow.
What are the key factors that have enabled recent advancements in deep learning?
The basic machine learning algorithms have been in place since the 1980s, but until very recently, we were applying these algorithms to neural networks with fewer neurons than a leech. Unsurprisingly, such small networks performed poorly. Fast computers with larger memory capacity and better software infrastructure have allowed us to train neural networks that are large enough to perform well. Larger datasets are also very important. Some changes in machine learning algorithms, like designing neural network layers to be very linear, have also led to noticeable improvements.
What are the main types of problems now being addressed in the deep learning space?
There is a gold rush to be the first to use existing deep learning algorithms on new application areas. Every day, there are new articles about deep learning for counting calories from photos, deep learning for separating two voices in a recording, etc.
What are the practical applications of your work and what sectors are most likely to be affected?
My work is generic enough that it impacts everything we use neural networks for. Anything you want to do with a neural net, I aim to make faster and more accurate.
What developments can we expect to see in deep learning in the next 5 years?
I expect within five years, we will have neural networks that can summarize what happens in a video clip, and will be able to generate short videos. Neural networks are already the standard solution to vision tasks. I expect they will become the standard solution to NLP and robotics tasks as well. I also predict that neural networks will become an important tool in other scientific disciplines. For example, neural networks could be trained to model the behavior of genes, drugs, and proteins and then used to design new medicines.
What advancements excite you most in the field?
Recent extensions of variational auto-encoders and generative adversarial networks have greatly improved the ability of neural networks to generate realistic images. Generating data has been a constantly studied problem for decades, and we still do not seem to have the right algorithm to do it. The last year or so has shown that we are getting much closer though.
Ian Goodfellow will be speaking at Deep Learning Summit in San Francisco, on 28-29 January 2016, alongside speakers from Baidu, Twitter, Clarifai, MIT and more.
Training Neural Networks: Q&A with Ian Goodfellow, Google的更多相关文章
- 实现径向变换用于样本增强《Training Neural Networks with Very Little Data-A Draft》
背景: 做大规模机器学习算法,特别是神经网络最怕什么--没有数据!!没有数据意味着,机器学不会,人工不智能!通常使用样本增强来扩充数据一直都是解决这个问题的一个好方法. 最近的一篇论文<Trai ...
- (转)A Recipe for Training Neural Networks
A Recipe for Training Neural Networks Andrej Karpathy blog 2019-04-27 09:37:05 This blog is copied ...
- 1506.01186-Cyclical Learning Rates for Training Neural Networks
1506.01186-Cyclical Learning Rates for Training Neural Networks 论文中提出了一种循环调整学习率来训练模型的方式. 如下图: 通过循环的线 ...
- A Recipe for Training Neural Networks [中文翻译, part 1]
最近拜读大神Karpathy的经验之谈 A Recipe for Training Neural Networks https://karpathy.github.io/2019/04/25/rec ...
- [Converge] Training Neural Networks
CS231n Winter 2016: Lecture 5: Neural Networks Part 2 CS231n Winter 2016: Lecture 6: Neural Networks ...
- [CS231n-CNN] Training Neural Networks Part 1 : activation functions, weight initialization, gradient flow, batch normalization | babysitting the learning process, hyperparameter optimization
课程主页:http://cs231n.stanford.edu/ Introduction to neural networks -Training Neural Network ________ ...
- [转]Binarized Neural Networks_ Training Neural Networks with Weights and Activations Constrained to +1 or −1
原文: 二值神经网络(Binary Neural Network,BNN) 在我刚刚过去的研究生毕设中,我在ImageNet数据集上验证了图像特征二值化后仍然具有很强的表达能力,可以在检索中达到较好的 ...
- [CS231n-CNN] Training Neural Networks Part 1 : parameter updates, ensembles, dropout
课程主页:http://cs231n.stanford.edu/ ___________________________________________________________________ ...
- Binarized Neural Networks_ Training Neural Networks with Weights and Activations Constrained to +1 or −1
转载请注明出处: http://www.cnblogs.com/sysuzyq/p/6248953.html by 少侠阿朱
随机推荐
- JAVA学习IO(1)
面向过程和面向对象的区别:面向过程:把问题分析成一个一个步骤组成的过程面向对象:从一个问题中分析出各个功能对象,并描述各个功能在整个解决问题的步骤的行为.面向对象的3大特征:封装,继承,多态封装:把多 ...
- Arduino IDE 安装esp8266 2.4.rc2的编译环境
2.4. 版本, 有一个我需要的功能, 串口缓存, 可以修改. Serial.setRxBufferSize(1024); //修改为1024个字节. 安装步骤: 1. 需要FQ. 推荐用" ...
- grunt入门讲解3:实例讲解使用 Gruntfile 配置任务
这个Gruntfile 实例使用到了5个 Grunt 插件: grunt-contrib-uglify grunt-contrib-qunitgrunt-contrib-concatgrun ...
- 第八周PSP&进度条
团队项目PSP 一.表格: C类型 C内容 S开始时间 E结束时间 I时间间隔 T净时间(mins) 预计花费时间(mins) 讨论 讨论各个模块页面设计 9:30 12:30 站立会议 分配 ...
- js dom学习
创建dom元素 var oLi = document.creteElement('li'); //创建livar aLi = oUl.getElementsByTagName('li');oLi.in ...
- sublime 对vue的高亮显示
前提概要: sublime的下载地址:http://www.sublimetext.com/ notepad++下载地址:https://notepad-plus-plus.org/ .vue的文件在 ...
- JAVA相关概念(一)
依赖注入和控制反转 首先,这两个词是同一个概念的不同角度的说法,依赖注入感觉是对描述了如何实现,而控制反转则像是描述了一种思想. 依赖注入的流行可以说是由spring的流行带动的,只要是做过sprin ...
- BZOJ3156 防御准备(动态规划+斜率优化)
设f[i]为在i放置守卫塔时1~i的最小花费.那么显然f[i]=min(f[j]+(i-j)*(i-j-1)/2)+a[i]. 显然这是个斜率优化入门题.将不与i.j同时相关的提出,得f[i]=min ...
- 使用URLConnection发送http请求实现简单爬虫(可以配置代理)
import java.io.BufferedReader;import java.io.IOException;import java.io.InputStreamReader;import jav ...
- STL Deque 容器
STL Deque 容器 Deque简介 deque是“double-ended queue”的缩写,和vector一样都是STL的容器,deque是双 端的,而vector是单端的. ...