Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week2, Optimization algorithms
Gradient descent
Batch Gradient Decent, Mini-batch gradient descent, Stochastic gradient descent
还有很多比gradient decent 更优化的算法,在了解这些算法前,需要先理解 Exponentially weighted averages 这个概念
Exponentially weighted average 是一种计算平均值的方法,非常省storage 和 memory, 但是不是很精确。 然后引出一个bias correction 的概念,就是为了能使得 Exponentially weighted average 更加精确.
momentum (or called Gradient descent with momentum)
传统的Gradient descent 算法有如下图所示的问题 - 每次迭代都会来回跳动,不直接指向optimum, 在没有做feature scaling 的时候尤其明显。所以引出一个修正的算法 - Gradient descent with momentum.
RMSprop
目的和上面讲到的Momentum是一样的,就是使得每次迭代都尽量指向optimum而不是来回跳动. 算法实现如下. RMSprop带来的好处是迭代更快,和可以选用更大的learning rate.
Adam optimation algorithm:
结合了Momentum 和 RMSprop 两种算法. Adam stands for Adaptive mement estimation.
Learning rate decay
why? to reduce the oscillation near the central point.
有哪些实现方式呢?
Local optima and saddle point
在大型神经网络里,saddle point 可能比local optima更常见.
Ref:
Coursera, Deep leaning, Andrew Ng
Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week2, Optimization algorithms的更多相关文章
- 《Improving Deep Neural Networks:Hyperparameter tuning, Regularization and Optimization》课堂笔记
Lesson 2 Improving Deep Neural Networks:Hyperparameter tuning, Regularization and Optimization 这篇文章其 ...
- [C4] Andrew Ng - Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
About this Course This course will teach you the "magic" of getting deep learning to work ...
- Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Initialization)
声明:所有内容来自coursera,作为个人学习笔记记录在这里. Initialization Welcome to the first assignment of "Improving D ...
- Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Gradient Checking)
声明:所有内容来自coursera,作为个人学习笔记记录在这里. Gradient Checking Welcome to the final assignment for this week! In ...
- Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Regularization)
声明:所有内容来自coursera,作为个人学习笔记记录在这里. Regularization Welcome to the second assignment of this week. Deep ...
- Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week2, Assignment(Optimization Methods)
声明:所有内容来自coursera,作为个人学习笔记记录在这里. 请不要ctrl+c/ctrl+v作业. Optimization Methods Until now, you've always u ...
- 课程二(Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization),第一周(Practical aspects of Deep Learning) —— 4.Programming assignments:Gradient Checking
Gradient Checking Welcome to this week's third programming assignment! You will be implementing grad ...
- 吴恩达《深度学习》-课后测验-第二门课 (Improving Deep Neural Networks:Hyperparameter tuning, Regularization and Optimization)-Week 1 - Practical aspects of deep learning(第一周测验 - 深度学习的实践)
Week 1 Quiz - Practical aspects of deep learning(第一周测验 - 深度学习的实践) \1. If you have 10,000,000 example ...
- 吴恩达《深度学习》-第二门课 (Improving Deep Neural Networks:Hyperparameter tuning, Regularization and Optimization)-第一周:深度学习的实践层面 (Practical aspects of Deep Learning) -课程笔记
第一周:深度学习的实践层面 (Practical aspects of Deep Learning) 1.1 训练,验证,测试集(Train / Dev / Test sets) 创建新应用的过程中, ...
- 课程二(Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization),第三周(Hyperparameter tuning, Batch Normalization and Programming Frameworks) —— 2.Programming assignments
Tensorflow Welcome to the Tensorflow Tutorial! In this notebook you will learn all the basics of Ten ...
随机推荐
- Spring 整合 Hibernate 时启用二级缓存实例详解
写在前面: 1. 本例使用 Hibernate3 + Spring3: 2. 本例的查询使用了 HibernateTemplate: 1. 导入 ehcache-x.x.x.jar 包: 2. 在 a ...
- BZOJ2288 生日礼物
本题是数据备份的进阶版. 首先去掉所有0,把连续的正数/负数连起来. 计算所有正数段的个数与总和. 然后考虑数据备份,有一点区别: 如果我们在数列中选出一个负数,相当于把它左右连起来. 选出一个正数, ...
- Flask flask_script扩展库
flask_script 1.安装:进入到虚拟环境中,pip install flask_script 2.flask_script 作用:可以通过命令行的形式来操作Flask,例如通过命令跑一个开发 ...
- 【非专业前端】vue+element+webpack
先点这里(- ̄▽ ̄)- 环境搭建 默认你已经安装了node.js 下面安装vue和webpack npm install -g @vue/cli npm install -g @vue/cli-ini ...
- Jquery Mobile事件
Jquery Mobile事件参考手册 on()方法用于添加事件处理程序 1.Touch类事件 在用户触摸屏幕时触发 1.1 tap事件 用户敲击某个元素时发生 $("p").on ...
- c#委托中的匿名方法和lambda表达式
一.一般委托方式 Func<int, int, int> AddMethodHander; public unName() { AddMethodHander += AddMethod; ...
- 高级组件——文件选择器JFileChooser
1.创建文件选择器 JFileChooser 2.设置选择模式 setFileSelectionMode(int mode) JFileChooser.FILES_ONLY 只能选择文件(默认) JF ...
- [POI2005] SKA-Piggy Banks
ps.有关Tarjan算法缩点的简要回顾. 今天上午在Luogu随机到了这道题 Luogu P3420,题目大概是这样: 题目描述 Byteazar the Dragon拥有N个小猪存钱罐.每一个存钱 ...
- C++: 带参数回调函数和不带参数的回调函数;
在C++中,回调函数的应用比较广泛且重要. 通过传递函数指针到其他地方,能够实现远程回调的作用,能够实现远程调用而不需要事件触发信号或者其他机制来实现,方便而快捷: 首先,回调函数有两种形式: 静态 ...
- Win7无法安装程序提示Installer integrity check has failed的解决方法
最近小明在Win7系统中下载了一款软件,但是在下载的时候弹出了NSIS Error:Installer integrity check has failed.Common causer include ...