Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week3, Hyperparameter tuning, Batch Normalization and Programming Frameworks
Tuning process
下图中的需要tune的parameter的先后顺序, 红色>黄色>紫色,其他基本不会tune.
先讲到怎么选hyperparameter, 需要随机选取(sampling at random)
随机选取的过程中,可以采用从粗到细的方法逐步确定参数
有些参数可以按照线性随机选取, 比如 n[l]
但是有些参数就不适合线性的sampling at radom, 比如 learning rate α,这时可以用 log
Andrew 很幽默的讲到了两种选参数的实际场景 pandas vs caviar. pandas approach 一般用在你的算力不够时候,要持续几天的training.
Batch norm
我们知道对input layer 做 normalizing, 其实对每一层的输入都可以做normalizing, 这就是 batch norm. 做batch norm 时,有对 activation后的结果做norm 的,也有对activation 前的结果 z 做batch norm 的,这里讲的是后一种,对z 做norm.
为什么Batch Norm 起作用呢?
先看下下面图讲到的convariate shift,如果traing set 的distribution 变了,就应该重新train model. 同样,对NN的每一层也有类似的问题.
Andrew讲到batch norm 是为了尽量使得不同layer decouple,这样相互影响就要小一点,整个NN比较稳定.
Batch norm 还有regularization 的作用,但是这个算法主要不是做这个的. 不建议专门用它来做regularization.
对 test set 求 μ, σ2, 采用了不一样的方法,就是基于签名mini-batch set 求出来的μ, σ2 应用exponetially weighted average 求平均值. 它和logistic regression 一样,decision boudary 是线性的.
Softmax Regression
Softmax regression 就是 logistic regression 的generaliazation 版本, 它可以用在multi-class clarification 问题上。和logistic regression 一样,decision boudary 都是线性的. 如果要使得decison boudary 是非线性的就需要deep network.
Programing framework
TensorFlow by google, an example
Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week3, Hyperparameter tuning, Batch Normalization and Programming Frameworks的更多相关文章
- [C2W3] Improving Deep Neural Networks : Hyperparameter tuning, Batch Normalization and Programming Frameworks
第三周:Hyperparameter tuning, Batch Normalization and Programming Frameworks 调试处理(Tuning process) 目前为止, ...
- Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Initialization)
声明:所有内容来自coursera,作为个人学习笔记记录在这里. Initialization Welcome to the first assignment of "Improving D ...
- Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Gradient Checking)
声明:所有内容来自coursera,作为个人学习笔记记录在这里. Gradient Checking Welcome to the final assignment for this week! In ...
- Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Regularization)
声明:所有内容来自coursera,作为个人学习笔记记录在这里. Regularization Welcome to the second assignment of this week. Deep ...
- Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week2, Assignment(Optimization Methods)
声明:所有内容来自coursera,作为个人学习笔记记录在这里. 请不要ctrl+c/ctrl+v作业. Optimization Methods Until now, you've always u ...
- 课程二(Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization),第三周(Hyperparameter tuning, Batch Normalization and Programming Frameworks) —— 2.Programming assignments
Tensorflow Welcome to the Tensorflow Tutorial! In this notebook you will learn all the basics of Ten ...
- 吴恩达《深度学习》-课后测验-第一门课 (Neural Networks and Deep Learning)-Week 3 - Shallow Neural Networks(第三周测验 - 浅层神 经网络)
Week 3 Quiz - Shallow Neural Networks(第三周测验 - 浅层神经网络) \1. Which of the following are true? (Check al ...
- [CS231n-CNN] Training Neural Networks Part 1 : activation functions, weight initialization, gradient flow, batch normalization | babysitting the learning process, hyperparameter optimization
课程主页:http://cs231n.stanford.edu/ Introduction to neural networks -Training Neural Network ________ ...
- Coursera, Deep Learning 2, Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Course
Train/Dev/Test set Bias/Variance Regularization 有下面一些regularization的方法. L2 regularation drop out da ...
随机推荐
- Electron一学习资源收集和练习demo
1.近日为了做项目查资料学习electron,简直头都要炸了,就官方的electron-quick-start的例子进行了基本的练习之后,不断的查资料终于发现一些有用的demo来看源代码学习,一遍看代 ...
- P3486 [POI2009]KON-Ticket Inspector
啊!这题做的真是爽!除了DP这个方法是有提示的之外,这题居然没有题解,哈哈哈嘿嘿嘿.很自豪的说:全是我自己独立解出来的一道题,包括设计状态,推倒(☺)转移方程,最后记录路径. 好了,首先,我们发现这题 ...
- JavaScript 正整数正则表达式
function testNumber(){ var yourinputValue=$("#yourinputId").val(); var reg = /^[1-9]\d*$/ ...
- TestNg 11. 超时测试
前沿:多久时间没有响应,就是超时. 代码:用timeOut这个属性,超过规定的时间就是fail,不超过就是success package com.course.testng; import org.t ...
- 3D游戏的角色移动和旋转
* -----英雄的移动控制 * * * * */ using System.Collections; using System.Collections.Generic; using UnityEng ...
- 委托delegate
委托delegate没有函数体.委托可以指向函数(要与指向的函数格式.类型相一致) namespace demo { public delegate double MyDelegate(double ...
- JDBC动态查询MySQL中的表(按条件筛选)
动态查询实现按条件筛选.PreparedStatement 准备语句指定要查询的表头列,.setString()通过赋值指定行,.executeQuery()执行语句 在数据库test里先创建表sch ...
- 新建体(2):create or replace object创建存储包、存储过程、函数
http://heisetoufa.iteye.com/blog/366957/ 创建一个package(包) 声明: create or replace package mpay_route is ...
- 获取日k数据
http://web.ifzq.gtimg.cn/appstock/app/fqkline/get?_var=kline_dayqfq¶m=sz002921,day,,,320,qfq ...
- 关于vue的小实例
学习网址:http://www.runoob.com/vue2/vue-tutorial.html 下面是我在上面学着写的两个小例子, 1. 实现点击全选,下面的均被选中,再点击一下,下面的均取消选择 ...