Gradient Vanishing Problem in Deep Learning
在所有依靠Gradient Descent和Backpropagation算法来学习的Neural Network中,普遍都会存在Gradient Vanishing Problem。Backpropagation的运作过程是,根据Cost Function进行反向传播,利用Chain Rule去计算n层之前某一weight上的梯度,从而更新该weight。而事实上,在网络层次较深的情况下,我们获得的weight梯度,随着反向传播层次的深入,会呈现越来越小的状态。从而,在靠近输出端的Layers中,weight可以被很好的更新,因为可以获得不错的gradient,而在靠近输入端的Layers中,weight则更新缓慢。
举个最简单的例子,来说明该问题。如下的神经网络有四层,每层有一个node:
我们可知w是weight,b是bias,每一层的节点输入是z,输出是a,activation function是a=σ(z),我们可以得出:
当我们已知Cost Function时,我们利用Backpropagation计算weight:
可以看到,第一层的weight梯度,依赖于之后各层activation function的一阶导数之积。而对于Machine Learning中常用的Sigmoid及tanh激励函数,其derivative图像如下:
Sigmoid的derivative是[0,0.25]的,而tanh的derivative是[0,1]的。通过上式,我们看出,通过Backpropagation求梯度时,每往回传播一层,就要多乘以一项δ‘(z),也就是说,随着向回传递的深入,梯度会呈指数级的衰减,直至缩减到0,导致前层的权重无法更新。tanh要略好于sigmoid,但依然难以解决Gradient Vanishing的问题。所以Relu Function应运而生,并且在Deep Learning方面取得了巨大成功。Relu的表达式及图形如下:
其当x>0时,derivative是1,小于0时,derivative为0。该函数很好的解决了Gradient Vanishing Problem,在大多数情况下,我们构建Deep Learning时可以使用Relu作为默认的Activation Function。
Gradient Vanishing Problem in Deep Learning的更多相关文章
- (转)WHY DEEP LEARNING IS SUDDENLY CHANGING YOUR LIFE
Main Menu Fortune.com E-mail Tweet Facebook Linkedin Share icons By Roger Parloff Illustration ...
- Growing Pains for Deep Learning
Growing Pains for Deep Learning Advances in theory and computer hardware have allowed neural network ...
- Deep Learning Libraries by Language
Deep Learning Libraries by Language Tweet Python Theano is a python library for defining and ...
- Deep learning with Python
一.导论 1.1 人工智能.机器学习.深度学习 人工智能.机器学习 人工智能:1980年代达到高峰的是专家系统,符号AI是之前的,但不能解决模糊.复杂的问题. 机器学习是把数据.答案做输入,规则作输出 ...
- This instability is a fundamental problem for gradient-based learning in deep neural networks. vanishing exploding gradient problem
The unstable gradient problem: The fundamental problem here isn't so much the vanishing gradient pro ...
- Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Gradient Checking)
声明:所有内容来自coursera,作为个人学习笔记记录在这里. Gradient Checking Welcome to the final assignment for this week! In ...
- 课程二(Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization),第一周(Practical aspects of Deep Learning) —— 4.Programming assignments:Gradient Checking
Gradient Checking Welcome to this week's third programming assignment! You will be implementing grad ...
- Deep Learning专栏--强化学习之从 Policy Gradient 到 A3C(3)
在之前的强化学习文章里,我们讲到了经典的MDP模型来描述强化学习,其解法包括value iteration和policy iteration,这类经典解法基于已知的转移概率矩阵P,而在实际应用中,我们 ...
- Deep Learning in a Nutshell: History and Training
Deep Learning in a Nutshell: History and Training This series of blog posts aims to provide an intui ...
随机推荐
- stl(set或map)
https://nanti.jisuanke.com/t/41384 There are nnn points in an array with index from 111 to nnn, and ...
- 错误提示控件errorProvider
http://www.cnblogs.com/suguoqiang/archive/2012/07/17/2596564.html 错误提示控件errorProvider VS显示: 核心代码: th ...
- k3 cloud支付申请单下推付款单时候提示未将对象引用设置到对象的实例
项目支付申请部门没有币别没有填写,没有把币别带过来才
- CentOS7搭建Flume与Kafka整合及基础操作与测试
前提 已完成Kafka的搭建,具体步骤参照CentOS7搭建Kafka单机环境及基础操作 Flume安装 下载 wget http://mirrors.tuna.tsinghua.edu.cn/apa ...
- spl_autoload_register() 函数实现的自动加载
和Python用module来区分代码块不同,PHP按照命名空间来区分,开始学PHP的时候一心认定了如果想用 use 关键字来导入(Python的习惯说法)一个类或者函数或者其他对象的话,必须先inc ...
- AOP拦截日志类,抛异常:java.lang.IllegalStateException: It is illegal to call this method if the current request is not in asynchronous mode
AOP的日志拦截类中,抛出异常: java.lang.IllegalStateException: It is illegal to call this method if the current r ...
- Codeforces 图论题板刷(2000~2400)
前言 首先先刷完这些在说 题单 25C Roads in Berland 25D Roads not only in Berland 9E Interestring graph and Apples ...
- thinkphp数据库连接
https://www.kancloud.cn/manual/thinkphp5/118059 一.配置文件定义 常用的配置方式是在应用目录或者模块目录下面的database.php中添加下面的配置参 ...
- javascript实现表单提交加密
javascript实现表单提交加密 通常表单的提交有两种方式,一是直接通过html的form提交,代码如下: <form action="" method="&q ...
- 关于Reporting Services网站
1.http://www.c-sharpcorner.com/search/sql%20server%20reporting%20services 2.https://msdn.microsoft.c ...