在所有依靠Gradient Descent和Backpropagation算法来学习的Neural Network中,普遍都会存在Gradient Vanishing Problem。Backpropagation的运作过程是,根据Cost Function进行反向传播,利用Chain Rule去计算n层之前某一weight上的梯度,从而更新该weight。而事实上,在网络层次较深的情况下,我们获得的weight梯度,随着反向传播层次的深入,会呈现越来越小的状态。从而,在靠近输出端的Layers中,weight可以被很好的更新,因为可以获得不错的gradient,而在靠近输入端的Layers中,weight则更新缓慢。

举个最简单的例子,来说明该问题。如下的神经网络有四层,每层有一个node:

我们可知w是weight,b是bias,每一层的节点输入是z,输出是a,activation function是a=σ(z),我们可以得出:

当我们已知Cost Function时,我们利用Backpropagation计算weight:

可以看到,第一层的weight梯度,依赖于之后各层activation function的一阶导数之积。而对于Machine Learning中常用的Sigmoid及tanh激励函数,其derivative图像如下:

Sigmoid的derivative是[0,0.25]的,而tanh的derivative是[0,1]的。通过上式,我们看出,通过Backpropagation求梯度时,每往回传播一层,就要多乘以一项δ‘(z),也就是说,随着向回传递的深入,梯度会呈指数级的衰减,直至缩减到0,导致前层的权重无法更新。tanh要略好于sigmoid,但依然难以解决Gradient Vanishing的问题。所以Relu Function应运而生,并且在Deep Learning方面取得了巨大成功。Relu的表达式及图形如下:

其当x>0时,derivative是1,小于0时,derivative为0。该函数很好的解决了Gradient Vanishing Problem,在大多数情况下,我们构建Deep Learning时可以使用Relu作为默认的Activation Function。

Gradient Vanishing Problem in Deep Learning的更多相关文章

  1. (转)WHY DEEP LEARNING IS SUDDENLY CHANGING YOUR LIFE

    Main Menu Fortune.com       E-mail Tweet Facebook Linkedin Share icons By Roger Parloff Illustration ...

  2. Growing Pains for Deep Learning

    Growing Pains for Deep Learning Advances in theory and computer hardware have allowed neural network ...

  3. Deep Learning Libraries by Language

    Deep Learning Libraries by Language Tweet         Python Theano is a python library for defining and ...

  4. Deep learning with Python

    一.导论 1.1 人工智能.机器学习.深度学习 人工智能.机器学习 人工智能:1980年代达到高峰的是专家系统,符号AI是之前的,但不能解决模糊.复杂的问题. 机器学习是把数据.答案做输入,规则作输出 ...

  5. This instability is a fundamental problem for gradient-based learning in deep neural networks. vanishing exploding gradient problem

    The unstable gradient problem: The fundamental problem here isn't so much the vanishing gradient pro ...

  6. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Gradient Checking)

    声明:所有内容来自coursera,作为个人学习笔记记录在这里. Gradient Checking Welcome to the final assignment for this week! In ...

  7. 课程二(Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization),第一周(Practical aspects of Deep Learning) —— 4.Programming assignments:Gradient Checking

    Gradient Checking Welcome to this week's third programming assignment! You will be implementing grad ...

  8. Deep Learning专栏--强化学习之从 Policy Gradient 到 A3C(3)

    在之前的强化学习文章里,我们讲到了经典的MDP模型来描述强化学习,其解法包括value iteration和policy iteration,这类经典解法基于已知的转移概率矩阵P,而在实际应用中,我们 ...

  9. Deep Learning in a Nutshell: History and Training

    Deep Learning in a Nutshell: History and Training This series of blog posts aims to provide an intui ...

随机推荐

  1. stl(set或map)

    https://nanti.jisuanke.com/t/41384 There are nnn points in an array with index from 111 to nnn, and ...

  2. 错误提示控件errorProvider

    http://www.cnblogs.com/suguoqiang/archive/2012/07/17/2596564.html 错误提示控件errorProvider VS显示: 核心代码: th ...

  3. k3 cloud支付申请单下推付款单时候提示未将对象引用设置到对象的实例

    项目支付申请部门没有币别没有填写,没有把币别带过来才

  4. CentOS7搭建Flume与Kafka整合及基础操作与测试

    前提 已完成Kafka的搭建,具体步骤参照CentOS7搭建Kafka单机环境及基础操作 Flume安装 下载 wget http://mirrors.tuna.tsinghua.edu.cn/apa ...

  5. spl_autoload_register() 函数实现的自动加载

    和Python用module来区分代码块不同,PHP按照命名空间来区分,开始学PHP的时候一心认定了如果想用 use 关键字来导入(Python的习惯说法)一个类或者函数或者其他对象的话,必须先inc ...

  6. AOP拦截日志类,抛异常:java.lang.IllegalStateException: It is illegal to call this method if the current request is not in asynchronous mode

    AOP的日志拦截类中,抛出异常: java.lang.IllegalStateException: It is illegal to call this method if the current r ...

  7. Codeforces 图论题板刷(2000~2400)

    前言 首先先刷完这些在说 题单 25C Roads in Berland 25D Roads not only in Berland 9E Interestring graph and Apples ...

  8. thinkphp数据库连接

    https://www.kancloud.cn/manual/thinkphp5/118059 一.配置文件定义 常用的配置方式是在应用目录或者模块目录下面的database.php中添加下面的配置参 ...

  9. javascript实现表单提交加密

    javascript实现表单提交加密 通常表单的提交有两种方式,一是直接通过html的form提交,代码如下: <form action="" method="&q ...

  10. 关于Reporting Services网站

    1.http://www.c-sharpcorner.com/search/sql%20server%20reporting%20services 2.https://msdn.microsoft.c ...