The unstable gradient problem: The fundamental problem here isn't so much the vanishing gradient problem or the exploding gradient problem. It's that the gradient in early layers is the product of terms from all the later layers. When there are many layers, that's an intrinsically unstable situation. The only way all layers can learn at close to the same speed is if all those products of terms come close to balancing out. Without some mechanism or underlying reason for that balancing to occur, it's highly unlikely to happen simply by chance. In short, the real problem here is that neural networks suffer from an unstable gradient problem. As a result, if we use standard gradient-based learning techniques, different layers in the network will tend to learn at wildly different speeds.

Again, early hidden layers learn much more slowly than later hidden layers. In this case, the first hidden layer is learning roughly 100 times slower than the final hidden layer. No wonder we were having trouble training these networks earlier!

We have here an important observation: in at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers. And while we've seen this in just a single network, there are fundamental reasons why this happens in many neural networks. The phenomenon is known as the vanishing gradient problem**See Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, by Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber (2001). This paper studied recurrent neural nets, but the essential phenomenon is the same as in the feedforward networks we are studying. See also Sepp Hochreiter's earlier Diploma Thesis,Untersuchungen zu dynamischen neuronalen Netzen (1991, in German)..

Why does the vanishing gradient problem occur? Are there ways we can avoid it? And how should we deal with it in training deep neural networks? In fact, we'll learn shortly that it's not inevitable, although the alternative is not very attractive, either: sometimes the gradient gets much larger in earlier layers! This is the exploding gradient problem, and it's not much better news than the vanishing gradient problem. More generally, it turns out that the gradient in deep neural networks is unstable, tending to either explode or vanish in earlier layers. This instability is a fundamental problem for gradient-based learning in deep neural networks. It's something we need to understand, and, if possible, take steps to address.

One response to vanishing (or unstable) gradients is to wonder if they're really such a problem. Momentarily stepping away from neural nets, imagine we were trying to numerically minimize a function f(x)f(x) of a single variable. Wouldn't it be good news if the derivative f′(x)f′(x) was small? Wouldn't that mean we were already near an extremum? In a similar way, might the small gradient in early layers of a deep network mean that we don't need to do much adjustment of the weights and biases?

http://neuralnetworksanddeeplearning.com/chap5.html

Design a Gradient Logo Illustrator Tutorial - YouTube

This instability is a fundamental problem for gradient-based learning in deep neural networks. vanishing exploding gradient problem的更多相关文章

  1. Exploring Adversarial Attack in Spiking Neural Networks with Spike-Compatible Gradient

    郑重声明:原文参见标题,如有侵权,请联系作者,将会撤销发布! arXiv:2001.01587v1 [cs.NE] 1 Jan 2020 Abstract 脉冲神经网络(SNN)被广泛应用于神经形态设 ...

  2. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Gradient Checking)

    声明:所有内容来自coursera,作为个人学习笔记记录在这里. Gradient Checking Welcome to the final assignment for this week! In ...

  3. 课程二(Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization),第一周(Practical aspects of Deep Learning) —— 4.Programming assignments:Gradient Checking

    Gradient Checking Welcome to this week's third programming assignment! You will be implementing grad ...

  4. [CS231n-CNN] Training Neural Networks Part 1 : activation functions, weight initialization, gradient flow, batch normalization | babysitting the learning process, hyperparameter optimization

    课程主页:http://cs231n.stanford.edu/   Introduction to neural networks -Training Neural Network ________ ...

  5. 梯度消失(vanishing gradient)与梯度爆炸(exploding gradient)问题

    (1)梯度不稳定问题: 什么是梯度不稳定问题:深度神经网络中的梯度不稳定性,前面层中的梯度或会消失,或会爆炸. 原因:前面层上的梯度是来自于后面层上梯度的乘乘积.当存在过多的层次时,就出现了内在本质上 ...

  6. 覆盖问题:最大覆盖问题(Maximum Covering Location Problem,MCLP)和集覆盖问题(Location Set Covering Problem,LSCP)

    集覆盖问题研究满足覆盖所有需求点顾客的前提下,服务站总的建站个数或建 设费用最小的问题.集覆盖问题最早是由 Roth和 Toregas等提出的,用于解决消防中心和救护车等的应急服务设施的选址问题,他们 ...

  7. Codeforces Round #602 (Div. 2, based on Technocup 2020 Elimination Round 3) A. Math Problem 水题

    A. Math Problem Your math teacher gave you the following problem: There are n segments on the x-axis ...

  8. Deep Learning专栏--强化学习之从 Policy Gradient 到 A3C(3)

    在之前的强化学习文章里,我们讲到了经典的MDP模型来描述强化学习,其解法包括value iteration和policy iteration,这类经典解法基于已知的转移概率矩阵P,而在实际应用中,我们 ...

  9. 随机梯度下降(Stochastic gradient descent)和 批量梯度下降(Batch gradient descent )的公式对比、实现对比[转]

    梯度下降(GD)是最小化风险函数.损失函数的一种常用方法,随机梯度下降和批量梯度下降是两种迭代求解思路,下面从公式和实现的角度对两者进行分析,如有哪个方面写的不对,希望网友纠正. 下面的h(x)是要拟 ...

随机推荐

  1. Angular 学习笔记——factory

    <!DOCTYPE HTML> <html ng-app="myApp"> <head> <meta http-equiv="C ...

  2. 利用 getsockname 和 getpeername 来获取某一个链接的本地地址和远端地址

    在两台计算机上建立一个网络连接,需要五个要素:本机地址 本机端口 协议类型 远端端口 远端地址.那么如何从一个建立好的连接上获取这些信息呢.就需要用到 getsockname  和 getpeerna ...

  3. Fragment生命周期和Activity生命周期的关系

    Fragment生命周期: 重点回调函数: 1. onAttach()当碎片和活动建立关联的时候调用.2. onCreateView()为碎片创建视图(加载布局)时调用.3. onActivityCr ...

  4. javascript判断是否为数组 面试题

    1.方法 instanceof:多框架下有问题(看示例代码). Object.prototype.toString.call():兼容性很好 Array.isArray: IE9+以上 constru ...

  5. Odoo 11 Backend

    Table of Contents 命令入口 服务器 启动server thread 模式 prefork 模式 gevent模式 wsgi 应用 响应 客户端请求 xmlrpc web http路由 ...

  6. Oracle case when then else end的两种用法

    查询表结构 SELECT T.COLUMN_ID, T.COLUMN_NAME, (CASE WHEN (T.DATA_TYPE = 'VARCHAR2' OR T.DATA_TYPE = 'RAW' ...

  7. Oracle 时间 MM-dd形式转换

    SELECT TO_CHAR( SYSDATE,'MM-dd') AS beginTime,TO_CHAR( TO_DATE(MAX(C.SUBSCRIBE_DATE),'YYYY-MM-dd'),' ...

  8. React 组件条件渲染的几种方式

    一.条件表达式渲染 (适用于两个组件二选一的渲染) render() { const isLoggedIn = this.state.isLoggedIn; return ( <div> ...

  9. Linux下安装jdk8的方法

    一.yum安装 只需要一条命令就可以安装jdk: yum install java--openjdk* -y 执行过这条命令无需配置,直接可以使用. 二.下载tar包安装 下载jdk8 登录网址:ht ...

  10. (二)spark算子 分为3大类

     transgormation的算子对key-value类型的数据有三种: (1)输入 与 输出为一对一关系 mapValue();针对key-value类型的数据并只对其中的value进行操作,不对 ...