Principles of training multi-layer neural network using backpropagation


http://galaxy.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html

The project describes teaching process of multi-layer neural network employing backpropagation algorithm. To illustrate this process the three layer neural network with two inputs and one output,which is shown in the picture below, is used:

Each neuron is composed of two units. First unit adds products of
weights coefficients and input signals. The second unit realise
nonlinear
function, called neuron activation function. Signal e is adder output signal, and y = f(e) is output signal of nonlinear
element. Signal y is also output signal of neuron.

To teach the neural network we need training data set. The training data set consists of input signals (x1 and
x2 ) assigned with corresponding target (desired output) z.
The network training is an iterative process. In each
iteration weights coefficients of nodes are modified using new data from
training data set. Modification is calculated using algorithm
described below:
Each teaching step starts with forcing both input signals from training
set. After this stage we can determine output signals values for
each neuron in each network layer. Pictures below illustrate how signal
is propagating through the network, Symbols w(xm)n
represent weights of connections between network input xm and neuron n in input layer. Symbols yn
represents output signal of neuron n.

Propagation of signals through the hidden layer. Symbols wmn represent weights of connections between output of neuron
m and input of neuron n in the next layer.

Propagation of signals through the output layer.

In the next algorithm step the output signal of the network y is compared with the desired output value (the target), which is found
in training data set. The difference is called error signal d of
output layer neuron.

It is impossible to compute error signal for internal neurons directly,
because output values of these neurons are unknown. For many years
the effective method for training multiplayer networks has been
unknown. Only in the middle eighties the backpropagation algorithm has
been
worked out. The idea is to propagate error signal d (computed in
single teaching step) back to all neurons, which output signals were input for discussed neuron.

The weights' coefficients wmn used to propagate errors
back are equal to this used during computing output value. Only the
direction of data flow is changed (signals are propagated from output to
inputs one after the other). This technique is used for all network
layers. If propagated errors came from few neurons they are added. The
illustration is below:

When the error signal for each neuron is computed, the weights
coefficients of each neuron input node may be modified. In formulas
below
df(e)/de represents derivative of neuron activation function (which weights are modified).

Coefficient h
affects network teaching speed. There are a few
techniques to select this parameter. The first method is to start
teaching process with large value of the parameter. While weights
coefficients are being established the parameter is being decreased
gradually. The second, more complicated, method starts teaching with
small parameter value. During the teaching process the parameter is
being increased when the teaching is advanced and then decreased again
in
the final stage. Starting teaching process with low parameter value
enables to determine weights coefficients signs.

References
Ryszard Tadeusiewcz "Sieci neuronowe", Kraków 1992

【转】Principles of training multi-layer neural network using backpropagation的更多相关文章

  1. 【论文考古】Training a 3-Node Neural Network is NP-Complete

    今天看到一篇1988年的老文章谈到了训练一个简单网络是NPC问题[1].也就是下面的网络结构,在线性激活函数下,如果要找到参数使得输入数据的标签估计准确,这个问题是一个NPC问题.这个文章的意义在于宣 ...

  2. 用matlab训练数字分类的深度神经网络Training a Deep Neural Network for Digit Classification

    This example shows how to use Neural Network Toolbox™ to train a deep neural network to classify ima ...

  3. CheeseZH: Stanford University: Machine Learning Ex4:Training Neural Network(Backpropagation Algorithm)

    1. Feedforward and cost function; 2.Regularized cost function: 3.Sigmoid gradient The gradient for t ...

  4. A Neural Network in 11 lines of Python

    A Neural Network in 11 lines of Python A bare bones neural network implementation to describe the in ...

  5. [Tensorflow] Cookbook - Neural Network

    In this chapter, we'll cover the following recipes: Implementing Operational Gates Working with Gate ...

  6. 课程一(Neural Networks and Deep Learning),第四周(Deep Neural Networks)——2.Programming Assignments: Building your Deep Neural Network: Step by Step

    Building your Deep Neural Network: Step by Step Welcome to your third programming exercise of the de ...

  7. Deep Learning 28:读论文“Multi Column Deep Neural Network for Traffic Sign Classification”-------MCDNN 简单理解

    读这篇论文“ Multi Column Deep Neural Network for Traffic Sign Classification”是为了更加理解,论文“Multi-column Deep ...

  8. Online handwriting recognition using multi convolution neural networks

    w可以考虑从计算机的“机械性.重复性”特征去设计“低效的”算法. https://www.codeproject.com/articles/523074/webcontrols/ Online han ...

  9. (转)The Neural Network Zoo

    转自:http://www.asimovinstitute.org/neural-network-zoo/ THE NEURAL NETWORK ZOO POSTED ON SEPTEMBER 14, ...

随机推荐

  1. php获取当月天数及当月第一天及最后一天

    1.获取上个月第一天及最后一天.   echo date('Y-m-01', strtotime('-1 month'));   echo "<br/>";   ech ...

  2. maven运行时的配置及命令详解

      上面是指定端口运行程序的,也可以先指定好,直接在上面的地方写jettty:run           当然,如果你是在控制台运行且安装了maven,直接可以进入项目的文件中:mvn jetty:r ...

  3. elasticsearch设置外部可访问

    修改/config/elasticsearch.yml文件,增加如下配置: network.host: 0.0.0.0 浏览器访问http://192.168.17.134:9200/效果: 实际操作 ...

  4. 学习笔记-AngularJs(三)

    学习笔记-AngularJs(二)写了个所有程序语言入门时都必须要写的Hello World,那么从现在开始做那个之前说过的互联网大佬介绍的学习例子,当然这里开始会慢慢按照之前说过的目录来搭建这个学习 ...

  5. PAT-GPLT训练集 L2-002 链表去重

    PAT-GPLT训练集 L2-002 链表去重 题目大意为给出一个单链表,去除重复的结点,输出删除后的链表,并且把被删除的结点也以链表形式输出 思路:把这个链表直接分成两个链表,再直接输出就可以 代码 ...

  6. daay04流程控制之for循环

    for循环主要用于循环取值 student=['egon','虎老师','lxxdsb','alexdsb','wupeiqisb'] # i=0 # while i < len(student ...

  7. flask下载zip文件报错TypeError

    报错内容:TypeError: make_conditional() got an unexpected keyword argument 'accept_ranges' 报错行自己代码如下: dir ...

  8. flask 开启多线程

    app.run(debug=True, threaded=True)

  9. wordpress有用的插件

    代码高亮 https://wordpress.org/plugins/crayon-syntax-highlighter/ 这个貌似比较好用 百度UEditor(富文本编辑器) https://www ...

  10. 卷积与反卷积以及步长stride

    1. 卷积与反卷积 如上图演示了卷积核反卷积的过程,定义输入矩阵为 I(4×4),卷积核为 K(3×3),输出矩阵为 O(2×2): 卷积的过程为:Conv(I,W)=O 反卷积的过称为:Deconv ...