UFLDL教程之（一）sparseae_exercise

下面，将UFLDL教程中的sparseae_exercise练习中的各函数及注释列举如下

首先，给出各函数的调用关系

主函数：train.m

（1）调用sampleIMAGES函数从已知图像中扣取多个图像块儿

（2）调用display_network函数，以网格的形式，随机显示多个扣取的图像块儿

（3）梯度校验，该部分的目的是测试函数是否正确，可以由单独的函数checkSparseAutoencoderCost实现

①利用sparseAutoencoderCost函数计算网路的代价函数和梯度值

②利用computeNumericalGradient函数计算梯度值（这里，要利用checkNumericalGradient函数验证该梯度计算函数是否正确）

③比较①和②的梯度计算结果，判断编写的sparseAutoencoderCost函数是否正确

如果sparseAutoencoderCost函数是正确的，那么，在实际训练中，不需要运行checkSparseAutoencoderCost

（4）利用L-BFGS方法对网络进行训练，从而得到最优化的网络的权值和偏执项

（5）对训练结果进行可视化

然后，对个函数给出注释

train.m

%% CS294A/CS294W Programming Assignment Starter Code

addpath ..common\

%%======================================================================
%% STEP : Here we provide the relevant parameters values that will
%  allow your sparse autoencoder to get good filters; you do not need to change the parameters below.
visibleSize = *;   % number of input units
hiddenSize = ;     % number of hidden units
sparsityParam = 0.01;   % desired average activation of the hidden units.
% (This was denoted by the Greek alphabet rho, which looks like a lower-case "p", in the lecture notes).
lambda = 0.0001;     % weight decay parameter
beta = ;            % weight of sparsity penalty term

%%======================================================================
%% STEP : Implement sampleIMAGES
%  After implementing sampleIMAGES, the display_network command should display a random sample of  patches from the dataset
%从图像中提取图像块儿，每一个提取到的图像块儿存放在patches的每一列中
patches = sampleIMAGES;
%随机提取patches中的200列，然后显示这200列所对应的图像
IMG=patches(:,randi(size(patches,),,));
display_network(IMG,);

%%======================================================================
%% STEP  and STEP ：Implement sparseAutoencoderCost and Gradient Checking
checkSparseAutoencoderCost()

%%======================================================================
%% STEP : After verifying that your implementation of

%  Randomly initialize the parameters
theta = initializeParameters(hiddenSize, visibleSize);

%  Use minFunc to minimize the function
addpath minFunc/
options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost function
% Generally, for minFunc to work, you need a function pointer with two outputs: the function value and the gradient.
% In our problem, sparseAutoencoderCost.m satisfies this.
options.maxIter = ;      % Maximum number of iterations of L-BFGS to run
options.display = 'on';

% opttheta是整个神经网络的权值和偏执项构成的向量
[opttheta, cost] = minFunc( @(p) sparseAutoencoderCost(p, ...
    visibleSize, hiddenSize, ...
    lambda, sparsityParam, ...
    beta, patches), ...
    theta, options);

%%======================================================================
%% STEP : Visualization
W1 = reshape(opttheta(:hiddenSize*visibleSize), hiddenSize, visibleSize);%第一层的权值矩阵
display_network(W1', 12);

print -djpeg weights.jpg   % save the visualization to a file

checkSparseAutoencoderCost.m

%% 该函数主要目的是检验SparseAutoencoderCost函数是否正确
function checkSparseAutoencoderCost()

%% 产生一个稀疏自编码网络（可以与主程序相同，也可以重新产生）
visibleSize = *;   % number of input units
hiddenSize = ;     % number of hidden units
sparsityParam = 0.01;   % desired average activation of the hidden units.
% (This was denoted by the Greek alphabet rho, which looks like a lower-case "p", in the lecture notes).
lambda = 0.0001;     % weight decay parameter
beta = ;            % weight of sparsity penalty term

patches = sampleIMAGES; 

%  Obtain random parameters theta
theta = initializeParameters(hiddenSize, visibleSize);

%% 计算代价函数和梯度
[cost, grad] = sparseAutoencoderCost(theta, visibleSize, hiddenSize, lambda, ...
        sparsityParam, beta, patches(:,:));

%% 利用近似方法计算梯度（要调用自编码器的代价函数计算程序）
    numgrad = computeNumericalGradient( @(x) sparseAutoencoderCost(x, visibleSize, ...
        hiddenSize, lambda, ...
        sparsityParam, beta, ...
        patches(:,:)), theta);

%% 比较cost函数计算得到的梯度和由近似得到的梯度之
% Use this to visually compare the gradients side by side
disp([numgrad grad]);

% Compare numerically computed gradients with the ones obtained from backpropagation
diff = norm(numgrad-grad)/norm(numgrad+grad);
disp(diff); % Should be small. In our implementation, these values are usually less than 1e-.

end

sparseAutoencoderCost.m

%% 计算网络的代价函数和梯度
function [cost,grad] = sparseAutoencoderCost(theta, visibleSize, hiddenSize, ...
                                                                       lambda, sparsityParam, beta, data)

% visibleSize: the number of input units (probably )
% hiddenSize: the number of hidden units (probably )
% lambda: weight decay parameter
% sparsityParam: The desired average activation for the hidden units (denoted in the lecture
%                           notes by the greek alphabet rho, which looks like a lower-case "p").
% beta: weight of sparsity penalty term
% data: Our 64x10000 matrix containing the training data.  So, data(:,i) is the i-th training example.

% The input theta is a vector (because minFunc expects the parameters to be a vector).
% We first convert theta to the (W1, W2, b1, b2) matrix/vector format, so that this
% follows the notation convention of the lecture notes.

W1 = reshape(theta(:hiddenSize*visibleSize), hiddenSize, visibleSize);
W2 = reshape(theta(hiddenSize*visibleSize+:*hiddenSize*visibleSize), visibleSize, hiddenSize);
b1 = theta(*hiddenSize*visibleSize+:*hiddenSize*visibleSize+hiddenSize);
b2 = theta(*hiddenSize*visibleSize+hiddenSize+:end);

% Cost and gradient variables (your code needs to compute these values).
% Here, we initialize them to zeros.
cost = ;

m=size(data,);

%% ---------- YOUR CODE HERE --------------------------------------
%  Instructions: Compute the cost/optimization objective J_sparse(W,b) for the Sparse Autoencoder,
%                and the corresponding gradients W1grad, W2grad, b1grad, b2grad.
%
% W1grad, W2grad, b1grad and b2grad should be computed using backpropagation.
% Note that W1grad has the same dimensions as W1, b1grad has the same dimensions
% as b1, etc.  Your code should set W1grad to be the partial derivative of J_sparse(W,b) with
% respect to W1.  I.e., W1grad(i,j) should be the partial derivative of J_sparse(W,b)
% with respect to the input parameter W1(i,j).  Thus, W1grad should be equal to the term
% [(/m) \Delta W^{()} + \lambda W^{()}] in the last block of pseudo-code in Section 2.2
% of the lecture notes (and similarly for W2grad, b1grad, b2grad).
%
% Stated differently, if we were using batch gradient descent to optimize the parameters,
% the gradient descent update to W1 would be W1 := W1 - alpha * W1grad, and similarly for W2, b1, b2.
%

%% 前向传播算法
a1=data;
z2=bsxfun(@plus,W1*a1,b1);
a2=sigmoid(z2);
z3=bsxfun(@plus,W2*a2,b2);
a3=sigmoid(z3);

%% 计算网络误差
% 误差项J1=所有样本代价函数均值
y=data; % 网络的理想输出值
Ei=sum((a3-y).^)/; %每一个样本的代价函数
J1=sum(Ei)/m;
% 正则化项J2=所有权值项平方和
J2=sum(W1(:).^)+sum(W2(:).^);
% 稀疏项J3=所有隐藏层的神经元相对熵之和
rho_hat=sum(a2,)/m;
KL=sum(sparsityParam*log(sparsityParam./rho_hat)+...
      (-sparsityParam)*log((-sparsityParam)./(-rho_hat)));
J3=KL;
% 网络的代价函数
cost=J1+lambda*J2/+beta*J3;

%% 反向传播算法计算各层敏感度delta
delta3=-(data-a3).*dsigmoid(z3);
spare_delta=beta*(-sparsityParam./rho_hat+(-sparsityParam)./(-rho_hat));
delta2=bsxfun(@plus,W2'*delta3,spare_delta).*dsigmoid(z2); % 这里加入了稀疏项 

%% 计算代价函数对各层权值和偏执项的梯度
W1grad=delta2*a1'/m+lambda*W1;
W2grad=delta3*a2'/m+lambda*W2;
b1grad=sum(delta2,)/m;
b2grad=sum(delta3,)/m;

%-------------------------------------------------------------------
% After computing the cost and gradient, we will convert the gradients back
% to a vector format (suitable for minFunc).  Specifically, we will unroll
% your gradient matrices into a vector.

grad = [W1grad(:) ; W2grad(:) ; b1grad(:) ; b2grad(:)];
%
end

%-------------------------------------------------------------------
% Here's an implementation of the sigmoid function, which you may find useful
% in your computation of the costs and the gradients.  This inputs a (row or
% column) vector (say (z1, z2, z3)) and returns (f(z1), f(z2), f(z3)).

function sigm = sigmoid(x)
sigm =  ./ ( + exp(-x));
end

%% 求解sigmoid函数的导数(这里的计算公式一定要注意啊，出过错)
function dsigm = dsigmoid(x)
sigx = sigmoid(x);
dsigm=sigx.*(-sigx);
end

梯度检验函数见另一篇博文