图Lasso求逆协方差矩阵(Graphical Lasso for inverse covariance matrix)

作者:凯鲁嘎吉 - 博客园 http://www.cnblogs.com/kailugaji/

1. 图Lasso方法的基本理论

2. 坐标下降算法

3. 图Lasso算法

4. MATLAB程序

数据见参考文献[2]

4.1 方法一

demo.m

load SP500
data = normlization(data);
S = cov(data); %样本协方差
[X, W] = glasso_1(double(S), 0.5);
%X:sigma^(-1), W:sigma
[~, idx] = sort(info(:,3));
colormap gray
imagesc(X(idx, idx) == 0)
axis off %% Data Normalization
function data = normlization(data)
data = bsxfun(@minus, data, mean(data));
data = bsxfun(@rdivide, data, std(data));
end

glasso_1.m

function [X, W] = glasso_1(S, lambda)
%% Graphical Lasso - Friedman et. al, Biostatistics, 2008
% Input:
% S - 样本的协方差矩阵
% lambda - 罚参数
% Output:
% X - 精度矩阵 sigma^(-1)
% W - 协方差矩阵 sigma
%%
p = size(S,1); %数据维度
W = S + lambda * eye(p); %W=S+λI
beta = zeros(p) - lambda * eye(p); %β=-λI
eps = 1e-4;
finished = false(p); %finished:p*p的逻辑0矩阵
while true
for j = 1 : p
idx = 1 : p; idx(j) = [];
beta(idx, j) = lasso(W(idx, idx), S(idx, j), lambda, beta(idx, j));
W(idx, j) = W(idx,idx) * beta(idx, j); %W=W*β
W(j, idx) = W(idx, j);
end
index = (beta == 0);
finished(index) = (abs(W(index) - S(index)) <= lambda);
finished(~index) = (abs(W(~index) -S(~index) + lambda * sign(beta(~index))) < eps);
if finished
break;
end
end
X = zeros(p);
for j = 1 : p
idx = 1 : p; idx(j) = [];
X(j,j) = 1 / (W(j,j) - dot(W(idx,j), beta(idx,j)));
X(idx, j) = -1 * X(j, j) * beta(idx,j);
end
% X = sparse(X);
end

lasso.m

function w = lasso(A, b, lambda, w)
% Lasso
p = size(A,1);
df = A * w - b;
eps = 1e-4;
finished = false(1, p);
while true
for j = 1 : p
wtmp = w(j);
w(j) = soft(wtmp - df(j) / A(j,j), lambda / A(j,j));
if w(j) ~= wtmp
df = df + (w(j) - wtmp) * A(:, j); % update df
end
end
index = (w == 0);
finished(index) = (abs(df(index)) <= lambda);
finished(~index) = (abs(df(~index) + lambda * sign(w(~index))) < eps);
if finished
break;
end
end
end
%% Soft thresholding
function x = soft(x, lambda)
x = sign(x) * max(0, abs(x) - lambda);
end

结果

注意:罚参数lamda的设定对逆协方差的稀疏性的影响很大,可以用交叉验证方式得到。

4.2 方法二

graphicalLasso.m

function [Theta, W] = graphicalLasso(S, rho, maxIt, tol)
% http://www.ece.ubc.ca/~xiaohuic/code/glasso/glasso.htm
% Solve the graphical Lasso
% minimize_{Theta > 0} tr(S*Theta) - logdet(Theta) + rho * ||Theta||_1
% Ref: Friedman et al. (2007) Sparse inverse covariance estimation with the
% graphical lasso. Biostatistics.
% Note: This function needs to call an algorithm that solves the Lasso
% problem. Here, we choose to use to the function *lassoShooting* (shooting
% algorithm) for this purpose. However, any Lasso algorithm in the
% penelized form will work.
%
% Input:
% S -- sample covariance matrix
% rho -- regularization parameter
% maxIt -- maximum number of iterations
% tol -- convergence tolerance level
%
% Output:
% Theta -- inverse covariance matrix estimate
% W -- regularized covariance matrix estimate, W = Theta^-1 p = size(S,1); if nargin < 4, tol = 1e-6; end
if nargin < 3, maxIt = 1e2; end % Initialization
W = S + rho * eye(p); % diagonal of W remains unchanged
W_old = W;
i = 0; % Graphical Lasso loop
while i < maxIt,
i = i+1;
for j = p:-1:1,
jminus = setdiff(1:p,j);
[V D] = eig(W(jminus,jminus));
d = diag(D);
X = V * diag(sqrt(d)) * V'; % W_11^(1/2)
Y = V * diag(1./sqrt(d)) * V' * S(jminus,j); % W_11^(-1/2) * s_12
b = lassoShooting(X, Y, rho, maxIt, tol);
W(jminus,j) = W(jminus,jminus) * b;
W(j,jminus) = W(jminus,j)';
end
% Stop criterion
if norm(W-W_old,1) < tol,
break;
end
W_old = W;
end
if i == maxIt,
fprintf('%s\n', 'Maximum number of iteration reached, glasso may not converge.');
end Theta = W^-1; % Shooting algorithm for Lasso (unstandardized version)
function b = lassoShooting(X, Y, lambda, maxIt, tol), if nargin < 4, tol = 1e-6; end
if nargin < 3, maxIt = 1e2; end % Initialization
[n,p] = size(X);
if p > n,
b = zeros(p,1); % From the null model, if p > n
else
b = X \ Y; % From the OLS estimate, if p <= n
end
b_old = b;
i = 0; % Precompute X'X and X'Y
XTX = X'*X;
XTY = X'*Y; % Shooting loop
while i < maxIt,
i = i+1;
for j = 1:p,
jminus = setdiff(1:p,j);
S0 = XTX(j,jminus)*b(jminus) - XTY(j); % S0 = X(:,j)'*(X(:,jminus)*b(jminus)-Y)
if S0 > lambda,
b(j) = (lambda-S0) / norm(X(:,j),2)^2;
elseif S0 < -lambda,
b(j) = -(lambda+S0) / norm(X(:,j),2)^2;
else
b(j) = 0;
end
end
delta = norm(b-b_old,1); % Norm change during successive iterations
if delta < tol, break; end
b_old = b;
end
if i == maxIt,
fprintf('%s\n', 'Maximum number of iteration reached, shooting may not converge.');
end

结果

>> A=[5.9436    0.0676    0.5844   -0.0143
0.0676 0.5347 -0.0797 -0.0115
0.5844 -0.0797 6.3648 -0.1302
-0.0143 -0.0115 -0.1302 0.2389
];
>> [Theta, W] = graphicalLasso(A, 1e-4) Theta = 0.1701 -0.0238 -0.0159 0.0003
-0.0238 1.8792 0.0278 0.1034
-0.0159 0.0278 0.1607 0.0879
0.0003 0.1034 0.0879 4.2369 W = 5.9437 0.0675 0.5843 -0.0142
0.0675 0.5348 -0.0796 -0.0114
0.5843 -0.0796 6.3649 -0.1301
-0.0142 -0.0114 -0.1301 0.2390

5. 补充:近端梯度下降(Proximal Gradient Descent, PGD)求解Lasso问题

6. 参考文献

[1] 林祝莹. 图Lasso及相关方法的研究与应用[D].燕山大学,2016.

[2] Graphical Lasso for sparse inverse covariance selection

[3] 周志华. 机器学习[M]. 清华大学出版社, 2016.

[4] Graphical lasso in R and Matlab

[5] Graphical Lasso

图Lasso求逆协方差矩阵(Graphical Lasso for inverse covariance matrix)的更多相关文章

  1. ZOJ3574(归并排序求逆数对)

    Under Attack II Time Limit: 5 Seconds      Memory Limit: 65536 KB Because of the sucessfully calcula ...

  2. Day2:T4求逆序对(树状数组+归并排序)

    T4: 求逆序对 A[I]为前缀和 推导 (A[J]-A[I])/(J-I)>=M A[j]-A[I]>=M(J-I) A[J]-M*J>=A[I]-M*I 设B[]=A[]-M*( ...

  3. 洛谷P4841 城市规划(生成函数 多项式求逆)

    题意 链接 Sol Orz yyb 一开始想的是直接设\(f_i\)表示\(i\)个点的无向联通图个数,枚举最后一个联通块转移,发现有一种情况转移不到... 正解是先设\(g(n)\)表示\(n\)个 ...

  4. 【bzoj3456】城市规划(多项式求逆+dp)

    Description 求\(~n~\)个点组成的有标号无向连通图的个数.\(~1 \leq n \leq 13 \times 10 ^ 4~\). Solution 这道题的弱化版是poj1737, ...

  5. POJ 2299树状数组求逆序对

    求逆序对最常用的方法就是树状数组了,确实,树状数组是非常优秀的一种算法.在做POJ2299时,接触到了这个算法,理解起来还是有一定难度的,那么下面我就总结一下思路: 首先:因为题目中a[i]可以到99 ...

  6. [bzoj3456] 城市规划 [递推+多项式求逆]

    题面 bzoj权限题面 离线题面 思路 orz Miskcoo ! 先考虑怎么算这个图的数量 设$f(i)$表示$i$个点的联通有标号无向图个数,$g(i)$表示$n$个点的有标号无向图个数(可以不连 ...

  7. SGU180 Inversions(树状数组求逆序数)

    题目: 思路:先离散化数据然后树状数组搞一下求逆序数. 离散化的方法:https://blog.csdn.net/gokou_ruri/article/details/7723378 自己对用树状数组 ...

  8. <Sicily>Inversion Number(线段树求逆序数)

    一.题目描述 There is a permutation P with n integers from 1 to n. You have to calculate its inversion num ...

  9. JDOJ 1927 求逆序对

    洛谷 P1908 逆序对 洛谷传送门 JDOJ 1927: 求逆序对 JDOJ传送门 题目描述 猫猫TOM和小老鼠JERRY最近又较量上了,但是毕竟都是成年人,他们已经不喜欢再玩那种你追我赶的游戏,现 ...

随机推荐

  1. /dev/random 和 /dev/urandmon的差别

    最近使用这两个设备的时候,发现 /dev/random生成随机数很慢:于是就查了查: 这两个设备的差异在于:/dev/random的random pool依赖于系统中断,因此在系统的中断数不足时,/d ...

  2. CodeForces - 763A(并查集/思维)

    题意 https://vjudge.net/problem/CodeForces-763A 一棵无根树中各个节点被染上了一种颜色c[i] 现在让你选择一个点作为根节点,使得这个根节点的所有儿子满足以该 ...

  3. Delphi 设置程序图标为系统默认图标

    Windows VCL 程序: 1. 用文本编辑器打开Delphi项目的.dproj文件.2. 搜索关键字“<Icon_MainIcon>”,查找Icon_MainIcon元素.3. 将所 ...

  4. 《算法问题实战策略》 BOGGLE

    oj地址是韩国网站 连接比较慢 https://algospot.com/judge/problem/read/BOGGLE大意如下 输入输出 输入 URLPM XPRET GIAET XTNZY X ...

  5. css发展史

      (接着上次博客的内容,现在我们进入css基础)   外部样式表  <link> 内部样式表  <style> 行内样式表  style  (多用于JS) * css注释   ...

  6. A bean with that name has already been defined in class path resource [org/springframework/transaction/annotation/ProxyTransactionManagementConfiguration.class] and overriding is disabled

    2019-12-19 13:26:17.594 WARN [main] o.s.boot.web.servlet.context.AnnotationConfigServletWebServerApp ...

  7. HDU 6556 (2018CCPC吉林 B题)

    ### HDU 6556 题目链接 ### 题目大意: 给你四个国家的时区,告诉你 A 国家的时间,让你输出这时候在 B 国家的时间,还需要输出对于 A 国家来说这是 昨天.今天 还是 明天. 分析前 ...

  8. virtualbox FAIL(0x80004005) VirtualBox VT-x is not available (VERR_VMX_NO_VMX)

    virtualbox启动虚拟机报错: FAIL(0x80004005) VirtualBox VT-x is not available (VERR_VMX_NO_VMX),无法创建新任务 这是win ...

  9. js使用工具将表单封装成json字符串传到后台,js截取字符串(学生笔记)

    <script src="js/jquery.min.js"></script> <script src="https://cdn.boot ...

  10. Python【day 17】面向对象-成员

    类的变量分成2种: 1.成员变量 概念:在构造方法中的变量,前面带有self 作用:可以在类中不同的方法间使用 2.类变量-静态变量 概念:在类中,构造方法和普通方法之外,定义的变量 作用: 1.调用 ...