[C3] 正则化(Regularization)

正则化（Regularization - Solving the Problem of Overfitting）

欠拟合(高偏差) VS 过度拟合(高方差)

Underfitting, or high bias, is when the form of our hypothesis function h maps poorly to the trend of the data.

It is usually caused by a function that is too simple or uses too few features.

欠拟合(高偏差)：没有很好的拟合训练集数据；

At the other extreme, overfitting, or high variance, is caused by a hypothesis function that fits the available data but does not generalize well to predict new data.

It is usually caused by a complicated function that creates a lot of unnecessary curves and angles unrelated to the data.

过度拟合(高方差)：可以很好的拟合训练集数据，但是函数太过庞大，变量太多，且缺少足够多的数据约束该模型(m < n)，无法泛化到新的数据样本。

This terminology is applied to both linear and logistic regression. There are two main options to address the issue of overfitting:

两种方法解决过度拟合：

Reduce the number of features

Manually select which features to keep
Use a model selection algorithm (studied later in the course)

Regularization

Keep all the features, but reduce the magnitude of parameters $\theta_j$.
Regularization works well when we have a lot of slightly useful features.

正则化 - 线性回归代价函数

所有正则化均不包括 $\theta_0$ 项

$J(\theta)=\frac{1}{2m} \Bigg[ \sum\limits_{i=1}^m \Big( h_\theta(x^{(i)}) - y^{(i)} \Big)^2 + \lambda \sum\limits_{j=1}^n \theta_j^2 \Bigg]$

向量化表示为（A vectorized implementation is）：

$\overrightarrow{h}=g(X \overrightarrow{\theta})$

$J(\theta)=\frac{1}{2m} \cdot \Bigg[ (\overrightarrow{h}-\overrightarrow{y})^T \cdot (\overrightarrow{h}-\overrightarrow{y}) + \lambda \cdot (\overrightarrow{l} \cdot \overrightarrow{\theta}^{.2}) \Bigg]$

$\overrightarrow{l} = [0, 1, 1, ...1]$

代码实现：

m = length(y);

l = ones(1, length(theta)); l(:,1) = 0;

J = 1/(2*m) * ((X * theta - y)' * (X * theta - y) + lambda * (l * (theta.^2));

or 

J = 1/(2*m) * ((X * theta - y)' * (X * theta - y) + lambda * (theta'*theta - theta(1,:).^2);

正则化 - 逻辑回归代价函数

所有正则化均不包括 $\theta_0$ 项

$J(\theta)=-\frac{1}{m} \sum\limits_{i=1}^m \Bigg[ y^{(i)} \cdot log \bigg(h_\theta(x^{(i)}) \bigg) + (1-y^{(i)}) \cdot log \bigg(1-h_\theta(x^{(i)}) \bigg) \Bigg] + \frac{\lambda}{2m} \sum\limits_{j=1}^n \theta_j^2$

向量化表示为（A vectorized implementation is）：

$\overrightarrow{h}=g(X \overrightarrow{\theta})$

$J(\theta)=\frac{1}{m} \cdot \Big( -\overrightarrow{y}^T \cdot log(\overrightarrow{h}) - (1- \overrightarrow{y})^T \cdot log(1- \overrightarrow{h}) \Big) + \frac{\lambda}{2m} (\overrightarrow{l} \cdot \overrightarrow{\theta}^{.2})$

$\overrightarrow{l} = [0, 1, 1, ...1]$

代码实现：

m = length(y);

l = ones(1, length(theta)); l(:,1) = 0;

J = (1/m)*(-y'*log(sigmoid(X*theta))-(1 - y)'* log(1-sigmoid(X*theta))) + ...

    (lambda/(2*m))*(l*(theta.^2)); 

or 

J = (1/m)*(-y'*log(sigmoid(X*theta))-(1 - y)'* log(1-sigmoid(X*theta))) + ...

    (lambda/(2*m))*(theta'*theta - theta(1,:).^2);

正则化后的线性回归和逻辑回归梯度下降

所有正则化均不包括 $\theta_0$ 项

$\begin{cases} \theta_0:=\theta_0 - \alpha \frac{1}{m} \sum\limits_{i=1}^m \Big( h_\theta(x^{(i)}) - y^{(i)} \Big) \cdot x_0^{(i)} \\ \\ \theta_j:=\theta_j - \alpha \Bigg[ \frac{1}{m} \sum\limits_{i=1}^m \Big( h_\theta(x^{(i)}) - y^{(i)} \Big) \cdot x_j^{(i)} + \frac{\lambda}{m} \cdot \theta_j \Bigg] \end{cases}$

向量化表示为（A vectorized implementation is）：

$\frac{1}{m} \cdot \Big( X^T \cdot (\overrightarrow{h} - \overrightarrow{y}) \Big) + \frac{\lambda}{m} \cdot \theta^{'}$

$\theta^{'} = \begin{bmatrix} 0\\[0.3em]\theta_1\\[0.3em]\theta_2\\[0.3em].\\[0.3em].\\[0.3em].\\[0.3em]\theta_n \end{bmatrix}$

代码实现：

reg_theta=theta; reg_theta(1, :) = 0;

grad = (1/m)*(X'*(sigmoid(X*theta) - y)) + (lambda/m)*reg_theta;

最终形式：对 $\theta_j$ 的梯度下降公式进行整理变形（With some manipulation our update rule can also be represented as）：

$\begin{cases} \theta_0:=\theta_0 - \alpha \frac{1}{m} \sum\limits_{i=1}^m \Big( h_\theta(x^{(i)}) - y^{(i)} \Big) \cdot x_0^{(i)} \\ \\ \theta_j:=\theta_j (1- \alpha \frac{\lambda}{m}) - \alpha \frac{1}{m} \sum\limits_{i=1}^m \Big( h_\theta(x^{(i)}) - y^{(i)} \Big) \cdot x_j^{(i)} \end{cases}$

对线性回归正规方程进行正则化

所有正则化均不包括 $\theta_0$ 项

$1 - \alpha\frac{\lambda}{m}$ will always be less than 1. Intuitively you can see it as reducing the value of $\theta_j$ by some amount on every update. Notice that the second term is now exactly the same as it was before.

Now let's approach regularization using the alternate method of the non-iterative normal equation.

To add in regularization, the equation is the same as our original, except that we add another term inside the parentheses:

原始形态 $\overrightarrow{\theta} = (X^TX)^{-1}X^T \overrightarrow{y}$

正则化后 $\overrightarrow{\theta} = (X^TX + \lambda L)^{-1}X^T \overrightarrow{y}$

$L = \begin{bmatrix} 0&&&&&&\\[0.3em]&1&&&&&\\[0.3em]&&1&&&&\\[0.3em]&&&·&&&\\[0.3em]&&&&·&&\\[0.3em]&&&&&·&\\[0.3em]&&&&&&1\end{bmatrix}$

L is a matrix with 0 at the top left and 1's down the diagonal, with 0's everywhere else. It should have dimension (n+1)×(n+1).

Intuitively, this is the identity matrix (though we are not including $x_0$）multiplied with a single real number $\lambda$.

Recall that if m < n, then $X^TX$ is non-invertible. However, when we add the term $\lambda⋅L$, then $X^TX + \lambda⋅L$ becomes invertible.

程序代码

正则化的特性已经全部添加到了其他练习代码中，如线性回归，逻辑回归，神经网络等。可在其他练习中查看到，如需非正则化，只要将Lambda=0即可。

获取源码以其他文件，可点击右上角 Fork me on GitHub 自行 Clone。