Gaussian Models

Warming Up

Before we talk about multivariate Gaussian, let's first review univariate Gaussian, which is usually called "Normal Distribution":
\[
X \sim N(\mu,\ \sigma^2) = \frac{1}{\sqrt{2\pi}\sigma} e^{ -\frac{(x-\mu)^2}{2\sigma^2}}
\]
where \(\mu=\mathbb{E}(X)\), \(\sigma = \mathrm{var}(X)\).

Now, if we have bivariate form of \(X = [x_1\ x_2]\), and also assume \(x_1\) and \(x_2\) are statistically independent, then we can get the joint distribution:
\[
\begin{align*}\notag
\mathrm{P}(x_1,x_2) &= \mathrm{P}(x_1)\mathrm{P}(x_2) \\
&=\frac{1}{\sqrt{2\pi}\sigma} e^{ -\frac{(x_1-\mu_1)^2}{2\sigma^2}} \frac{1}{\sqrt{2\pi}\sigma} e^{ -\frac{(x_2-\mu_2)^2}{2\sigma^2}} \\
&= \frac{1}{\left( \sqrt{2\pi} \sigma \right)^2} \exp \left\{ -\frac{(x_1-\mu_1)^2}{2\sigma^2} - \frac{(x_2-\mu_2)^2}{2\sigma^2} \right\} \\
&=\frac{1}{\left( \sqrt{2\pi} \sigma \right)^2} \exp \left\{ -\frac{1}{2} \left[ (x_1-\mu_1) \sigma^{-2} (x_1-\mu_1) + (x_2-\mu_2) \sigma^{-2} (x_2-\mu_2) \right] \right\}
\end{align*}
\]

Rewrite formula into matrix form:
\[
\frac{1}{\left( \sqrt{2\pi} \sigma \right)^2} \exp \left\{ -\frac{1}{2} \begin{bmatrix} (x_1-\mu_1)^\mathtt{T} \sigma^{-2} & (x_2-\mu_2)^\mathtt{T} \sigma^{-2} \end{bmatrix}
\begin{bmatrix}
(x_1-\mu_1) \\ (x_2-\mu_2)
\end{bmatrix} \right\} \\
= \frac{1}{\left( \sqrt{2\pi} \sigma \right)^2} \exp \left\{ -\frac{1}{2}
\begin{bmatrix} (x_1-\mu_1)^\mathtt{T} & (x_2-\mu_2)^\mathtt{T} \end{bmatrix}
\begin{bmatrix} \sigma^{-2} & 0 \\ 0 & \sigma^{-2} \end{bmatrix}
\begin{bmatrix} (x_1-\mu_1) \\ (x_2-\mu_2) \end{bmatrix}
\right\}
\]

Let \(\begin{bmatrix}\sigma^{-2} & 0 \\ 0 & \sigma^{-2}\end{bmatrix} = \Sigma^{-1},\mathbf{x}=\begin{bmatrix} x_1 \\ x_2 \end{bmatrix}, \mathbf{\mu}= \begin{bmatrix} \mu_1 \\ \mu_2 \end{bmatrix}\), then we also get \(\Sigma = \begin{bmatrix} \sigma^2 & 0 \\ 0 & \sigma^2 \end{bmatrix}\) and \(\det(\Sigma)=\sigma^4\). Plug \(\Sigma,\mathbf{x},\mathbf{\mu}\) in equation above and we obtain:
\[
\frac{1}{\left( \sqrt{2\pi} \right)^2 \det (\Sigma)^{1/2} } \exp \left\{ -\frac{1}{2}
(\mathbf{x-\mu})^\mathtt{T} \Sigma^{-1} (\mathbf{x-\mu}) \right\}
\]
This is exactly the probability density distribution (PDF) of bivariate Gaussian distribution.

Multivariate Gaussian Distribution

In general, the PDF of multivariate Gaussian distribution (a.k.a. multivariate normal distribution, MVN) is as below:
\[
\frac{1}{\left( \sqrt{2\pi} \right)^d \det (\Sigma)^{1/2} } \exp \left\{ \frac{1}{2}
(\mathbf{x-\mu})^\mathtt{T} \Sigma^{-1} (\mathbf{x-\mu}) \right\}
\]

---

Written with StackEdit.