Matrix Calculus

1 Scalar Function

\(\text{If }f(\mathbf{x})\in\mathbf{R},\mathrm{then}\)

\[df=\frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz=\begin{bmatrix}\frac{\partial f}{\partial x}&\frac{\partial f}{\partial y}&\frac{\partial f}{\partial z}\end{bmatrix}\begin{bmatrix}dx\\dy\\dz\end{bmatrix}= f(\mathbf{x})' d\mathbf{x}.
\]

1.1 Derivative

So

\[\dfrac{\partial f}{\partial\mathbf{x}}=\begin{bmatrix}\dfrac{\partial f}{\partial x}&\dfrac{\partial f}{\partial y}&\dfrac{\partial f}{\partial z}\end{bmatrix} = f(\mathbf{x})'
\]

这里用的是 Numerator layout.

1.2 Gradient

\[\left.\nabla f=\left(\frac{\partial f}{\partial\mathbf{x}}\right)^{\mathsf{T}}=\left[\begin{array}{ccc}\frac{\partial f}{\partial x}&\frac{\partial f}{\partial y}&\frac{\partial f}{\partial z}\end{array}\right.\right]^{\mathsf{T}} = \begin{bmatrix}\frac{\partial f}{\partial x}\\\frac{\partial f}{\partial y}\\\frac{\partial f}{\partial z}\end{bmatrix}.
\]

2 Vector Function

\(\text{if }\mathbf{f}(\mathbf{x})=\begin{bmatrix}f(\mathbf{x})\\g(\mathbf{x})\\h(\mathbf{x})\end{bmatrix}\in\mathbf{R}^3,\text{then:}\)

2.1 Jacobian

\[\mathbf{J}(\mathbf{x})=\dfrac{\partial\mathbf{f}}{\partial\mathbf{x}}=\begin{bmatrix}\frac{\partial f}{\partial x}&\frac{\partial f}{\partial y}&\frac{\partial f}{\partial z}\\\frac{\partial g}{\partial x}&\frac{\partial g}{\partial y}&\frac{\partial g}{\partial z}\\\frac{\partial h}{\partial x}&\frac{\partial h}{\partial y}&\frac{\partial h}{\partial z}\end{bmatrix}
\]

2.2 Divergence

\[\nabla\cdot\mathbf{f}=\frac{\partial f}{\partial x}+\frac{\partial g}{\partial y}+\frac{\partial h}{\partial z}
\]

2.3 Curl

\[\nabla\times\mathbf{f}=\begin{bmatrix}\frac{\partial h}{\partial y}-\frac{\partial g}{\partial z}\\\frac{\partial f}{\partial z}-\frac{\partial h}{\partial x}\\\frac{\partial g}{\partial x}-\frac{\partial f}{\partial y}\end{bmatrix}
\]

2.4 Hessian

\[\mathbf{H}=\mathbf{J}(\nabla f(\mathbf{x}))=\begin{bmatrix}\frac{\partial^2f}{\partial x^2}&\frac{\partial^2f}{\partial x\partial y}&\frac{\partial^2f}{\partial x\partial z}\\\frac{\partial^2f}{\partial y\partial x}&\frac{\partial^2f}{\partial y^2}&\frac{\partial^2f}{\partial y\partial z}\\\frac{\partial^2f}{\partial z \partial x}&\frac{\partial^2f}{\partial z \partial y}&\frac{\partial^2f}{\partial z^2}\end{bmatrix}
\]

https://www.value-at-risk.net/functions/

X ref

1. HKU : Vector calculus for engineers
2. MIT : s096-matrix-calculus-for-machine-learning
3. GAMES103 : Physics-Based Animation
4. Wikipedia : Matrix Calculus