Derivative of Softmax Loss Function

Derivative of Softmax Loss Function

A softmax classifier:
\[
p_j = \frac{\exp{o_j}}{\sum_{k}\exp{o_k}}
\]
It has been used in a loss function of the form
\[
L = - \sum_{j} y_j \log p_j
\]
where o is a vector. We need the derivative of \(L\) with respect to \(o\). We can get the partial of \(o_i\) :
\[
\frac{\partial{p_j}}{\partial{o_i}} = p_i (1-p_i), \quad i = j \\
\frac{\partial{p_j}}{\partial{o_i}} = - p_i p_j, \quad i \ne j
\]
Hence the derivative of Loss with respect to \(o\) is:

\[
\begin{align}
\frac{\partial{L}}{\partial{o_i}} & = - \sum_k y_k \frac{\partial{\log p_k}}{\partial{o_i}} \\
& = - \sum_k y_k \frac{1}{p_k} \frac{\partial{p_k}}{\partial{o_i}} \\
& = -y_i(1-p_i) - \sum_{k\ne i} y_k \frac{1}{p_k} (-p_kp_i) \\
& = -y_i + y_i p_i + \sum_{k\ne i} y_k p_i \\
& = p_i (\sum_k y_k) - y_i \\
\end{align}
\]
Given that \(\sum_k y_k = 1\) as \(y\) is a vector with only one non-zero element, which is 1. By other words, this is a classification problem.
\[
\frac{\partial L}{\partial o_i} = p_i - y_i
\]

Reference

Derivative of Softmax loss function