Noise Contrastive Estimation

Notes from Notes on Noise Contrastive Estimation and Negative Sampling
one sample:
\[x_i \to [y_i^0,\cdots,y_{i}^{k}]\]
where \(y_i^0\) are true labeled words , and \(y_i^1,\cdots,y_i^{k}\) are noise samples word index, which is generated by unigram distribution \(q(w)\) of the dataset.
the probability of true data:
\[p(y_i^0=1|x_i,\theta)=\frac{\exp(y_i^0,h_\theta)}{\exp(y_i^0 h_\theta) + k*q(y_i^0)}\]
the noise sample probability:
\[p(y_i^t=0|x_i,\theta)=\frac{k*q(y_i^t)}{\exp(y_i^t h_\theta) + k*q(y_i^t)},t=1,\cdots,k\]
the cost function of this sample:
\[l_{nce}=\log p(y_i^0|x_i,\theta)+\sum_{t=1}^k{\log p(y_i^t|x_i,\theta)}\]
the overall cost function of the dataset:
\[\mathcal{L}_{nce}=\frac{1}{N}\sum_i^N{\left\{\log p(y_i^0|x_i,\theta)+\sum_{t=1}^k{\log p(y_i^t|x_i,\theta)}\right\}}\]

Related Paper

[Noise-Contrastive Estimation of Unnormalized Statistical Models with Applications to Natural Image Statistics]

[Word2vec Parameter Learning Explained]

[Efficient Estimation of Word Representation in Vector Space]

[Distributed Representations of Words and Phrases and their Compositionality]

[Notes on Noise Contrastive Estimation and Negative Sampling]