Semantic Compositionality through Recursive Matrix-Vector Spaces-paper

Semantic Compositionality through Recursive Matrix-Vector Spaces

作者信息：
Richard Socher Brody Huval Christopher D. Manning Andrew Y. Ng
richard@socher.org, {brodyh,manning,ang}@stanford.edu
Computer Science Department, Stanford University
代码数据公开：
https://www.socher.org/index.php/Main/

MX-RNN模型：
We present a novel recursive neural network model for semantic compositionality. In our context,compositionality is the ability to learn compositional vector representations for various types of phrases and sentences of arbitrary length.
不需要任何hand-designed semantic resources比如WorNet，FrameNet
通过加入WordNet hypernyms, POS and NER tags性能更好

2 MV-RNN: A Recursive Matrix-Vector Model
之前多个词向量组合成一个短语或者句子是用的线性关系，但是只能捕捉‘sum’的关系，不能捕捉可以改变另外一个词的含义的词的功能，例如extremely strong
1)给每个词分配了一个向量和一个矩阵
2) learning an input-specific, nonlinear, compositional function for computing vector and matrix representations for multi-word sequences of any syntactic type
如果一个词缺少operator semantics，那么它的矩阵就是一个单位矩阵；然而，如果一个词主要是作为operator，比如extremely，那么它的向量会接近0，它的矩阵gain a clear operator meaning，正向和负向都会增大被修饰词的含义

2.2 Matrix-Vector Neural Word Representation
与训练词向量为50维，矩阵50*50维，初始化矩阵是单位矩阵+高斯噪声，所以每个句子表示为（（单词向量1，单词矩阵1），。。。，（），（））

2.2 composition models for two words
2010年的工作，用了表示句法关系的矩阵R，背景知识的矩阵K
我们的工作：u需要任何手工涉及的semantic resources比如背景知识K，也不需要explicit knownledge of relation R，使用输入independant的组合函数
W是n*2n，可以将输入的词统一到相同的维度，W可以捕捉compositional信息
得到的是p

2.3 Recursive Compositions of Multiple Words and Phrases
This section describes how we extend a word-pair matrix-vector-based compositional model to learn vectors and matrices for longer sequences of words
Wm
得到的是P

2.4 Objective Functions for Training
One of the advantages of RNN-based models is that each node of a tree has associated with it a distributed vector representation (the parent vector p) which can also be seen as features describing that
phrase.
softmax对节点p分类

2.5learning

2.6 Low-Rank Matrix Approximations

5 combination od semantic relationship
semantic relationships between pairs of nominals. For instance, in the sentence “My [apartment]e1 has a pretty large [kitchen]e2.”, we want to predict that the kitchen and apartment are in a component-whole relationship.
figure5分类名词关系：
1）先找到要分类的两个词
2)使用节点向量分类
数据是9个顺序关系（两个方向）*2+没有方向的一个 = 19类，比如因果，文本-主题