N-Gram的数据结构

ARPA的n-gram语法如下：

[html] view plaincopyprint?
\data\
ngram 1=64000
ngram 2=522530
ngram 3=173445  

\1-grams:
-5.24036        'cause  -0.2084827
-4.675221       'em     -0.221857
-4.989297       'n      -0.05809768
-5.365303       'til    -0.1855581
-2.111539       </s>    0.0
-99     <s>     -0.7736475
-1.128404       <unk>   -0.8049794
-2.271447       a       -0.6163939
-5.174762       a's     -0.03869072
-3.384722       a.      -0.1877073
-5.789208       a.'s    0.0
-6.000091       aachen  0.0
-4.707208       aaron   -0.2046838
-5.580914       aaron's -0.06230035
-5.789208       aarons  -0.07077657
-5.881973       aaronson        -0.2173971

具体说明见 ：ARPA的n-gram语言模型格式

整个ARPA-LM由很多个n-gram项组成，分别说明这两个的数据结构

一，n-gram数据结构

n-gram的数据结构如下：

typedef struct
{
    real        log_prob ;
    real        log_bo ;
    int         *words ;
} ARPALMEntry ;

words，表示当前的n-gram所涉及的单词，如果是1-gram，那就只有一个，如果是2-gram，那么words就包括这两个单词的序号。

log_bo，表示ngram的回退概率。

log_prob，表示ngram的组合概率。

二，ARPA-LM数据结构

多个项组成的整个n-gram语言模型的数据结构如下：

[cpp] view plaincopyprint?

class ARPALM
 {
    public:
        Vocabulary *vocab ;  

        int            order ;
        ARPALMEntry    **entries ; // 语言模型的所有项，组成一个数组
        int            *n_ngrams ; // 一元语言模型、二元语言模型、三元语言模型等组成的数组，数组每一项都表示对应的的元有多少个。  

        char           *unk_wrd ; // 词典中不在语言模型中的词。
        int            unk_id ;// 词典中不在语言模型中的词ID，这个ID指定为词典的最后一个序号。  

        int            n_unk_words ;
        int            *unk_words ;
    private:
        bool           *words_in_lm ; // 布尔类型数组，标识词是否在语言模型中。
}

vocab，用于构建语言模型的词典指针。词典定义见：词典内存存储模型

entries，语言模型的所有ngram项，是ARPALMEntry类型的一个二维数组。entries[0]存储1-gram，entries[1]存储2-gram，依此类推。

n_ngrams，整型数组，依次包含1-gram,2-gram,3-gram,....所包含的ngram项个数。

unk_wrd，词典中可以不在语言模型中的词。

unk_id，词典中可以不在语言模型中的词的ID，这个ID指定为词典的最后一个词序号。

n_unk_words，在读语言模型之后，统计在词典中，但没有用来建立语言模型的词个数，如果没有指定unk_wrd的话，是不允许的，就表示所有的词典中的词都应该用来建语言模型。

unk_words，存储6中统计的词序号。

words_in_lm，这个标识词典中的词是否在语言模型中出现。