nltk处理文本

nltk(Natural Language Toolkit)是处理文本的利器。

安装

pip install nltk

进入python命令行，键入nltk.download()可以下载nltk需要的语料库等等。

分词

按词语分割（传入句子）

sentence='hello,world!'
tokens=nltk.word_tokenize(sentence)

tokens就是一个分割好的词表，如下：

['hello', ',', 'world', '!']

按句子分割（传入多个句子组成的文档）

text='This is a text. I want to split it.'
sens=nltk.sent_tokenize(text)

sens就是分割好的句子组成的list,如下：

['This is a text.', 'I want to split it.']

词性标注

tags = [nltk.pos_tag(tokens) for tokens in words]
[[('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('text', 'NN'), ('for', 'IN'), ('test', 'NN'), ('.', '.')], [('And', 'CC'), ('I', 'PRP'), ('want', 'VBP'), ('to', 'TO'), ('learn', 'VB'), ('how', 'WRB'), ('to', 'TO'), ('use', 'VB'), ('nltk', 'NN'), ('.', '.')]]

附录：nltk的词性：

1.  CC      Coordinating conjunction 连接词
   

2. CD     Cardinal number  基数词
   

3. DT     Determiner  限定词（如this,that,these,those,such，不定限定词：no,some,any,each,every,enough,either,neither,all,both,half,several,many,much,(a) few,(a) little,other,another.
   

4. EX     Existential there 存在句
   

5. FW     Foreign word 外来词
   

6. IN     Preposition or subordinating conjunction 介词或从属连词
   

7. JJ     Adjective 形容词或序数词
   

8. JJR     Adjective, comparative 形容词比较级
   

9. JJS     Adjective, superlative 形容词最高级
   

10. LS     List item marker 列表标示
    

11. MD     Modal 情态助动词
    

12. NN     Noun, singular or mass 常用名词 单数形式
    

13. NNS     Noun, plural  常用名词 复数形式
    

14. NNP     Proper noun, singular  专有名词，单数形式
    

15. NNPS     Proper noun, plural  专有名词，复数形式
    

16. PDT     Predeterminer 前位限定词
    

17. POS     Possessive ending 所有格结束词
    

18. PRP     Personal pronoun 人称代词
    

19. PRP$     Possessive pronoun 所有格代名词
    

20. RB     Adverb 副词
    

21. RBR     Adverb, comparative 副词比较级
    

22. RBS     Adverb, superlative 副词最高级
    

23. RP     Particle 小品词
    

24. SYM     Symbol 符号
    

25. TO     to 作为介词或不定式格式
    

26. UH     Interjection 感叹词
    

27. VB     Verb, base form 动词基本形式
    

28. VBD     Verb, past tense 动词过去式
    

29. VBG     Verb, gerund or present participle 动名词和现在分词
    

30. VBN     Verb, past participle 过去分词
    

31. VBP     Verb, non-3rd person singular present 动词非第三人称单数
    

32. VBZ     Verb, 3rd person singular present 动词第三人称单数
    

33. WDT     Wh-determiner 限定词（如关系限定词：whose,which.疑问限定词：what,which,whose.）
    

34. WP      Wh-pronoun 代词（who whose which）
    

35. WP$     Possessive wh-pronoun 所有格代词
    

36. WRB     Wh-adverb   疑问代词（how where when）
    

提取关键词

如何对一段话提取关键词呢？主要思想就是先分词，再标词性。

# -*- coding=UTF-8 -*-
import nltk
from nltk.corpus import brown
from nltk.stem import SnowballStemmer
from nltk.corpus import stopwords

# This is our fast Part of Speech tagger
#############################################################################
brown_train = brown.tagged_sents(categories='news')
regexp_tagger = nltk.RegexpTagger(
    [(r'^-?[0-9]+(.[0-9]+)?$', 'CD'),
    (r'(-|:|;)$', ':'),
    (r'\'*$', 'MD'),
    (r'(The|the|A|a|An|an)$', 'AT'),
    (r'.*able$', 'JJ'),
    (r'^[A-Z].*$', 'NNP'),
    (r'.*ness$', 'NN'),
    (r'.*ly$', 'RB'),
    (r'.*s$', 'NNS'),
    (r'.*ing$', 'VBG'),
    (r'.*ed$', 'VBD'),
    (r'.*', 'NN')
])
unigram_tagger = nltk.UnigramTagger(brown_train, backoff=regexp_tagger)
bigram_tagger = nltk.BigramTagger(brown_train, backoff=unigram_tagger)
#############################################################################

# This is our semi-CFG; Extend it according to your own needs
#############################################################################
cfg = {}
cfg["NNP+NNP"] = "NNP"
cfg["NN+NN"] = "NNI"
cfg["NNI+NN"] = "NNI"
cfg["JJ+JJ"] = "JJ"
cfg["JJ+NN"] = "NNI"
#############################################################################

class NPExtractor(object):
    # Split the sentence into singlw words/tokens
    def tokenize_sentence(self, sentence):
        tokens = nltk.word_tokenize(sentence)
        #去除停用词,标点，数字,长度小于2的词
        tokens=[w.lower() for w in tokens if(w.isalpha())&(len(w)>1)]#使用tfid，不必去除停用词
        #词干提取
        stemmer=SnowballStemmer('english')
        tokens=[stemmer.stem(w) for w in tokens]
        return tokens

    # Normalize brown corpus' tags ("NN", "NN-PL", "NNS" > "NN")
    def normalize_tags(self, tagged):
        n_tagged = []
        for t in tagged:
            if t[1] == "NP-TL" or t[1] == "NP":
                n_tagged.append((t[0], "NNP"))
                continue
            if t[1].endswith("-TL"):
                n_tagged.append((t[0], t[1][:-3]))
                continue
            if t[1].endswith("S"):
                n_tagged.append((t[0], t[1][:-1]))
                continue
            n_tagged.append((t[0], t[1]))
        return n_tagged

    # Extract the main topics from the sentence
    def extract(self,sentence):

        tokens = self.tokenize_sentence(sentence)
        tags = self.normalize_tags(bigram_tagger.tag(tokens))

        merge = True
        while merge:
            merge = False
            for x in range(0, len(tags) - 1):
                t1 = tags[x]
                t2 = tags[x + 1]
                key = "%s+%s" % (t1[1], t2[1])
                value = cfg.get(key, '')
                if value:
                    merge = True
                    tags.pop(x)
                    tags.pop(x)
                    match = "%s %s" % (t1[0], t2[0])
                    pos = value
                    tags.insert(x, (match, pos))
                    break

        matches = []
        for t in tags:
            if t[1] == "NNP" or t[1] == "NNI" or t[1]=="NN":
                matches.append(t[0])
        return matches

利用这里的extract函数就可以提取文本的关键词。

更多参见nltk官方文档：nltk