3阶马尔可夫链 自然语言处理python

一、简介：

      把每三个三个单词作为一个整体进行训练。

举一个例子：

input:

      my dream is that I can be an engineer, so I design more applications for people to use.

      my dream is that I can be a bird, so I can fly to everywhere I want.

      it is also my dream that I can be a house, so I can warm you in the cold winter.

生成的马尔可夫链：

    

{'START': ['my dream is'], 'my dream is': ['that i can'], 'dream is that': ['i can be'], 'is that i': ['can be a'], 'that i can': ['be a house,'], 'i can be': ['a house, so'], 'can be an': ['engineer, so i'], 'be an engineer,': ['so i design'], 'an engineer, so': ['i design more'], 'engineer, so i': ['design more applications'], 'so i design': ['more applications for'], 'i design more': ['applications for people'], 'design more applications': ['for people to'], 'more applications for': ['people to use.\nmy'], 'applications for people': ['to use.\nmy dream'], 'for people to': ['use.\nmy dream is'], 'people to use.\nmy': ['dream is that'], 'to use.\nmy dream': ['is that i'], 'use.\nmy dream is': ['that i can'], 'can be a': ['house, so i'], 'be a bird,': ['so i can'], 'a bird, so': ['i can fly'], 'bird, so i': ['can fly to'], 'so i can': ['warm you in'], 'i can fly': ['to everywhere i'], 'can fly to': ['everywhere i want.\nit'], 'fly to everywhere': ['i want.\nit is'], 'to everywhere i': ['want.\nit is also'], 'everywhere i want.\nit': ['is also my'], 'i want.\nit is': ['also my dream'], 'want.\nit is also': ['my dream that'], 'is also my': ['dream that i'], 'also my dream': ['that i can'], 'my dream that': ['i can be'], 'dream that i': ['can be a'], 'be a house,': ['so i can'], 'a house, so': ['i can warm'], 'house, so i': ['can warm you'], 'i can warm': ['you in the'], 'can warm you': ['in the cold'], 'warm you in': ['the cold winter.'], 'you in the': ['cold winter.'], 'in the cold': ['winter.'], 'END': ['the cold winter.', 'winter.', 'cold winter.']}

生成的文本：

my dream is that i can be a house, so i can warm you in the cold winter.

代码:

 # I sperate the input.txt with space, and use dictionary to store the next three words after the current 3 words.
 # in the same time, store the first three word as the beginning, and the last three or two or one words as the end
 # how to generate output.txt: form the start, start to look for the next three words in ramdom, once meets the end, the geration is end.
 import random

 fhand=open("E:\\a2.txt",'r',encoding='UTF-8')
 dataset_file=fhand.read()

 # dataset_file='my friend makes the best raspberry pies'
 dataset_file=dataset_file.lower().split(' ')
 model={}

 for i, word in enumerate(dataset_file):
     if i == len(dataset_file) - 3:
         model['END'] = model.get('END', []) + [dataset_file[i] + " " + dataset_file[i + 1] + " " + dataset_file[i + 2]]
         model['END'] = model.get('END', []) + [dataset_file[i + 2]]
         model['END'] = model.get('END', []) + [dataset_file[i + 1] +" "+dataset_file[i + 2]]
     elif i == 0:
         model['START'] = model.get('START', []) + [dataset_file[i] + " " + dataset_file[i + 1] + " " + dataset_file[i + 2]]
         # model['START']=model.get('START',[])+[dataset_file[i]]
         # model['START']=model.get('START',[])+[dataset_file[i]+" "+dataset_file[i+1]]
         model[dataset_file[i] + " " + dataset_file[i + 1] + " " + dataset_file[i + 2]] = model.get(word, []) + [
             dataset_file[i + 3] + " " + dataset_file[i + 4] + " " + dataset_file[i + 5]]
     elif i <= (len(dataset_file) - 6):
         model[dataset_file[i] + " " + dataset_file[i + 1] + " " + dataset_file[i + 2]] = model.get(word, []) + [
             dataset_file[i + 3] + " " + dataset_file[i + 4] + " " + dataset_file[i + 5]]
     elif i == (len(dataset_file) - 5):
         model[dataset_file[i] + " " + dataset_file[i + 1] + " " + dataset_file[i + 2]] = model.get(word, []) + [
             dataset_file[i + 3] + " " + dataset_file[i + 4]]
     elif i == (len(dataset_file) - 4):
         model[dataset_file[i] + " " + dataset_file[i + 1] + " " + dataset_file[i + 2]] = model.get(word, []) + [
             dataset_file[i + 3]]
 print(model)

 generated = []
 while True:
     if not generated:
         words = model['START']
     elif generated[-1] in model['END']:
         break
     else:
         words = model[generated[-1]]
     generated.append(random.choice(words))

 fhand=open("E:\output.txt",'a')
 for word in generated:
     fhand.write(word+" ")

     print(word,end=' ')