代码来源于:tensorflow机器学习实战指南(曾益强 译,2017年9月)——第七章:自然语言处理

代码地址:https://github.com/nfmcclure/tensorflow-cookbook

在讲述skip-gram,CBOW,Word2Vec,Doc2Vec模型时需要复用的函数

  • 加载数据函数
  • 归一化文本函数
  • 生成词汇表函数
  • 生成单词索引表
  • 生成批量数据函数

加载数据函数

# Load the movie review data
# Check if data was downloaded, otherwise download it and save for future use
def load_movie_data(data_folder_name):
pos_file = os.path.join(data_folder_name, 'rt-polarity.pos')
neg_file = os.path.join(data_folder_name, 'rt-polarity.neg') # Check if files are already downloaded
if os.path.isfile(pos_file):
pos_data = []
with open(pos_file, 'r') as temp_pos_file:
for row in temp_pos_file:
pos_data.append(row)
neg_data = []
with open(neg_file, 'r') as temp_neg_file:
for row in temp_neg_file:
neg_data.append(row)
else: # If not downloaded, download and save
movie_data_url = 'http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz'
stream_data = urllib.request.urlopen(movie_data_url)
tmp = io.BytesIO()
while True:
s = stream_data.read(16384)
if not s:
break
tmp.write(s)
stream_data.close()
tmp.seek(0) tar_file = tarfile.open(fileobj=tmp, mode="r:gz")
pos = tar_file.extractfile('rt-polaritydata/rt-polarity.pos')
neg = tar_file.extractfile('rt-polaritydata/rt-polarity.neg')
# Save pos/neg reviews
pos_data = []
for line in pos:
pos_data.append(line.decode('ISO-8859-1').encode('ascii',errors='ignore').decode())
neg_data = []
for line in neg:
neg_data.append(line.decode('ISO-8859-1').encode('ascii',errors='ignore').decode())
tar_file.close()
# Write to file
if not os.path.exists(save_folder_name):
os.makedirs(save_folder_name)
# Save files
with open(pos_file, "w") as pos_file_handler:
pos_file_handler.write(''.join(pos_data))
with open(neg_file, "w") as neg_file_handler:
neg_file_handler.write(''.join(neg_data))
texts = pos_data + neg_data
target = [1]*len(pos_data) + [0]*len(neg_data)
return(texts, target)

归一化文本函数

# Normalize text
def normalize_text(texts, stops):
# Lower case
texts = [x.lower() for x in texts] # Remove punctuation
texts = [''.join(c for c in x if c not in string.punctuation) for x in texts] # Remove numbers
texts = [''.join(c for c in x if c not in '') for x in texts] # Remove stopwords
texts = [' '.join([word for word in x.split() if word not in (stops)]) for x in texts] # Trim extra whitespace
texts = [' '.join(x.split()) for x in texts] return(texts)

生成词汇表函数

# Build dictionary of words构建词汇表(单词和单词数对),词频不够的单词(即标记为unknown的单词)标记为RARE
def build_dictionary(sentences, vocabulary_size):
# Turn sentences (list of strings) into lists of words
split_sentences = [s.split() for s in sentences]
words = [x for sublist in split_sentences for x in sublist] # Initialize list of [word, word_count] for each word, starting with unknown
count = [['RARE', -1]] # Now add most frequent words, limited to the N-most frequent (N=vocabulary size)
count.extend(collections.Counter(words).most_common(vocabulary_size-1)) # Now create the dictionary
word_dict = {}
# For each word, that we want in the dictionary, add it, then make it
# the value of the prior dictionary length
for word, word_count in count:
word_dict[word] = len(word_dict) return(word_dict)

生成单词索引表

# Turn text data into lists of integers from dictionary
def text_to_numbers(sentences, word_dict):
# Initialize the returned data
data = []
for sentence in sentences:
sentence_data = []
# For each word, either use selected index or rare word index
for word in sentence.split():
if word in word_dict:
word_ix = word_dict[word]
else:
word_ix = 0
sentence_data.append(word_ix)
data.append(sentence_data)
return(data)

生成批量数据函数

# Generate data randomly (N words behind, target, N words ahead)
def generate_batch_data(sentences, batch_size, window_size, method='skip_gram'):
# Fill up data batch
batch_data = []
label_data = []
while len(batch_data) < batch_size:
# select random sentence to start
rand_sentence_ix = int(np.random.choice(len(sentences), size=1))
rand_sentence = sentences[rand_sentence_ix]
# Generate consecutive windows to look at
window_sequences = [rand_sentence[max((ix-window_size),0):(ix+window_size+1)] for ix, x in enumerate(rand_sentence)]
# Denote which element of each window is the center word of interest
label_indices = [ix if ix<window_size else window_size for ix,x in enumerate(window_sequences)] # Pull out center word of interest for each window and create a tuple for each window
if method=='skip_gram':
batch_and_labels = [(x[y], x[:y] + x[(y+1):]) for x,y in zip(window_sequences, label_indices)]
# Make it in to a big list of tuples (target word, surrounding word)
tuple_data = [(x, y_) for x,y in batch_and_labels for y_ in y]
batch, labels = [list(x) for x in zip(*tuple_data)]
elif method=='cbow':
batch_and_labels = [(x[:y] + x[(y+1):], x[y]) for x,y in zip(window_sequences, label_indices)]
# Only keep windows with consistent 2*window_size
batch_and_labels = [(x,y) for x,y in batch_and_labels if len(x)==2*window_size]
batch, labels = [list(x) for x in zip(*batch_and_labels)]
elif method=='doc2vec':
# For doc2vec we keep LHS window only to predict target word
batch_and_labels = [(rand_sentence[i:i+window_size], rand_sentence[i+window_size]) for i in range(0, len(rand_sentence)-window_size)]
batch, labels = [list(x) for x in zip(*batch_and_labels)]
# Add document index to batch!! Remember that we must extract the last index in batch for the doc-index
batch = [x + [rand_sentence_ix] for x in batch]
else:
raise ValueError('Method {} not implmented yet.'.format(method)) # extract batch and labels
batch_data.extend(batch[:batch_size])
label_data.extend(labels[:batch_size])
# Trim batch and label at the end
batch_data = batch_data[:batch_size]
label_data = label_data[:batch_size] # Convert to numpy array
batch_data = np.array(batch_data)
label_data = np.transpose(np.array([label_data])) return(batch_data, label_data)

tensorflow在文本处理中的使用——辅助函数的更多相关文章

  1. tensorflow在文本处理中的使用——Doc2Vec情感分析

    代码来源于:tensorflow机器学习实战指南(曾益强 译,2017年9月)——第七章:自然语言处理 代码地址:https://github.com/nfmcclure/tensorflow-coo ...

  2. tensorflow在文本处理中的使用——Word2Vec预测

    代码来源于:tensorflow机器学习实战指南(曾益强 译,2017年9月)——第七章:自然语言处理 代码地址:https://github.com/nfmcclure/tensorflow-coo ...

  3. tensorflow在文本处理中的使用——CBOW词嵌入模型

    代码来源于:tensorflow机器学习实战指南(曾益强 译,2017年9月)——第七章:自然语言处理 代码地址:https://github.com/nfmcclure/tensorflow-coo ...

  4. tensorflow在文本处理中的使用——skip-gram模型

    代码来源于:tensorflow机器学习实战指南(曾益强 译,2017年9月)——第七章:自然语言处理 代码地址:https://github.com/nfmcclure/tensorflow-coo ...

  5. tensorflow在文本处理中的使用——TF-IDF算法

    代码来源于:tensorflow机器学习实战指南(曾益强 译,2017年9月)——第七章:自然语言处理 代码地址:https://github.com/nfmcclure/tensorflow-coo ...

  6. tensorflow在文本处理中的使用——词袋

    代码来源于:tensorflow机器学习实战指南(曾益强 译,2017年9月)——第七章:自然语言处理 代码地址:https://github.com/nfmcclure/tensorflow-coo ...

  7. tensorflow在文本处理中的使用——skip-gram & CBOW原理总结

    摘自:http://www.cnblogs.com/pinard/p/7160330.html 先看下列三篇,再理解此篇会更容易些(个人意见) skip-gram,CBOW,Word2Vec 词向量基 ...

  8. TensorFlow实现文本情感分析详解

    http://c.biancheng.net/view/1938.html 前面我们介绍了如何将卷积网络应用于图像.本节将把相似的想法应用于文本. 文本和图像有什么共同之处?乍一看很少.但是,如果将句 ...

  9. jQuery文本框中的事件应用

    jQuery文本框中的事件应用 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "ht ...

随机推荐

  1. Spark day05

    Shark Shark是基于Spark计算框架之上且兼容Hive语法的SQL执行引擎,由于底层的计算采用了Spark,性能比MapReduce的Hive普遍快2倍以上,当数据全部load在内存的话,将 ...

  2. 2018-11-3-WPF-内部的5个窗口之-MediaContextNotificationWindow

    title author date CreateTime categories WPF 内部的5个窗口之 MediaContextNotificationWindow lindexi 2018-11- ...

  3. Person Re-identification 系列论文笔记(四):Re-ID done right: towards good practices for person re-identification

    Re-ID done right: towards good practices for person re-identification Almazan J, Gajic B, Murray N, ...

  4. 从Kubernetes 1.14 发布,看技术社区演进方向

    Kubernetes 1.14 正式发布已经过去了一段时间,相信你已经从不同渠道看过了各种版本的解读. 不过,相比于代码 Release,马上就要迎来5周岁生日的Kubernetes 项目接下来如何演 ...

  5. asp 与php 的不同之处

    1.asp语句完成后不用加逗号:php要: 2.asp连接字符窜是用&:php用.; 3.asp需要用dim声明变量才能使用:php不用: 4.asp数组用(),php用[]; 5.asp的条 ...

  6. 转载 LibGDX: 使用 Gradle 命令运行和打包项目

    版权声明:本文为csdn xietansheng 博主原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接和本声明. 本文链接: http://blog.csdn.net/xieta ...

  7. DateFormat 日期格式化类(必须掌握)

    package com.sxt.utils.date1; import java.util.Date; import java.text.DateFormat; import java.text.Si ...

  8. UVa-10986_Sending email (向前星+Dijkstra)

    题意:给你点.边,求起点到终点的最短距离. 题解:由于题目的数据量特别大,所以需要用邻接表来存边,之后对Dijkstra算法稍微魔改一下就可以了,本来以为会超时,做好了打堆优化的准备,结果卡时间过了, ...

  9. android service中stub作用是什么?

    AIDL(android 接口描述语言)是一个IDL语言,它可以生成一段代码,可以使在一个android设备上运行的两个进程使用内部通信进程进行交互.如果你需要在一个进程中(例如:在一个Activit ...

  10. url地址栏参数<==>对象(将对象转换成地址栏的参数以及将地址栏的参数转换为对象)的实用函数

    /** * @author web得胜 * @param {Object} obj 需要拼接的参数对象 * @return {String} * */ function obj2qs(obj) { i ...