nltk_29_pickle保存和导入分类器
python机器学习-乳腺癌细胞挖掘(博主亲自录制视频)https://study.163.com/course/introduction.htm?courseId=1005269003&utm_campaign=commission&utm_source=cp-400000000398149&utm_medium=share
pickle用于持久化保存数据,分类器生成需要消耗大量时间,避免以后重复消耗时间
import pickle #保存分类器 save_classifier = open("naivebayes.pickle","wb")
pickle.dump(classifier, save_classifier)
save_classifier.close() #导入分类器
classifier_f = open("naivebayes.pickle", "rb")
classifier = pickle.load(classifier_f)
classifier_f.close()
Training classifiers and machine learning algorithms can take a very long time, especially if you're training against a larger data set. Ours is actually pretty small. Can you imagine having to train the classifier every time you wanted to fire it up and use it? What horror! Instead, what we can do is use the Pickle module to go ahead and serialize our classifier object, so that all we need to do is load that file in real quick.
So, how do we do this? The first step is to save the object. To do this, first you need to import pickle at the top of your script, then, after you have trained with .train() the classifier, you can then call the following lines:
This opens up a pickle file, preparing to write in bytes some data. Then, we use pickle.dump() to dump the data. The first parameter to pickle.dump() is what are you dumping, the second parameter is where are you dumping it.
After that, we close the file as we're supposed to, and that is that, we now have a pickled, or serialized, object saved in our script's directory!
Next, how would we go about opening and using this classifier? The .pickle file is a serialized object, all we need to do now is read it into memory, which will be about as quick as reading any other ordinary file. To do this:
Here, we do a very similar process. We open the file to read as bytes. Then, we use pickle.load() to load the file, and we save the data to the classifier variable. Then we close the file, and that is that. We now have the same classifier object as before!
Now, we can use this object, and we no longer need to train our classifier every time we wanted to use it to classify.
While this is all fine and dandy, we're probably not too content with the 60-75% accuracy we're getting. What about other classifiers? Turns out, there are many classifiers, but we need the scikit-learn (sklearn) module. Luckily for us, the people at NLTK recognized the value of incorporating the sklearn module into NLTK, and they have built us a little API to do it. That's what we'll be doing in the next tutorial.
把documents,word_features,classifier三个数据保存,以免以后做大量重复时间消耗
# -*- coding: utf-8 -*-
"""
Created on Thu Jan 12 10:44:19 2017 @author: Administrator 用于短评论分析-- Twitter 保存后的"positive.txt","negative.txt"需要转码为utf-8
在线转码网址
http://www.esk365.com/tools/GB2312-UTF8.asp features=5000,准确率百分之60以上
features=10000,准确率百分之 以上 运行时间可能长达一个小时
""" import nltk
import random
import pickle
from nltk.tokenize import word_tokenize
import time time1=time.time()
short_pos = open("positive.txt","r").read()
short_neg = open("negative.txt","r").read() # move this up here
documents = []
all_words = [] for r in short_pos.split('\n'):
documents.append( (r, "pos") ) for r in short_neg.split('\n'):
documents.append( (r, "neg") ) # j is adject, r is adverb, and v is verb
#allowed_word_types = ["J","R","V"] 允许形容词类别
allowed_word_types = ["J"] for p in short_pos.split('\n'):
documents.append( (p, "pos") )
words = word_tokenize(p)
pos = nltk.pos_tag(words)
for w in pos:
if w[1][0] in allowed_word_types:
all_words.append(w[0].lower()) for p in short_neg.split('\n'):
documents.append( (p, "neg") )
words = word_tokenize(p)
pos = nltk.pos_tag(words)
for w in pos:
if w[1][0] in allowed_word_types:
all_words.append(w[0].lower()) #保存文档
save_documents = open("documents.pickle","wb")
pickle.dump(documents, save_documents)
save_documents.close() #时间测试
time2=time.time()
print("time1 consuming:",time2-time1)
#保存特征
all_words = nltk.FreqDist(all_words)
#最好改成2万以上
word_features = list(all_words.keys())[:5000]
save_word_features = open("word_features5k.pickle","wb")
pickle.dump(word_features, save_word_features)
save_word_features.close() def find_features(document):
words = word_tokenize(document)
features = {}
for w in word_features:
features[w] = (w in words) return features featuresets = [(find_features(rev), category) for (rev, category) in documents] random.shuffle(featuresets)
print(len(featuresets)) testing_set = featuresets[10000:]
training_set = featuresets[:10000] classifier = nltk.NaiveBayesClassifier.train(training_set)
print("Original Naive Bayes Algo accuracy percent:", (nltk.classify.accuracy(classifier, testing_set))*100)
classifier.show_most_informative_features(15) #保存分类器
save_classifier = open("originalnaivebayes5k.pickle","wb")
pickle.dump(classifier, save_classifier)
save_classifier.close() time3=time.time()
print("time2 consuming:",time3-time2)
https://study.163.com/provider/400000000398149/index.htm?share=2&shareId=400000000398149(博主视频教学主页)
nltk_29_pickle保存和导入分类器的更多相关文章
- matlab学习(3) 保存和导入工作区
1.保存和导入工作区变量mat文件 假如创建了两个矩阵A=[1,2;3,4],B=[0,1;1,0] 则工作区就是这样的: 当函数有一个数据量非常大的返回值时,每次调用函数都要执行一遍函数,每次都要等 ...
- VS做简历的第三天(将文件中的样式保存并且导入)
VS做简历的第三天(将文件中的样式保存并且导入) 1.先在文件栏新建一个CSS文件 如 2.将第二天如下代码,删除<stype></stype>保留中间部分,复制在CSS文件并 ...
- docker 镜像的保存以及导入
docker 镜像的保存 docker save -o davename.tar images docker 镜像的导入 docker import - importname < tar ...
- TensorFlow 模型保存和导入、加载
在TensorFlow中,保存模型与加载模型所用到的是tf.train.Saver()这个类.我们一般的想法就是,保存模型之后,在另外的文件中重新将模型导入,我可以利用模型中的operation和va ...
- (qsf文件 、 tcl文件 和 csv(txt)文件的区别) FPGA管脚分配文件保存、导入导出方法
FPGA管脚分配文件保存方法 使用别人的工程时,有时找不到他的管脚文件,但可以把他已经绑定好的管脚保存下来,输出到文件里. 方法一: 查看引脚绑定情况,quartus -> assignment ...
- scikit_learn,NLTK导入分类器相关流程命令
- SecureCRT配置文件保存和导入
每次重装系统,都要重新配置SecureCRT,为了减少重复工作.直接在SecureCRT软件中找到:选项---全局选项---常规---配置文件夹下面路径:C:\Users\Administrator\ ...
- auto-keras 测试保存导入模型
# coding:utf-8 import time import matplotlib.pyplot as plt from autokeras import ImageClassifier# 保存 ...
- 多个Excel文件快速导入到DB里面
1 . 文件比较多,需要把这么多的数据都导入到DB里面,一个个导入太慢了,能想到的是先把数据整个到一个Excel中,然后再导入 2. 第一步准备合并Excel,新建一个新的excel,命名为total ...
随机推荐
- 个人作业-Week 1
1)快速看完整部教材,列出你仍然不懂的5到10个问题,发布在你的个人博客上. Q1:"Scrum Master不是一个官,而是一个没有行政权力的沟通者,就像微软的PM那样.他/她同时还要在团 ...
- 实验三— —敏捷开发与XP实践
---恢复内容开始--- java的第二个实验——JAVA面向对象程序设计 北京电子科技学院 实 验 报 告 课程:Java程序设计 班级:1352 姓名:林涵锦 学号:2013 ...
- 第五周作业总结(内含用Junit测试ArrayStack和LinkedStack课堂练习报告)
---恢复内容开始--- 学号 20162310<程序设计与数据结构>第五周学习总结 教材学习内容总结 集合分为线性集合(集合中的元素排成一行)和非线性集合(按不同于一行的方式来组织元素, ...
- YQCB绩效表
标准 队员 工作质量 20% 工作态度 20% 工作量 30% 工作难易程度 20% 团队意识 10% 总分 陈美琪 17 18 28 19 9 91 张晨阳 16 16 25 17 9 83 刘昭为 ...
- 进阶系列(10)—— C#元数据和动态编程
一.元数据的介绍 元数据是用来描述数据的数据(Data that describes other data).单单这样说,不太好理解,我来举个例子.下面是契诃夫的小说<套中人>中的一段,描 ...
- PMS---团队展示
点我查看作业原题 [队名] PMS(一群pm) [拟做的团队项目描述] 基于监控场景的视频摘要与人车检测跟踪系统 A system, under monitor scene, for video su ...
- Mongodb 分片操作 介绍
为什么需要分片操作?由于数据量太大,使得CPU,内存,磁盘I/O等压力过大.当MongoDB存储海量的数据时,一台机器可能不足以存储数据,也可能不足以提供可接受的读写吞吐量.这时,我们就可以通过在多台 ...
- 微信公众平台实现获取用户OpenID的方法
这篇文章主要介绍了微信公众平台实现获取用户OpenID的方法,需要开发人员经过微信授权后获取高级接口才能使用此功能,用户OpenID对于微信公众平台建设有着非常广泛的用途,需要的朋友可以参考下 本文实 ...
- NULL,"",String.Empty三者在C#中的区别
(1)NULLnull 关键字是表示不引用任何对象的空引用的文字值.null 是引用类型变量的默认值.那么也只有引用型的变量可以为NULL,如果int i=null,的话,是不可以的,因为Int是值类 ...
- 反射就是获取该类的.class文件里面的方法,属性
反射就是获取该类的.class文件里面的方法,属性