我们运行看结果

安装包sklearn

安装numpy

安装scipy

终于可以啦

我们把安装的包都写在文件里面吧

4行4列 轴对称 只需要看一半就可以 横着看 竖着看都行 数值越接近1 表示越相似

我们通过这个可以将新的新闻(还未加入数据库的新闻)放在左上角 然后mongodb存的老新闻和他比较 如果超一定值

比如0.8 表示相似度高 我们就帮他当成一个新闻 那么这个左上角新的新闻

就会被踢掉  如果相似度很低 说明是新的新闻 那么就 执行命令加入mongodb中来  大概这个意思

0.12693309表示的是0号文件和1号文件的相似度

我们来试验一个比较长的

里面有特殊字符 不加注释出爆NO-ASXIAL码的问题

# -*- coding: utf- -*-

from sklearn.feature_extraction.text import TfidfVectorizer

news1 = """
(CNN)President Donald Trump on Saturday again attacked a federal judge whose decision he disliked, blasting Judge James Robart, a George W. Bush appointee who temporarily stopped his controversial travel ban Friday night.
Trump's increasingly heated responses quickly drew objections from Democrats, who said he was improperly attacking an independent judiciary. By Saturday afternoon, Trump had stepped up his criticism: "Because the ban was lifted by a judge, many very bad and dangerous people may be pouring into our country. A terrible decision."
Shortly after a.m. ET, the President tweeted, "The opinion of this so-called judge, which essentially takes law-enforcement away from our country, is ridiculous and will be overturned."
The opinion of this so-called judge, which essentially takes law-enforcement away from our country, is ridiculous and will be overturned!
— Donald J. Trump (@realDonaldTrump) February ,
That tweet was one of several Trump issued Saturday morning in which he defended his executive order on immigration, which bars citizens of seven Muslim-majority countries from entering the US for days, all refugees for days and indefinitely halts refugees from Syria.
RELATED: James Robart: things to know about judge who blocked travel ban
"When a country is no longer able to say who can, and who cannot , come in & out, especially for reasons of safety &.security - big trouble," Trump next tweeted.
When a country is no longer able to say who can, and who cannot , come in & out, especially for reasons of safety &.security - big trouble!
— Donald J. Trump (@realDonaldTrump) February ,
"Interesting that certain Middle-Eastern countries agree with the ban. They know if certain people are allowed in it's death & destruction," he added, though he didn't name any countries.
Interesting that certain Middle-Eastern countries agree with the ban. They know if certain people are allowed in it's death & destruction!
— Donald J. Trump (@realDonaldTrump) February ,
Saturday afternoon, Trump resumed his criticism, tweeting: "What is our country coming to when a judge can halt a Homeland Security travel ban and anyone, even with bad intentions, can come into U.S.?"
What is our country coming to when a judge can halt a Homeland Security travel ban and anyone, even with bad intentions, can come into U.S.?
— Donald J. Trump (@realDonaldTrump) February ,
He followed up with, "Because the ban was lifted by a judge, many very bad and dangerous people may be pouring into our country. A terrible decision."
Because the ban was lifted by a judge, many very bad and dangerous people may be pouring into our country. A terrible decision
— Donald J. Trump (@realDonaldTrump) February ,
And he was still tweeting about it early Saturday evening: "Why aren't the lawyers looking at and using the Federal Court decision in Boston, which is at conflict with ridiculous lift ban decision?"
Why aren't the lawyers looking at and using the Federal Court decision in Boston, which is at conflict with ridiculous lift ban decision?
— Donald J. Trump (@realDonaldTrump) February ,
Trump was referring to a decision by a federal judge in Boston earlier Friday, a more limited ruling that declined to renew a temporary restraining order in Massachusetts. It would have prohibited the detention or removal of foreign travelers legally authorized to come to the Boston area, and the decision represented the Trump administration's first court victory regarding the order.
Unusual criticism
It is highly unusual for a President to publicly criticize a federal judge, but during the campaign, Trump memorably railed against Judge Gonzalo Curiel, who was overseeing a lawsuit against Trump University. Trump said Curiel, who was born in Indiana, was unable to fairly preside over the lawsuit because of his "Mexican heritage." Trump had introduced plans to build a wall along the Mexican border and take a hard stance on immigration.
Vice President Mike Pence later defended Trump in an interview with ABC News' George Stephanopoulos.
"Is it right for the President to say 'so-called' judge'? Doesn't that undermine the separation of powers in the Constitution?" Stephanopoulos asked Pence on "This Week" in a clip released Saturday afternoon.
"I don't think it does," Pence replied. "I think the American people are very accustomed to this president speaking his mind and speaking very straight with them."
ABC Breaking News | Latest News Videos
But Democrats pounced on Trump's criticism of Robart, with Democratic senators flatly saying the President's comments will factor into the confirmation hearings for Supreme Court nominee Neil Gorsuch.
"Attack on federal judge from POTUS is beneath the dignity of that office. That attitude can lead America to calamity," Washington Gov. Jay Inslee tweeted Saturday.
Attack on federal judge from POTUS is beneath the dignity of that office. That attitude can lead America to calamity.
— Governor Jay Inslee (@GovInslee) February ,
"The President's attack on Judge James Robart, a Bush appointee who passed with 99 votes, shows a disdain for an independent judiciary that doesn't always bend to his wishes and a continued lack of respect for the Constitution, making it more important that the Supreme Court serve as an independent check on the administration," Senate Minority Leader Chuck Schumer said in a statement.
"With each action testing the Constitution, and each personal attack on a judge, President Trump raises the bar even higher for Judge Gorsuch's nomination to serve on the Supreme Court. His ability to be an independent check will be front and center throughout the confirmation process."
Vermont. Sen. Patrick Leahy, the ranking member of the Judiciary Committee, said Trump's "hostility toward the rule of law is not just embarrassing, it is dangerous."
"We need a nominee for the Supreme Court willing to demonstrate he or she will not cower to an overreaching executive. This makes it even more important that Judge Gorsuch, and every other judge this president may nominate, demonstrates the ability to be an independent check and balance on an administration that shamefully and harmfully seems to reject the very concept."
Robart's order on Friday was a significant setback to Trump's ban and set up the nation for a second straight weekend of confusion about the policy's legality.
The White House said Friday the Department of Justice will challenge the decision. In a statement, White House press secretary Sean Spicer initially called Robart's order "outrageous" before quickly issuing another statement that dropped that word.
Robart has presided in the US District Court for the Western District of Washington state since . He assumed senior status in .
"""
news2 = """
President Donald Trump on Saturday again attacked a federal judge whose decision he disliked, criticizing Judge James Robart, a George W. Bush appointee who temporarily stopped his controversial travel ban Friday night.
President Trump’s attacks quickly drew objections from Democrats, who said he was attacking an independent judiciary. And by Saturday afternoon, President Trump was openly accusing Robart of potentially allowing “many very bad and dangerous people” to flow into the US and warning of dire consequences if the executive order is not enforced.
He also said, “What is our country coming to when a judge can halt a Homeland Security ban and anyone, even with bad intentions, can come into the U.S.?”
What is our country coming to when a judge can halt a Homeland Security travel ban and anyone, even with bad intentions, can come into U.S.?
— Donald J. Trump (@realDonaldTrump) February ,
Shortly after a.m. ET, the President tweeted, “The opinion of this so-called judge, which essentially takes law-enforcement away from our country, is ridiculous and will be overturned.”
The opinion of this so-called judge, which essentially takes law-enforcement away from our country, is ridiculous and will be overturned!
— Donald J. Trump (@realDonaldTrump) February ,
The tweet was one of several President Trump issued Saturday morning in which he defended his executive order on immigration, which bars citizens of seven Muslim-majority countries from entering the US for days, all refugees for days and indefinitely halts refugees from Syria.
“When a country is no longer able to say who can, and who cannot , come in & out, especially for reasons of safety &.security – big trouble,” President Trump next tweeted.
When a country is no longer able to say who can, and who cannot , come in & out, especially for reasons of safety &.security – big trouble!
— Donald J. Trump (@realDonaldTrump) February ,
“Interesting that certain Middle-Eastern countries agree with the ban. They know if certain people are allowed in it’s death & destruction,” he added, though he didn’t name any countries.
Interesting that certain Middle-Eastern countries agree with the ban. They know if certain people are allowed in it's death & destruction!
— Donald J. Trump (@realDonaldTrump) February ,
Saturday afternoon, President Trump resumed his criticism, tweeting: “What is our country coming to when a judge can halt a Homeland Security travel ban and anyone, even with bad intentions, can come into U.S.?”
He followed up with, “Because the ban was lifted by a judge, many very bad and dangerous people may be pouring into our country. A terrible decision.”
Because the ban was lifted by a judge, many very bad and dangerous people may be pouring into our country. A terrible decision
— Donald J. Trump (@realDonaldTrump) February ,
It is highly unusual for a President to publicly criticize a federal judge, but during the campaign, President Trump memorably railed against Judge Gonzalo Curiel, who was overseeing a lawsuit against Trump University. President Trump said Curiel, who was born in Indiana, was unable to fairly preside over the lawsuit because of his “Mexican heritage.” President Trump had introduced plans to build a wall along the Mexican border and take a hard stance on immigration.
Democrats pounced on President Trump’s criticism of Robart, with Democratic senators flatly saying the President’s comments will factor into the confirmation hearings for Supreme Court nominee Neil Gorsuch.
“Attack on federal judge from POTUS is beneath the dignity of that office. That attitude can lead America to calamity,” Washington Gov. Jay Inslee tweeted Saturday.
Attack on federal judge from POTUS is beneath the dignity of that office. That attitude can lead America to calamity.
— Governor Jay Inslee (@GovInslee) February ,
“The President’s attack on Judge James Robart, a Bush appointee who passed with votes, shows a disdain for an independent judiciary that doesn’t always bend to his wishes and a continued lack of respect for the Constitution, making it more important that the Supreme Court serve as an independent check on the administration,” Senate Minority Leader Chuck Schumer said in a statement.
“With each action testing the Constitution, and each personal attack on a judge, President Trump raises the bar even higher for Judge Gorsuch’s nomination to serve on the Supreme Court. His ability to be an independent check will be front and center throughout the confirmation process.”
Vermont. Sen. Patrick Leahy, the ranking member of the Judiciary Committee, said President Trump’s “hostility toward the rule of law is not just embarrassing, it is dangerous.”
“We need a nominee for the Supreme Court willing to demonstrate he or she will not cower to an overreaching executive. This makes it even more important that Judge Gorsuch, and every other judge this president may nominate, demonstrates the ability to be an independent check and balance on an administration that shamefully and harmfully seems to reject the very concept.”
Robart’s order on Friday was a significant setback to President Trump’s ban and set up the nation for a second straight weekend of confusion about the policy’s legality.
The White House said Friday the Department of Justice will challenge the decision. In a statement, White House press secretary Sean Spicer initially called Robart’s order “outrageous” before quickly issuing another statement that dropped that word.
Robart has presided in the US District Court for the Western District of Washington state since . He assumed senior status in .
""" documents = [news1, news2] tfidf = TfidfVectorizer().fit_transform(documents)
pairwise_sim = tfidf * tfidf.T print pairwise_sim.A

test02

上面是2个新闻 都是Trump的新闻 第一个是cnn的 第二个是其他网站的

我们发现相似度相当高 0.96

那么我就可以认为他俩是一个新闻

下面我就用这个算法来查重

week07 13.3 NewsPipeline之 三News Deduper之 tf_idf 查重的更多相关文章

  1. week07 13.4 NewsPipeline之 三 News Deduper

    还是循环将Q2中的东西拿出来 然后查重(去mongodb里面把一天之内的新闻都拿出来,然后把拿到的新的新闻和mongodb里一天内的新闻组一个 tf-idf的对比)可看13.3 相似度检查 如果超过一 ...

  2. week07 13.1 NewsPipeline之 一 NewsMonitor

    我们要重构一下代码 因为我们之前写了utils 我们的NewsPipeline部分也要用到 所以我们把他们单独独立得拿出来 删掉原来的 将requirements.txt也拿出去 现在我们搬家完成 我 ...

  3. week07 13.2 NewsPipeline之 二 News Fetcher - Xpath

    我们使用Xpath来专门做一个scrapter 我们专门弄个文件夹 里面全部是 各个新闻源(CNN BBC等)的scraper来抓取网站的text内容 主要函数(就是传入text内容的那个url)然后 ...

  4. C# 基础知识系列-13 常见类库(三)

    0. 前言 在<C# 基础知识系列- 13 常见类库(二)>中,我们介绍了一下DateTime和TimeSpan这两个结构体的内容,也就是C#中日期时间的简单操作.本篇将介绍Guid和Nu ...

  5. Python for Infomatics 第13章 网页服务三(译)

    注:文章原文为Dr. Charles Severance 的 <Python for Informatics>.文中代码用3.4版改写,并在本机测试通过. 13.6 应用程序接口API 现 ...

  6. 13 (H5*) JS第三天 数组、函数

    目录 1:数组的定义和创建方式 2:数组的总结 3:for循环遍历数组 4:数组的案例 5:冒泡排序 6:函数的定义 7:函数的参数 8:函数的返回值 复习 <script> /* * * ...

  7. 外包公司派遣到网易,上班地点网易大厦,转正后工资8k-10k,13薪,包三餐,值得去吗?

    作为一个人,我们必须时时刻刻清醒地看待自己,做到不卑不亢才能坚强地活下去. 请肆无忌惮地点赞吧,微信搜索[沉默王二]关注这个在九朝古都洛阳苟且偷生的程序员.本文 GitHub github.com/i ...

  8. 2018/2/13 ElasticSearch学习笔记三 自动映射以及创建自动映射模版,ElasticSearch聚合查询

    终于把这些命令全敲了一遍,话说ELK技术栈L和K我今天花了一下午全部搞定,学完后还都是花式玩那种...E却学了四天(当然主要是因为之前上班一直没时间学,还有安装服务时出现的各种error真是让我扎心了 ...

  9. 深入刨析tomcat 之---第13篇 tomcat的三种部署方法

    writedby 张艳涛 一般我们都知道将web 应用打成war包,放到tomcat的webapp目录下,就是部署了,这是部署方法1 第2种部署方法我们也知道,就是讲web应用的文件夹拷贝到webap ...

随机推荐

  1. 几个特殊的IP地址

    1)私有地址     IP地址在全世界范围内唯一,看到这句话你可能有这样的疑问,像192.168.0.1这样的地址在许多地方都能看到,并不唯一,这是为何?Internet管理委员会规定如下地址段为私有 ...

  2. 软件工程 week 02

    一.地址链接 1.作业地址:https://edu.cnblogs.com/campus/nenu/2016CS/homework/2110 2.git仓库地址:https://git.coding. ...

  3. day04元组

    元组 1.元祖书写规范 users=[11,22,33,"老男孩"]#列表(可变) users=(11,22,33,"老纳海")#元祖(不可变) 1.公共方法( ...

  4. spring boot 1.4 整合 mybatis druid

    http://www.jianshu.com/p/cef49ad91ba9spring boot 1.4 整合 mybatis druid

  5. 1.golang的环境搭建及入门

    安装包下载 下载链接:https://dl.google.com/go/go1.12.3.windows-amd64.msi 环境搭建 安装完成之后,找一个来存放go语言文件的文件夹,我这里选的是G: ...

  6. JavaScript数据类型(第一天)

    ECMAScript为JavaScript的标准,javascript为网景公司定义,但并不标准,所以欧洲的组织定义了ESMAScript,定义了网页脚本的标准. js组成 ECMAScript js ...

  7. C#编码、解码与ASP.NET编码解码对应函数

    JavaScript函数分别为:encodeURIComponent/decodeURIComponent C#对应的函数分别为:Uri.EscapeUriString/Uri.EscapeDataS ...

  8. int和integer的区别和使用

    基本数据类型和引用数据类型的区别和介绍:https://www.cnblogs.com/bekeyuan123/p/7468845.html 1.int是基本数据类型,integer是引用数据类型,是 ...

  9. MySQL 和 Oracle 在 MyBatis 使用中的区别

    MySQL 和 Oracle 在 MyBatis 使用中的区别: 区别 MySQL Oracle 存储过程的参数模式 mode 为 IN 时,是否需要指定 jdbcType 不需要:MyBatis 为 ...

  10. Centos6.8通过yum安装mysql5.7 centos7.5适用

    1.安装mysql的yum源 a.下载配置mysql的yum源的rpm包 根据上面3张图片中的操作下载下来的rpm文件可以通过如下命令获取: wget https://dev.mysql.com/ge ...