从数据源拉取数据，将数据内容与一组搜索项做比对 go func() chanel
https://github.com/goinaction/code
[root@hadoop3 sample]# go run main.go
2018/07/30 17:45:39 Register default matcher
2018/07/30 17:45:39 Register rss matcher
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1001]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=46]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=7]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=10]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=39]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=43]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/business/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1008]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1006]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_topstories.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1007]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_world.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1057]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_us.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1021]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_allpolitics.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1012]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_crime.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1003]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=2]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_tech.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_health.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_topstories.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/opinion?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/politics?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/national?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/world?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://feeds.nbcnews.com/feeds/topstories]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://feeds.nbcnews.com/feeds/usnews]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/21491043/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=3]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/21491571/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=5]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/28180066/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=13]
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_allpolitics.rss: read tcp 192.168.3.103:54478->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_topstories.rss: read tcp 192.168.3.103:54484->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_health.rss: read tcp 192.168.3.103:54482->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_crime.rss: read tcp 192.168.3.103:54476->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_us.rss: read tcp 192.168.3.103:54488->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_world.rss: read tcp 192.168.3.103:54486->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Description:
If Ronald Reagan was the “Teflon president,” Donald Trump must be the Incredible Hulk president. The more the left and the media attack him, the stronger he gets.<img src="http://feeds.feedburner.com/~r/foxnews/opinion/~4/bVrx9JLgLWI" height="1" width="1" alt=""/> 2018/07/30 17:45:39 Description:
We look at the latest strong economic numbers and the effect tariffs may have on the president's base. 2018/07/30 17:45:39 Description:
Freeman Hrabowski, president of the University of Maryland, Baltimore County, talks about the importance of mentoring in academia, especially in science, technology, engineering and math fields. 2018/07/30 17:45:39 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
Retail and fast-food workers protesting for higher pay are creating a new kind of U.S. labor movement. <em>Chicago Tribune</em> columnist Clarence Page argues that the president could "set a good example" by requiring fast-food vendors who have contracts with the federal government to pay minimum wage. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
The moment when the president articulated his attitude with pithy perfection came when he told the Veterans of Foreign Wars that "what you're seeing and what you're reading is not what's happening." 2018/07/30 17:45:40 Description:
The U.S. president publicly humiliated our most important allies and then said everything was great. We need positive, unifying leadership, writes a former U.S. Army Europe commander. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
Last year a song about freedom became a rallying cry as Zimbabwe unseated longtime president Robert Mugabe. Now the anthem has become tainted by the lack of tangible change for people there. 2018/07/30 17:45:40 Description:
Last year a song about freedom became a rallying cry as Zimbabwe unseated longtime president Robert Mugabe. Now the anthem has become tainted by the lack of tangible change for people there. 2018/07/30 17:45:40 Description:
We look at the latest strong economic numbers and the effect tariffs may have on the president's base. 2018/07/30 17:45:40 Description:
The president goes back and forth about what he accepts and denies about Russian election interference. Even so, his recent warning about an attack that helps Democrats is a cogent one. 2018/07/30 17:45:40 Description:
The president demanded cooperation from the Democrats over immigration, threatening to close the government in order to achieve his goals. 2018/07/30 17:45:40 Get http://rss.cnn.com/rss/cnn_tech.rss: read tcp 192.168.3.103:54480->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:40 Get http://rss.cnn.com/rss/cnn_topstories.rss: read tcp 192.168.3.103:54474->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/business/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
[root@hadoop3 sample]# tree
.
├── data
│ └── data.json
├── main.go
├── matchers
│ └── rss.go
└── search
├── default.go
├── feed.go
├── match.go
└── search.go 3 directories, 7 files
[root@hadoop3 sample]# tree /home/ctGO/goInAction/
/home/ctGO/goInAction/
└── code
├── chapter1
│ └── channels
│ └── hellochannels.go
├── chapter10
│ ├── listing01
│ │ └── listing01.go
│ ├── listing02
│ │ └── listing02.go
│ ├── listing03
│ │ └── listing03.go
│ ├── listing04
│ │ └── listing04.go
│ ├── listing05
│ │ └── listing05.go
│ └── listing06
│ ├── listing06
│ ├── listing06.go
│ └── pubsub
│ └── pubsub.go
├── chapter2
│ └── sample
│ ├── data
│ │ └── data.json
│ ├── main.go
│ ├── matchers
│ │ └── rss.go
│ └── search
│ ├── default.go
│ ├── feed.go
│ ├── match.go
│ └── search.go
├── chapter3
│ ├── dbdriver
│ │ ├── main.go
│ │ └── postgres
│ │ └── postgres.go
│ ├── wordcount
│ │ ├── gowords.txt
│ │ └── wordcount.go
│ └── words
│ └── count.go
├── chapter5
│ ├── listing11
│ │ └── listing11.go
│ ├── listing34
│ │ └── listing34.go
│ ├── listing35
│ │ └── listing35.go
│ ├── listing36
│ │ └── listing36.go
│ ├── listing46
│ │ └── listing46.go
│ ├── listing48
│ │ └── listing48.go
│ ├── listing50
│ │ └── listing50.go
│ ├── listing56
│ │ └── listing56.go
│ ├── listing60
│ │ └── listing60.go
│ ├── listing64
│ │ ├── counters
│ │ │ └── counters.go
│ │ └── listing64.go
│ ├── listing68
│ │ ├── counters
│ │ │ └── counters.go
│ │ └── listing68.go
│ ├── listing71
│ │ ├── entities
│ │ │ └── entities.go
│ │ └── listing71.go
│ └── listing74
│ ├── entities
│ │ └── entities.go
│ └── listing74.go
├── chapter6
│ ├── listing01
│ │ └── listing01.go
│ ├── listing04
│ │ └── listing04.go
│ ├── listing07
│ │ └── listing07.go
│ ├── listing09
│ │ └── listing09.go
│ ├── listing13
│ │ └── listing13.go
│ ├── listing15
│ │ └── listing15.go
│ ├── listing16
│ │ └── listing16.go
│ ├── listing20
│ │ └── listing20.go
│ ├── listing22
│ │ └── listing22.go
│ └── listing24
│ └── listing24.go
├── chapter7
│ └── patterns
│ ├── pool
│ │ ├── main
│ │ │ └── main.go
│ │ └── pool.go
│ ├── runner
│ │ ├── main
│ │ │ └── main.go
│ │ └── runner.go
│ ├── search
│ │ ├── main
│ │ │ └── main.go
│ │ ├── searchers.go
│ │ └── search.go
│ ├── semaphore
│ │ └── semaphore.go
│ └── work
│ ├── main
│ │ └── main.go
│ └── work.go
├── chapter8
│ ├── listing03
│ │ └── listing03.go
│ ├── listing11
│ │ └── listing11.go
│ ├── listing24
│ │ └── listing24.go
│ ├── listing27
│ │ └── listing27.go
│ ├── listing29
│ │ └── listing29.go
│ ├── listing31
│ │ └── listing31.go
│ ├── listing37
│ │ └── listing37.go
│ └── listing46
│ └── listing46.go
├── chapter9
│ ├── listing01
│ │ └── listing01_test.go
│ ├── listing08
│ │ └── listing08_test.go
│ ├── listing12
│ │ └── listing12_test.go
│ ├── listing17
│ │ ├── handlers
│ │ │ ├── handlers_example_test.go
│ │ │ ├── handlers.go
│ │ │ └── handlers_test.go
│ │ └── listing17.go
│ └── listing28
│ └── listing28_test.go
└── README.md 77 directories, 75 files
[root@hadoop3 sample]#
https://github.com/goinaction/code/blob/master/chapter2/sample/search/search.go
// Launch a goroutine to monitor when all the work is done.
go func() {
// Wait for everything to be processed.
waitGroup.Wait() // Close the channel to signal to the Display
// function that we can exit the program.
close(results)
}()
// Run performs the search logic.
func Run(searchTerm string) {
// Retrieve the list of feeds to search through.
feeds, err := RetrieveFeeds()
if err != nil {
log.Fatal(err)
} // Create an unbuffered channel to receive match results to display.
results := make(chan *Result) // Setup a wait group so we can process all the feeds.
var waitGroup sync.WaitGroup // Set the number of goroutines we need to wait for while
// they process the individual feeds.
waitGroup.Add(len(feeds)) // Launch a goroutine for each feed to find the results.
for _, feed := range feeds {
// Retrieve a matcher for the search.
matcher, exists := matchers[feed.Type]
if !exists {
matcher = matchers["default"]
} // Launch the goroutine to perform the search.
go func(matcher Matcher, feed *Feed) {
Match(matcher, feed, searchTerm, results)
waitGroup.Done()
}(matcher, feed)
} // Launch a goroutine to monitor when all the work is done.
go func() {
// Wait for everything to be processed.
waitGroup.Wait() // Close the channel to signal to the Display
// function that we can exit the program.
close(results)
}() // Start displaying results as they are available and
// return after the final result is displayed.
Display(results)
}
go通道消息共享
从数据源拉取数据，将数据内容与一组搜索项做比对 go func() chanel的更多相关文章
- FLUME安装&环境（二）:拉取MySQL数据库数据到Kafka
Flume安装成功,环境变量配置成功后,开始进行agent配置文件设置. 1.agent配置文件(mysql+flume+Kafka) #利用Flume将MySQL表数据准实时抽取到Kafka a1. ...
- 使用Jenkins的Git Parameter插件来从远程仓库拉取指定目录的内容
更换插件安装源 系统管理---插件管理---高级---升级站点,把默认的插件下载地址换成下面这个: https://mirrors.tuna.tsinghua.edu.cn/jenkins/updat ...
- 解决 MySQL 比如我要拉取一个消息表中用户id为1的前10条最新数据
我们都知道,各种主流的社交应用或者阅读应用,基本都有列表类视图,并且都有滑到底部加载更多这一功能, 对应后端就是分页拉取数据.好处不言而喻,一般来说,这些数据项都是按时间倒序排列的,用户只关心最新的动 ...
- canal从mysql拉取数据，并以protobuf的格式往kafka中写数据
大致思路: canal去mysql拉取数据,放在canal所在的节点上,并且自身对外提供一个tcp服务,我们只要写一个连接该服务的客户端,去拉取数据并且指定往kafka写数据的格式就能达到以proto ...
- 版本控制git之三-多人协作 变基 推送 拉取 删除远程分支
版本控制git之三-多人协作 wangfeng7399已关注0人评论350人阅读2019-02-20 21:33:08 如果你想获得一份已经存在了的 Git 仓库的拷贝,比如说,你想为某个开源 ...
- 源码分析Kafka 消息拉取流程
目录 1.KafkaConsumer poll 详解 2.Fetcher 类详解 本节重点讨论 Kafka 的消息拉起流程. @(本节目录) 1.KafkaConsumer poll 详解 消息拉起主 ...
- Mysql日期类型大小比较---拉取给定时间段的记录
我们知道,mysql里边,日期类型有很多表现形式,date, datetime,timestamp等类型.考虑这样一种场景: 按时间段拉取给定时间段的内容,这时,我们就得使用日期类型的比较了. 表结构 ...
- idea配置git，查看git代码&拉取git项目至本地
1.点击file,右键选择setting 选择本地git安装路径 Ps:从git上导入一个全新的maven项目 点击clone按钮后,会弹出如下截图弹窗,点击 NO 项目已经拉取到本地,然后点击ope ...
- Spark Streaming中向flume拉取数据
在这里看到的解决方法 https://issues.apache.org/jira/browse/SPARK-1729 请是个人理解,有问题请大家留言. 其实本身flume是不支持像KAFKA一样的发 ...
随机推荐
- 九度oj 题目1352：和为S的两个数字
题目描述: 输入一个递增排序的数组和一个数字S,在数组中查找两个数,是的他们的和正好是S,如果有多对数字的和等于S,输出两个数的乘积最小的. 输入: 每个测试案例包括两行: 第一行包含一个整数n和k, ...
- iOS学习笔记08-Quartz2D绘图
一.Quartz2D简单介绍 在iOS中常用的绘图框架就是Quartz2D,Quartz2D是Core Graphics框架的一部分,我们日常开发使用的所有UIKit组件都是由Core Graphic ...
- [luoguP2221] [HAOI2012]高速公路（线段树）
传送门 考虑每一段对答案的贡献 用每一段的左端点来表示当前这一段,那么区间就变成了[1,n-1] 如果询问区间[l,r],其中一个点的位置为x,则它对答案的贡献为(x-l)*(r-x)*s[x](s[ ...
- 服务器内部转发forward，action到action
如果request.getRequestDispatcher();中不是页面而是传action的话,参考以下内容修改: web.xml 2.4版本里,默认的filter只拦截request. 如果使用 ...
- 征途（bzoj 4518）
Description Pine开始了从S地到T地的征途. 从S地到T地的路可以划分成n段,相邻两段路的分界点设有休息站. Pine计划用m天到达T地.除第m天外,每一天晚上Pine都必须在休息站过夜 ...
- 从dataset表中获取某一列的所有值方法
原文发布时间为:2008-07-31 -- 来源于本人的百度文章 [由搬家工具导入] 可以datarow遍历所有行即可，如下：pubauthor这个表中的au_lname的所有值加到listbox上面 ...
- hdu3491最小割转最大流+拆点
题意:求最小割,即求最大流即可.此题之关键为拆点(限制在点),每条边都是双向边,注意一下. 未1A原因:在拆点之后添加边的过程中,要注意,出去的是i`,进来的是i,!!所以,写addegde函数时候 ...
- (48)C#网络4 web
WebClient 类 提供用于将数据发送到和接收来自通过 URI 确认的资源数据的常用方法 private delegate string delegatehWeb(); private void ...
- Struts2 文件上传和下载
首先我们写一个单文件长传的fileupload.jsp: <body> <s:fielderror></s:fielderror> <!-- 报错信息 --& ...
- luogu P2296 寻找道路
题目描述 在有向图G 中,每条边的长度均为1 ,现给定起点和终点,请你在图中找一条从起点到终点的路径,该路径满足以下条件: 1 ．路径上的所有点的出边所指向的点都直接或间接与终点连通. 2 ．在满足条 ...