从数据源拉取数据，将数据内容与一组搜索项做比对 go func() chanel
https://github.com/goinaction/code
[root@hadoop3 sample]# go run main.go
2018/07/30 17:45:39 Register default matcher
2018/07/30 17:45:39 Register rss matcher
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1001]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=46]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=7]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=10]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=39]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=43]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/business/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1008]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1006]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_topstories.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1007]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_world.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1057]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_us.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1021]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_allpolitics.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1012]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_crime.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1003]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=2]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_tech.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_health.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_topstories.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/opinion?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/politics?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/national?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/world?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://feeds.nbcnews.com/feeds/topstories]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://feeds.nbcnews.com/feeds/usnews]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/21491043/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=3]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/21491571/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=5]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/28180066/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=13]
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_allpolitics.rss: read tcp 192.168.3.103:54478->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_topstories.rss: read tcp 192.168.3.103:54484->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_health.rss: read tcp 192.168.3.103:54482->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_crime.rss: read tcp 192.168.3.103:54476->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_us.rss: read tcp 192.168.3.103:54488->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_world.rss: read tcp 192.168.3.103:54486->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Description:
If Ronald Reagan was the “Teflon president,” Donald Trump must be the Incredible Hulk president. The more the left and the media attack him, the stronger he gets.<img src="http://feeds.feedburner.com/~r/foxnews/opinion/~4/bVrx9JLgLWI" height="1" width="1" alt=""/> 2018/07/30 17:45:39 Description:
We look at the latest strong economic numbers and the effect tariffs may have on the president's base. 2018/07/30 17:45:39 Description:
Freeman Hrabowski, president of the University of Maryland, Baltimore County, talks about the importance of mentoring in academia, especially in science, technology, engineering and math fields. 2018/07/30 17:45:39 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
Retail and fast-food workers protesting for higher pay are creating a new kind of U.S. labor movement. <em>Chicago Tribune</em> columnist Clarence Page argues that the president could "set a good example" by requiring fast-food vendors who have contracts with the federal government to pay minimum wage. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
The moment when the president articulated his attitude with pithy perfection came when he told the Veterans of Foreign Wars that "what you're seeing and what you're reading is not what's happening." 2018/07/30 17:45:40 Description:
The U.S. president publicly humiliated our most important allies and then said everything was great. We need positive, unifying leadership, writes a former U.S. Army Europe commander. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
Last year a song about freedom became a rallying cry as Zimbabwe unseated longtime president Robert Mugabe. Now the anthem has become tainted by the lack of tangible change for people there. 2018/07/30 17:45:40 Description:
Last year a song about freedom became a rallying cry as Zimbabwe unseated longtime president Robert Mugabe. Now the anthem has become tainted by the lack of tangible change for people there. 2018/07/30 17:45:40 Description:
We look at the latest strong economic numbers and the effect tariffs may have on the president's base. 2018/07/30 17:45:40 Description:
The president goes back and forth about what he accepts and denies about Russian election interference. Even so, his recent warning about an attack that helps Democrats is a cogent one. 2018/07/30 17:45:40 Description:
The president demanded cooperation from the Democrats over immigration, threatening to close the government in order to achieve his goals. 2018/07/30 17:45:40 Get http://rss.cnn.com/rss/cnn_tech.rss: read tcp 192.168.3.103:54480->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:40 Get http://rss.cnn.com/rss/cnn_topstories.rss: read tcp 192.168.3.103:54474->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/business/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
[root@hadoop3 sample]# tree
.
├── data
│ └── data.json
├── main.go
├── matchers
│ └── rss.go
└── search
├── default.go
├── feed.go
├── match.go
└── search.go 3 directories, 7 files
[root@hadoop3 sample]# tree /home/ctGO/goInAction/
/home/ctGO/goInAction/
└── code
├── chapter1
│ └── channels
│ └── hellochannels.go
├── chapter10
│ ├── listing01
│ │ └── listing01.go
│ ├── listing02
│ │ └── listing02.go
│ ├── listing03
│ │ └── listing03.go
│ ├── listing04
│ │ └── listing04.go
│ ├── listing05
│ │ └── listing05.go
│ └── listing06
│ ├── listing06
│ ├── listing06.go
│ └── pubsub
│ └── pubsub.go
├── chapter2
│ └── sample
│ ├── data
│ │ └── data.json
│ ├── main.go
│ ├── matchers
│ │ └── rss.go
│ └── search
│ ├── default.go
│ ├── feed.go
│ ├── match.go
│ └── search.go
├── chapter3
│ ├── dbdriver
│ │ ├── main.go
│ │ └── postgres
│ │ └── postgres.go
│ ├── wordcount
│ │ ├── gowords.txt
│ │ └── wordcount.go
│ └── words
│ └── count.go
├── chapter5
│ ├── listing11
│ │ └── listing11.go
│ ├── listing34
│ │ └── listing34.go
│ ├── listing35
│ │ └── listing35.go
│ ├── listing36
│ │ └── listing36.go
│ ├── listing46
│ │ └── listing46.go
│ ├── listing48
│ │ └── listing48.go
│ ├── listing50
│ │ └── listing50.go
│ ├── listing56
│ │ └── listing56.go
│ ├── listing60
│ │ └── listing60.go
│ ├── listing64
│ │ ├── counters
│ │ │ └── counters.go
│ │ └── listing64.go
│ ├── listing68
│ │ ├── counters
│ │ │ └── counters.go
│ │ └── listing68.go
│ ├── listing71
│ │ ├── entities
│ │ │ └── entities.go
│ │ └── listing71.go
│ └── listing74
│ ├── entities
│ │ └── entities.go
│ └── listing74.go
├── chapter6
│ ├── listing01
│ │ └── listing01.go
│ ├── listing04
│ │ └── listing04.go
│ ├── listing07
│ │ └── listing07.go
│ ├── listing09
│ │ └── listing09.go
│ ├── listing13
│ │ └── listing13.go
│ ├── listing15
│ │ └── listing15.go
│ ├── listing16
│ │ └── listing16.go
│ ├── listing20
│ │ └── listing20.go
│ ├── listing22
│ │ └── listing22.go
│ └── listing24
│ └── listing24.go
├── chapter7
│ └── patterns
│ ├── pool
│ │ ├── main
│ │ │ └── main.go
│ │ └── pool.go
│ ├── runner
│ │ ├── main
│ │ │ └── main.go
│ │ └── runner.go
│ ├── search
│ │ ├── main
│ │ │ └── main.go
│ │ ├── searchers.go
│ │ └── search.go
│ ├── semaphore
│ │ └── semaphore.go
│ └── work
│ ├── main
│ │ └── main.go
│ └── work.go
├── chapter8
│ ├── listing03
│ │ └── listing03.go
│ ├── listing11
│ │ └── listing11.go
│ ├── listing24
│ │ └── listing24.go
│ ├── listing27
│ │ └── listing27.go
│ ├── listing29
│ │ └── listing29.go
│ ├── listing31
│ │ └── listing31.go
│ ├── listing37
│ │ └── listing37.go
│ └── listing46
│ └── listing46.go
├── chapter9
│ ├── listing01
│ │ └── listing01_test.go
│ ├── listing08
│ │ └── listing08_test.go
│ ├── listing12
│ │ └── listing12_test.go
│ ├── listing17
│ │ ├── handlers
│ │ │ ├── handlers_example_test.go
│ │ │ ├── handlers.go
│ │ │ └── handlers_test.go
│ │ └── listing17.go
│ └── listing28
│ └── listing28_test.go
└── README.md 77 directories, 75 files
[root@hadoop3 sample]#
https://github.com/goinaction/code/blob/master/chapter2/sample/search/search.go
// Launch a goroutine to monitor when all the work is done.
go func() {
// Wait for everything to be processed.
waitGroup.Wait() // Close the channel to signal to the Display
// function that we can exit the program.
close(results)
}()
// Run performs the search logic.
func Run(searchTerm string) {
// Retrieve the list of feeds to search through.
feeds, err := RetrieveFeeds()
if err != nil {
log.Fatal(err)
} // Create an unbuffered channel to receive match results to display.
results := make(chan *Result) // Setup a wait group so we can process all the feeds.
var waitGroup sync.WaitGroup // Set the number of goroutines we need to wait for while
// they process the individual feeds.
waitGroup.Add(len(feeds)) // Launch a goroutine for each feed to find the results.
for _, feed := range feeds {
// Retrieve a matcher for the search.
matcher, exists := matchers[feed.Type]
if !exists {
matcher = matchers["default"]
} // Launch the goroutine to perform the search.
go func(matcher Matcher, feed *Feed) {
Match(matcher, feed, searchTerm, results)
waitGroup.Done()
}(matcher, feed)
} // Launch a goroutine to monitor when all the work is done.
go func() {
// Wait for everything to be processed.
waitGroup.Wait() // Close the channel to signal to the Display
// function that we can exit the program.
close(results)
}() // Start displaying results as they are available and
// return after the final result is displayed.
Display(results)
}
go通道消息共享
从数据源拉取数据，将数据内容与一组搜索项做比对 go func() chanel的更多相关文章
- FLUME安装&环境（二）:拉取MySQL数据库数据到Kafka
Flume安装成功,环境变量配置成功后,开始进行agent配置文件设置. 1.agent配置文件(mysql+flume+Kafka) #利用Flume将MySQL表数据准实时抽取到Kafka a1. ...
- 使用Jenkins的Git Parameter插件来从远程仓库拉取指定目录的内容
更换插件安装源 系统管理---插件管理---高级---升级站点,把默认的插件下载地址换成下面这个: https://mirrors.tuna.tsinghua.edu.cn/jenkins/updat ...
- 解决 MySQL 比如我要拉取一个消息表中用户id为1的前10条最新数据
我们都知道,各种主流的社交应用或者阅读应用,基本都有列表类视图,并且都有滑到底部加载更多这一功能, 对应后端就是分页拉取数据.好处不言而喻,一般来说,这些数据项都是按时间倒序排列的,用户只关心最新的动 ...
- canal从mysql拉取数据，并以protobuf的格式往kafka中写数据
大致思路: canal去mysql拉取数据,放在canal所在的节点上,并且自身对外提供一个tcp服务,我们只要写一个连接该服务的客户端,去拉取数据并且指定往kafka写数据的格式就能达到以proto ...
- 版本控制git之三-多人协作 变基 推送 拉取 删除远程分支
版本控制git之三-多人协作 wangfeng7399已关注0人评论350人阅读2019-02-20 21:33:08 如果你想获得一份已经存在了的 Git 仓库的拷贝,比如说,你想为某个开源 ...
- 源码分析Kafka 消息拉取流程
目录 1.KafkaConsumer poll 详解 2.Fetcher 类详解 本节重点讨论 Kafka 的消息拉起流程. @(本节目录) 1.KafkaConsumer poll 详解 消息拉起主 ...
- Mysql日期类型大小比较---拉取给定时间段的记录
我们知道,mysql里边,日期类型有很多表现形式,date, datetime,timestamp等类型.考虑这样一种场景: 按时间段拉取给定时间段的内容,这时,我们就得使用日期类型的比较了. 表结构 ...
- idea配置git，查看git代码&拉取git项目至本地
1.点击file,右键选择setting 选择本地git安装路径 Ps:从git上导入一个全新的maven项目 点击clone按钮后,会弹出如下截图弹窗,点击 NO 项目已经拉取到本地,然后点击ope ...
- Spark Streaming中向flume拉取数据
在这里看到的解决方法 https://issues.apache.org/jira/browse/SPARK-1729 请是个人理解,有问题请大家留言. 其实本身flume是不支持像KAFKA一样的发 ...
随机推荐
- SoapUI offen used scripts
[ Groovy Step ] //get properties from testCase, testSuite and project in groovy step def testCasePro ...
- Tomcat 调优技巧
Tomcat 调优技巧:1.Tomcat自身调优: ①采用动静分离节约Tomcat的性能: ②调整Tomcat的线程池: ③调整Tomcat的连接器: ④修改Tomcat的运行模式: ⑤禁用AJP连接 ...
- cf711E ZS and The Birthday Paradox
ZS the Coder has recently found an interesting concept called the Birthday Paradox. It states that g ...
- 维修队列（bzoj 1500）
Description Input 输入的第1 行包含两个数N 和M(M ≤20 000),N 表示初始时数列中数的个数,M表示要进行的操作数目.第2行包含N个数字,描述初始时的数列.以下M行,每行一 ...
- 洛谷 [P1995] 程序自动分析
并查集+ 离散化 首先本题的数据范围很大,需要离散化, STL离散化代码: //dat是原数据,id是编号,sub是数据的副本 sort(sub + 1, sub + tot + 1); size = ...
- 自动化测试框架之robot framework的应用分析
序言:很多人都对自动化测试框架痴迷,我曾经也痴迷过一段时间,以前觉得自己对框架说的头头是道,现在回过头来看以前,说归说,但在如何应用还是欠缺,这一段时间,自己经历了一系列框架的构建和应用的时期,所以, ...
- (47)C#运行时序列化
序列化是将对象或对象图转化成字节流的过程.反序列化是将字节流转换回对象图的过程.
- 【WEB基础】HTML & CSS 基础入门（6）超链接
超链接--文字链接 超链接[hyperlink]是网页中最为常见的元素之一,我们几乎可以在所有的网站页面中找到超链接.每个网站都不止一个页面,这些页面就是利用超链接进行串接.超链接帮我们实现了网页与网 ...
- VS调试时不加载符号
加载符号,能够让调试时进入系统的dll中,甚至能达到汇编语言的深度,让你调试时能够逐行进行.但是加载符号的话,会使得每次调试,增加进入到首个断点的时间. 而且有时候也非必要加载符号,它的取消可以使用下 ...
- 使用uilabel重新自调整高度后显示横线和竖线问题
这个使用uilabel自调节高度发现的问题,代码如下: //content label NSString *contentValue = ((Messag ...