从数据源拉取数据，将数据内容与一组搜索项做比对 go func() chanel
https://github.com/goinaction/code
[root@hadoop3 sample]# go run main.go
2018/07/30 17:45:39 Register default matcher
2018/07/30 17:45:39 Register rss matcher
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1001]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=46]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=7]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=10]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=39]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=43]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/business/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1008]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1006]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_topstories.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1007]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_world.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1057]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_us.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1021]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_allpolitics.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1012]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_crime.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1003]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=2]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_tech.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_health.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_topstories.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/opinion?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/politics?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/national?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/world?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://feeds.nbcnews.com/feeds/topstories]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://feeds.nbcnews.com/feeds/usnews]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/21491043/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=3]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/21491571/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=5]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/28180066/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=13]
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_allpolitics.rss: read tcp 192.168.3.103:54478->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_topstories.rss: read tcp 192.168.3.103:54484->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_health.rss: read tcp 192.168.3.103:54482->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_crime.rss: read tcp 192.168.3.103:54476->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_us.rss: read tcp 192.168.3.103:54488->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_world.rss: read tcp 192.168.3.103:54486->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Description:
If Ronald Reagan was the “Teflon president,” Donald Trump must be the Incredible Hulk president. The more the left and the media attack him, the stronger he gets.<img src="http://feeds.feedburner.com/~r/foxnews/opinion/~4/bVrx9JLgLWI" height="1" width="1" alt=""/> 2018/07/30 17:45:39 Description:
We look at the latest strong economic numbers and the effect tariffs may have on the president's base. 2018/07/30 17:45:39 Description:
Freeman Hrabowski, president of the University of Maryland, Baltimore County, talks about the importance of mentoring in academia, especially in science, technology, engineering and math fields. 2018/07/30 17:45:39 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
Retail and fast-food workers protesting for higher pay are creating a new kind of U.S. labor movement. <em>Chicago Tribune</em> columnist Clarence Page argues that the president could "set a good example" by requiring fast-food vendors who have contracts with the federal government to pay minimum wage. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
The moment when the president articulated his attitude with pithy perfection came when he told the Veterans of Foreign Wars that "what you're seeing and what you're reading is not what's happening." 2018/07/30 17:45:40 Description:
The U.S. president publicly humiliated our most important allies and then said everything was great. We need positive, unifying leadership, writes a former U.S. Army Europe commander. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
Last year a song about freedom became a rallying cry as Zimbabwe unseated longtime president Robert Mugabe. Now the anthem has become tainted by the lack of tangible change for people there. 2018/07/30 17:45:40 Description:
Last year a song about freedom became a rallying cry as Zimbabwe unseated longtime president Robert Mugabe. Now the anthem has become tainted by the lack of tangible change for people there. 2018/07/30 17:45:40 Description:
We look at the latest strong economic numbers and the effect tariffs may have on the president's base. 2018/07/30 17:45:40 Description:
The president goes back and forth about what he accepts and denies about Russian election interference. Even so, his recent warning about an attack that helps Democrats is a cogent one. 2018/07/30 17:45:40 Description:
The president demanded cooperation from the Democrats over immigration, threatening to close the government in order to achieve his goals. 2018/07/30 17:45:40 Get http://rss.cnn.com/rss/cnn_tech.rss: read tcp 192.168.3.103:54480->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:40 Get http://rss.cnn.com/rss/cnn_topstories.rss: read tcp 192.168.3.103:54474->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/business/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
[root@hadoop3 sample]# tree
.
├── data
│ └── data.json
├── main.go
├── matchers
│ └── rss.go
└── search
├── default.go
├── feed.go
├── match.go
└── search.go 3 directories, 7 files
[root@hadoop3 sample]# tree /home/ctGO/goInAction/
/home/ctGO/goInAction/
└── code
├── chapter1
│ └── channels
│ └── hellochannels.go
├── chapter10
│ ├── listing01
│ │ └── listing01.go
│ ├── listing02
│ │ └── listing02.go
│ ├── listing03
│ │ └── listing03.go
│ ├── listing04
│ │ └── listing04.go
│ ├── listing05
│ │ └── listing05.go
│ └── listing06
│ ├── listing06
│ ├── listing06.go
│ └── pubsub
│ └── pubsub.go
├── chapter2
│ └── sample
│ ├── data
│ │ └── data.json
│ ├── main.go
│ ├── matchers
│ │ └── rss.go
│ └── search
│ ├── default.go
│ ├── feed.go
│ ├── match.go
│ └── search.go
├── chapter3
│ ├── dbdriver
│ │ ├── main.go
│ │ └── postgres
│ │ └── postgres.go
│ ├── wordcount
│ │ ├── gowords.txt
│ │ └── wordcount.go
│ └── words
│ └── count.go
├── chapter5
│ ├── listing11
│ │ └── listing11.go
│ ├── listing34
│ │ └── listing34.go
│ ├── listing35
│ │ └── listing35.go
│ ├── listing36
│ │ └── listing36.go
│ ├── listing46
│ │ └── listing46.go
│ ├── listing48
│ │ └── listing48.go
│ ├── listing50
│ │ └── listing50.go
│ ├── listing56
│ │ └── listing56.go
│ ├── listing60
│ │ └── listing60.go
│ ├── listing64
│ │ ├── counters
│ │ │ └── counters.go
│ │ └── listing64.go
│ ├── listing68
│ │ ├── counters
│ │ │ └── counters.go
│ │ └── listing68.go
│ ├── listing71
│ │ ├── entities
│ │ │ └── entities.go
│ │ └── listing71.go
│ └── listing74
│ ├── entities
│ │ └── entities.go
│ └── listing74.go
├── chapter6
│ ├── listing01
│ │ └── listing01.go
│ ├── listing04
│ │ └── listing04.go
│ ├── listing07
│ │ └── listing07.go
│ ├── listing09
│ │ └── listing09.go
│ ├── listing13
│ │ └── listing13.go
│ ├── listing15
│ │ └── listing15.go
│ ├── listing16
│ │ └── listing16.go
│ ├── listing20
│ │ └── listing20.go
│ ├── listing22
│ │ └── listing22.go
│ └── listing24
│ └── listing24.go
├── chapter7
│ └── patterns
│ ├── pool
│ │ ├── main
│ │ │ └── main.go
│ │ └── pool.go
│ ├── runner
│ │ ├── main
│ │ │ └── main.go
│ │ └── runner.go
│ ├── search
│ │ ├── main
│ │ │ └── main.go
│ │ ├── searchers.go
│ │ └── search.go
│ ├── semaphore
│ │ └── semaphore.go
│ └── work
│ ├── main
│ │ └── main.go
│ └── work.go
├── chapter8
│ ├── listing03
│ │ └── listing03.go
│ ├── listing11
│ │ └── listing11.go
│ ├── listing24
│ │ └── listing24.go
│ ├── listing27
│ │ └── listing27.go
│ ├── listing29
│ │ └── listing29.go
│ ├── listing31
│ │ └── listing31.go
│ ├── listing37
│ │ └── listing37.go
│ └── listing46
│ └── listing46.go
├── chapter9
│ ├── listing01
│ │ └── listing01_test.go
│ ├── listing08
│ │ └── listing08_test.go
│ ├── listing12
│ │ └── listing12_test.go
│ ├── listing17
│ │ ├── handlers
│ │ │ ├── handlers_example_test.go
│ │ │ ├── handlers.go
│ │ │ └── handlers_test.go
│ │ └── listing17.go
│ └── listing28
│ └── listing28_test.go
└── README.md 77 directories, 75 files
[root@hadoop3 sample]#
https://github.com/goinaction/code/blob/master/chapter2/sample/search/search.go
// Launch a goroutine to monitor when all the work is done.
go func() {
// Wait for everything to be processed.
waitGroup.Wait() // Close the channel to signal to the Display
// function that we can exit the program.
close(results)
}()
// Run performs the search logic.
func Run(searchTerm string) {
// Retrieve the list of feeds to search through.
feeds, err := RetrieveFeeds()
if err != nil {
log.Fatal(err)
} // Create an unbuffered channel to receive match results to display.
results := make(chan *Result) // Setup a wait group so we can process all the feeds.
var waitGroup sync.WaitGroup // Set the number of goroutines we need to wait for while
// they process the individual feeds.
waitGroup.Add(len(feeds)) // Launch a goroutine for each feed to find the results.
for _, feed := range feeds {
// Retrieve a matcher for the search.
matcher, exists := matchers[feed.Type]
if !exists {
matcher = matchers["default"]
} // Launch the goroutine to perform the search.
go func(matcher Matcher, feed *Feed) {
Match(matcher, feed, searchTerm, results)
waitGroup.Done()
}(matcher, feed)
} // Launch a goroutine to monitor when all the work is done.
go func() {
// Wait for everything to be processed.
waitGroup.Wait() // Close the channel to signal to the Display
// function that we can exit the program.
close(results)
}() // Start displaying results as they are available and
// return after the final result is displayed.
Display(results)
}
go通道消息共享
从数据源拉取数据，将数据内容与一组搜索项做比对 go func() chanel的更多相关文章
- FLUME安装&环境（二）:拉取MySQL数据库数据到Kafka
Flume安装成功,环境变量配置成功后,开始进行agent配置文件设置. 1.agent配置文件(mysql+flume+Kafka) #利用Flume将MySQL表数据准实时抽取到Kafka a1. ...
- 使用Jenkins的Git Parameter插件来从远程仓库拉取指定目录的内容
更换插件安装源 系统管理---插件管理---高级---升级站点,把默认的插件下载地址换成下面这个: https://mirrors.tuna.tsinghua.edu.cn/jenkins/updat ...
- 解决 MySQL 比如我要拉取一个消息表中用户id为1的前10条最新数据
我们都知道,各种主流的社交应用或者阅读应用,基本都有列表类视图,并且都有滑到底部加载更多这一功能, 对应后端就是分页拉取数据.好处不言而喻,一般来说,这些数据项都是按时间倒序排列的,用户只关心最新的动 ...
- canal从mysql拉取数据，并以protobuf的格式往kafka中写数据
大致思路: canal去mysql拉取数据,放在canal所在的节点上,并且自身对外提供一个tcp服务,我们只要写一个连接该服务的客户端,去拉取数据并且指定往kafka写数据的格式就能达到以proto ...
- 版本控制git之三-多人协作 变基 推送 拉取 删除远程分支
版本控制git之三-多人协作 wangfeng7399已关注0人评论350人阅读2019-02-20 21:33:08 如果你想获得一份已经存在了的 Git 仓库的拷贝,比如说,你想为某个开源 ...
- 源码分析Kafka 消息拉取流程
目录 1.KafkaConsumer poll 详解 2.Fetcher 类详解 本节重点讨论 Kafka 的消息拉起流程. @(本节目录) 1.KafkaConsumer poll 详解 消息拉起主 ...
- Mysql日期类型大小比较---拉取给定时间段的记录
我们知道,mysql里边,日期类型有很多表现形式,date, datetime,timestamp等类型.考虑这样一种场景: 按时间段拉取给定时间段的内容,这时,我们就得使用日期类型的比较了. 表结构 ...
- idea配置git，查看git代码&拉取git项目至本地
1.点击file,右键选择setting 选择本地git安装路径 Ps:从git上导入一个全新的maven项目 点击clone按钮后,会弹出如下截图弹窗,点击 NO 项目已经拉取到本地,然后点击ope ...
- Spark Streaming中向flume拉取数据
在这里看到的解决方法 https://issues.apache.org/jira/browse/SPARK-1729 请是个人理解,有问题请大家留言. 其实本身flume是不支持像KAFKA一样的发 ...
随机推荐
- 【Kubernetes】Deployment控制器模型
在Kubernetes中,Deployment是最基本的控制器对象 apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deploym ...
- Understanding performance, load and stress testing
What are performance, load and stress testing? Performance testing, load testing and stress testing ...
- Codeforces Round #354 (Div. 2)——C. Vasya and String（尺取）
C. Vasya and String time limit per test 1 second memory limit per test 256 megabytes input standard ...
- VB6 post图片
在VBA中怎样用XMLhttp 模拟http上传二进制文件? https://www.zhihu.com/question/40974557 作者:付杨 链接:https://www.zhihu.co ...
- [SCOI2003]字符串折叠 (区间DP)
题目描述 折叠的定义如下: 一个字符串可以看成它自身的折叠.记作S = S X(S)是X(X>1)个S连接在一起的串的折叠.记作X(S) = SSSS…S(X个S). 如果A = A’, B = ...
- jmeter监控linux cpu 内存 网络 IO
下载地址:http://jmeter-plugins.org/downloads/all/ PerfMon: 用来监控Server的CPU.I/O.Memory等情况 ServerAgent-2.2. ...
- 扰动法--*BZOJ3157: 国王奇遇记
求$\sum_{i=1}^ni^mm^i$.$n \leq 1e9,m \leq 200$. 其实我也不知道这东西为啥叫“扰动法”,大概是在黑暗的边缘试探?就是那种,人家再多一点就被您看破了,然后您就 ...
- msp430项目编程47
msp430综合项目---有线采集传输平台系统47 1.电路工作原理 2.代码(显示部分) 3.代码(功能实现) 4.项目总结
- yii 之增加数据
模型代码: <?php namespace app\models; use yii\db\ActiveRecord; class Test extends ActiveRecord{ publi ...
- Spring Boot 2 快速教程：WebFlux 集成 Mongodb（四）
摘要: 原创出处 https://www.bysocket.com 「公众号:泥瓦匠BYSocket 」欢迎关注和转载,保留摘要,谢谢! 这是泥瓦匠的第104篇原创 文章工程:* JDK 1.8* M ...