spark下统计单词频次
写了一个简单的语句，还没有优化：
scala> sc.
| textFile("/etc/profile").
| flatMap((s:String)=>s.split("\\s")).
| map(_.toUpperCase).
| map((s:String)=>(s, 1)).
| filter((pair)=>pair._1.forall((ch)=>ch>'A'&&ch<'Z')).
| reduceByKey(_+_).
| sortByKey().
| foreach(println)注意这代码还可以优化：
scala> sc.
| textFile("/etc/profile").
| flatMap(_.split("\\s")).
| map(_.toUpperCase).
| map((_, 1)).
| filter(_._1.forall((ch)=>ch>'A'&&ch<'Z')).
| reduceByKey(_+_).
| sortByKey().
| foreach(println)输出结果如下：
15/03/06 08:50:44 INFO MemoryStore: ensureFreeSpace(75904) called with curMem=259812, maxMem=277842493
15/03/06 08:50:44 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 74.1 KB, free 264.7 MB)
15/03/06 08:50:44 INFO FileInputFormat: Total input paths to process : 1
15/03/06 08:50:44 INFO SparkContext: Starting job: sortByKey at <console>:20
15/03/06 08:50:44 INFO DAGScheduler: Registering RDD 25 (filter at <console>:18)
15/03/06 08:50:44 INFO DAGScheduler: Got job 4 (sortByKey at <console>:20) with 2 output partitions (allowLocal=false)
15/03/06 08:50:44 INFO DAGScheduler: Final stage: Stage 10(sortByKey at <console>:20)
15/03/06 08:50:44 INFO DAGScheduler: Parents of final stage: List(Stage 11)
15/03/06 08:50:44 INFO DAGScheduler: Missing parents: List(Stage 11)
15/03/06 08:50:44 INFO DAGScheduler: Submitting Stage 11 (FilteredRDD[25] at filter at <console>:18), which has no missing parents
15/03/06 08:50:44 INFO MemoryStore: ensureFreeSpace(3736) called with curMem=335716, maxMem=277842493
15/03/06 08:50:44 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 3.6 KB, free 264.6 MB)
15/03/06 08:50:44 INFO DAGScheduler: Submitting 2 missing tasks from Stage 11 (FilteredRDD[25] at filter at <console>:18)
15/03/06 08:50:44 INFO TaskSchedulerImpl: Adding task set 11.0 with 2 tasks
15/03/06 08:50:44 INFO TaskSetManager: Starting task 0.0 in stage 11.0 (TID 16, localhost, PROCESS_LOCAL, 1162 bytes)
15/03/06 08:50:44 INFO TaskSetManager: Starting task 1.0 in stage 11.0 (TID 17, localhost, PROCESS_LOCAL, 1162 bytes)
15/03/06 08:50:44 INFO Executor: Running task 1.0 in stage 11.0 (TID 17)
15/03/06 08:50:44 INFO Executor: Running task 0.0 in stage 11.0 (TID 16)
15/03/06 08:50:44 INFO HadoopRDD: Input split: file:/etc/profile:1189+1189
15/03/06 08:50:44 INFO HadoopRDD: Input split: file:/etc/profile:0+1189
15/03/06 08:50:44 INFO Executor: Finished task 1.0 in stage 11.0 (TID 17). 1863 bytes result sent to driver
15/03/06 08:50:44 INFO TaskSetManager: Finished task 1.0 in stage 11.0 (TID 17) in 43 ms on localhost (1/2)
15/03/06 08:50:44 INFO Executor: Finished task 0.0 in stage 11.0 (TID 16). 1863 bytes result sent to driver
15/03/06 08:50:44 INFO TaskSetManager: Finished task 0.0 in stage 11.0 (TID 16) in 51 ms on localhost (2/2)
15/03/06 08:50:44 INFO DAGScheduler: Stage 11 (filter at <console>:18) finished in 0.054 s
15/03/06 08:50:44 INFO DAGScheduler: looking for newly runnable stages
15/03/06 08:50:44 INFO DAGScheduler: running: Set()
15/03/06 08:50:44 INFO DAGScheduler: waiting: Set(Stage 10)
15/03/06 08:50:44 INFO DAGScheduler: failed: Set()
15/03/06 08:50:44 INFO TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool
15/03/06 08:50:44 INFO DAGScheduler: Missing parents for Stage 10: List()
15/03/06 08:50:44 INFO DAGScheduler: Submitting Stage 10 (MapPartitionsRDD[28] at sortByKey at <console>:20), which is now runnable
15/03/06 08:50:44 INFO MemoryStore: ensureFreeSpace(2856) called with curMem=339452, maxMem=277842493
15/03/06 08:50:44 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 2.8 KB, free 264.6 MB)
15/03/06 08:50:44 INFO DAGScheduler: Submitting 2 missing tasks from Stage 10 (MapPartitionsRDD[28] at sortByKey at <console>:20)
15/03/06 08:50:44 INFO TaskSchedulerImpl: Adding task set 10.0 with 2 tasks
15/03/06 08:50:44 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 18, localhost, PROCESS_LOCAL, 948 bytes)
15/03/06 08:50:44 INFO TaskSetManager: Starting task 1.0 in stage 10.0 (TID 19, localhost, PROCESS_LOCAL, 948 bytes)
15/03/06 08:50:44 INFO Executor: Running task 0.0 in stage 10.0 (TID 18)
15/03/06 08:50:44 INFO Executor: Running task 1.0 in stage 10.0 (TID 19)
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, targetRequestSize: 10066329
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote fetches in 0 ms
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, targetRequestSize: 10066329
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote fetches in 0 ms
15/03/06 08:50:44 INFO Executor: Finished task 0.0 in stage 10.0 (TID 18). 1165 bytes result sent to driver
15/03/06 08:50:44 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 18) in 18 ms on localhost (1/2)
15/03/06 08:50:44 INFO Executor: Finished task 1.0 in stage 10.0 (TID 19). 1293 bytes result sent to driver
15/03/06 08:50:44 INFO TaskSetManager: Finished task 1.0 in stage 10.0 (TID 19) in 28 ms on localhost (2/2)
15/03/06 08:50:44 INFO DAGScheduler: Stage 10 (sortByKey at <console>:20) finished in 0.031 s
15/03/06 08:50:44 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
15/03/06 08:50:44 INFO SparkContext: Job finished: sortByKey at <console>:20, took 0.107864348 s
15/03/06 08:50:44 INFO SparkContext: Starting job: foreach at <console>:21
15/03/06 08:50:44 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 4 is 144 bytes
15/03/06 08:50:44 INFO DAGScheduler: Registering RDD 26 (reduceByKey at <console>:19)
15/03/06 08:50:44 INFO DAGScheduler: Got job 5 (foreach at <console>:21) with 2 output partitions (allowLocal=false)
15/03/06 08:50:44 INFO DAGScheduler: Final stage: Stage 12(foreach at <console>:21)
15/03/06 08:50:44 INFO DAGScheduler: Parents of final stage: List(Stage 14)
15/03/06 08:50:44 INFO DAGScheduler: Missing parents: List(Stage 14)
15/03/06 08:50:44 INFO DAGScheduler: Submitting Stage 14 (ShuffledRDD[26] at reduceByKey at <console>:19), which has no missing parents
15/03/06 08:50:44 INFO MemoryStore: ensureFreeSpace(2472) called with curMem=342308, maxMem=277842493
15/03/06 08:50:44 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 2.4 KB, free 264.6 MB)
15/03/06 08:50:44 INFO DAGScheduler: Submitting 2 missing tasks from Stage 14 (ShuffledRDD[26] at reduceByKey at <console>:19)
15/03/06 08:50:44 INFO TaskSchedulerImpl: Adding task set 14.0 with 2 tasks
15/03/06 08:50:44 INFO TaskSetManager: Starting task 0.0 in stage 14.0 (TID 20, localhost, PROCESS_LOCAL, 937 bytes)
15/03/06 08:50:44 INFO TaskSetManager: Starting task 1.0 in stage 14.0 (TID 21, localhost, PROCESS_LOCAL, 937 bytes)
15/03/06 08:50:44 INFO Executor: Running task 1.0 in stage 14.0 (TID 21)
15/03/06 08:50:44 INFO Executor: Running task 0.0 in stage 14.0 (TID 20)
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, targetRequestSize: 10066329
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote fetches in 0 ms
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, targetRequestSize: 10066329
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
15/03/06 08:50:44 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote fetches in 1 ms
15/03/06 08:50:44 INFO Executor: Finished task 1.0 in stage 14.0 (TID 21). 996 bytes result sent to driver
15/03/06 08:50:44 INFO TaskSetManager: Finished task 1.0 in stage 14.0 (TID 21) in 14 ms on localhost (1/2)
15/03/06 08:50:44 INFO Executor: Finished task 0.0 in stage 14.0 (TID 20). 996 bytes result sent to driver
15/03/06 08:50:44 INFO TaskSetManager: Finished task 0.0 in stage 14.0 (TID 20) in 21 ms on localhost (2/2)
15/03/06 08:50:44 INFO TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool
15/03/06 08:50:44 INFO DAGScheduler: Stage 14 (reduceByKey at <console>:19) finished in 0.022 s
15/03/06 08:50:44 INFO DAGScheduler: looking for newly runnable stages
15/03/06 08:50:44 INFO DAGScheduler: running: Set()
15/03/06 08:50:44 INFO DAGScheduler: waiting: Set(Stage 12)
15/03/06 08:50:44 INFO DAGScheduler: failed: Set()
15/03/06 08:50:44 INFO DAGScheduler: Missing parents for Stage 12: List()
15/03/06 08:50:44 INFO DAGScheduler: Submitting Stage 12 (ShuffledRDD[29] at sortByKey at <console>:20), which is now runnable
15/03/06 08:50:44 INFO MemoryStore: ensureFreeSpace(2304) called with curMem=344780, maxMem=277842493
15/03/06 08:50:44 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 2.3 KB, free 264.6 MB)
15/03/06 08:50:44 INFO DAGScheduler: Submitting 2 missing tasks from Stage 12 (ShuffledRDD[29] at sortByKey at <console>:20)
15/03/06 08:50:44 INFO TaskSchedulerImpl: Adding task set 12.0 with 2 tasks
15/03/06 08:50:44 INFO TaskSetManager: Starting task 0.0 in stage 12.0 (TID 22, localhost, PROCESS_LOCAL, 948 bytes)
15/03/06 08:50:44 INFO TaskSetManager: Starting task 1.0 in stage 12.0 (TID 23, localhost, PROCESS_LOCAL, 948 bytes)
15/03/06 08:50:45 INFO Executor: Running task 1.0 in stage 12.0 (TID 23)
15/03/06 08:50:45 INFO Executor: Running task 0.0 in stage 12.0 (TID 22)
15/03/06 08:50:45 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, targetRequestSize: 10066329
15/03/06 08:50:45 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
15/03/06 08:50:45 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote fetches in 0 ms
15/03/06 08:50:45 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, targetRequestSize: 10066329
15/03/06 08:50:45 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
15/03/06 08:50:45 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote fetches in 0 ms
(LOGIN,2)
(MERGING,1)
(MUCH,1)
(NEED,1)
(NOT,1)
(PREVENT,1)
(RESERVED,1)
(SCRIPT,1)
(SETS,1)
(SETUP,1)
(SHELL,2)
(SYSTEM,2)
(THE,1)
(THEN,8)
(THIS,3)
(THRESHOLD,1)
(TO,5)
(UIDGID,1)
(UNLESS,1)
(UNSET,2)
(USER,1)
(WE,1)
(WIDE,1)
(WILL,1)
(YOU,3)
(YOUR,1)
15/03/06 08:50:45 INFO Executor: Finished task 1.0 in stage 12.0 (TID 23). 826 bytes result sent to driver
15/03/06 08:50:45 INFO TaskSetManager: Finished task 1.0 in stage 12.0 (TID 23) in 13 ms on localhost (1/2)
(,260)
(BETTER,1)
(BY,1)
(CHECK,1)
(COULD,1)
(CURRENT,1)
(CUSTOM,1)
(DO,1)
(DONE,1)
(ELSE,5)
(ENVIRONMENT,1)
(EXPORT,15)
(FI,8)
(FILE,2)
(FOR,5)
(FUNCTIONS,1)
(FUTURE,1)
(GET,1)
(GO,1)
(GOOD,1)
(HISTCONTROL,1)
(I,2)
(IF,8)
(IN,6)
(IS,1)
(IT,1)
(KNOW,1)
(KSH,1)
15/03/06 08:50:45 INFO Executor: Finished task 0.0 in stage 12.0 (TID 22). 826 bytes result sent to driver
15/03/06 08:50:45 INFO TaskSetManager: Finished task 0.0 in stage 12.0 (TID 22) in 27 ms on localhost (2/2)
15/03/06 08:50:45 INFO TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool
15/03/06 08:50:45 INFO DAGScheduler: Stage 12 (foreach at <console>:21) finished in 0.025 s
15/03/06 08:50:45 INFO SparkContext: Job finished: foreach at <console>:21, took 0.07397057 s通过如下代码，可以输出参与计算的节点名称，注意start-all并指定shell的–master参数：
spark-shell --master spark://bluejoe0:7077
代码如下：
rdd.mapPartitions(_=>Array[String](("hostname" !!).trim).iterator, false).collect
res28: Array[String] = Array(bluejoe4, bluejoe5)spark下统计单词频次的更多相关文章
- Perl 笔试题2 -- 统计单词频次
Nvidia 2019 perl 笔试题 统计一个文件内单词的频次并排序 文本如下: "ALL happy families resemble one another; every unha ...
- Linux下统计出现次数最多的指定字段值
假设桌面上有一个叫“data.txt”的文本,内容如下: {id='xxx' info='xxx' kk='xxx' target='111111' dd='xxx'}{id='xxx' info=' ...
- [Ext JS 4] 实战之多选下拉单 (带checkbox)
前言 Ext js 创建一个多选下拉单的方式很简单, 使用Ext.form.ComboBox, 设置 multiSelect 为true 就可以了. 但是如果要在每个下拉之前加上一个checkbox, ...
- Windows 下统计行数的命令
大家都知道在Linux下统计文本行数能够用wc -l 命令.比如: -bash-3.2$ cat pif_install.log | wc -l 712 但在Windows下怎样统计输出文 ...
- SharePoint 2013 InfoPath 无法保存下列表单
转载自:http://www.cnblogs.com/jianyus/p/3470121.html 在使用InfoPath发布表单,发布到SharePoint服务器报错,如下介绍: 环境:Window ...
- 使用jdk8 stream 统计单词数
在我的SpringBoot2.0不容错过的新特性 WebFlux响应式编程里面,有同学问如何使用stream统计单词数.这是个好例子,也很典型,在这里补上. 下面的例子实现了从一个文本文件读取(英文) ...
- 【转】【Linux】Linux下统计当前文件夹下的文件个数、目录个数
[转][Linux]Linux下统计当前文件夹下的文件个数.目录个数 统计当前文件夹下文件的个数,包括子文件夹里的 ls -lR|grep "^-"|wc -l 统计文件夹下目录的 ...
- Spark2.2+ES6.4.2（三十一）：Spark下生成测试数据，并在Spark环境下使用BulkProcessor将测试数据入库到ES
Spark下生成2000w测试数据(每条记录150列) 使用spark生成大量数据过程中遇到问题,如果sc.parallelize(fukeData, 64);的记录数特别大比如500w,1000w时 ...
- JS实现下拉单的二级联动
因工作需要,做了一个下拉单的二级联动. 第一级是固定的选项,有A.B两个选项,第二级的选项随着第一级选项的变化而变化. 一开始是这样的: HTML代码 <html> <head> ...
随机推荐
- 2.1CUDA-Thread
在HOST端我们会分配block的dimension, grid的dimension.但是对应到实际的硬件是如何执行这些硬件的呢? 如下图: lanuch kernel 执行一个grid. 一个Gri ...
- Java自定义日志输出文件
Java自定义日志输出文件 日志的打印,在程序中是必不可少的,如果需要将不同的日志打印到不同的地方,则需要定义不同的Appender,然后定义每一个Appender的日志级别.打印形式和日志的输出路径 ...
- Struts2.x jsp页面无法使用jsp:forward跳转到action
问题:使用<jsp:forward page="test"></jsp:forward>语句无法跳转到test所对应的action. 解决办法:在web.x ...
- HDU-4616 Game 树形DP
题目链接:http://acm.hdu.edu.cn/showproblem.php?pid=4616 比较典型的树形DP题目,f[u][j][k]表示以点u为子树,经过 j 个陷阱的最大值,其中k= ...
- 转载 C#匿名函数 委托和Lambda表达式
转载原出处: http://blog.csdn.net/honantic/article/details/46331875 匿名函数 匿名函数(Anonymous Function)是表示“内联”方法 ...
- hdoj 2829 Lawrence 四边形不等式优化dp
dp[i][j]表示前i个,炸j条路,并且最后一个炸在i的后面时,一到i这一段的最小价值. dp[i][j]=min(dp[i][k]+w[k+1][i]) w[i][j]表示i到j这一段的价值. # ...
- 【Stage3D学习笔记续】真正的3D世界（五）：粒子特效
先看效果,按下空格键添加粒子特效: 一般而言粒子特效的实现都是比较复杂的,且不说实现粒子特效的编码和设计,光是编写一个粒子编辑器就不是简单的一件事,但是作者使用了很取巧的方式来完成,我们接下来深入代码 ...
- C#- 将秒数转化成任意时间格式
将秒数转化成任意时间格式,可以使用C#的一个函数TimeSpan,看示例: TimeSpan ts = new TimeSpan(0, 0, 3661); richTextBox2.Text = ts ...
- 从零开始学android开发-查看sqlite数据库
C:\Users\Administrator>cd E:\ProSoft\adt-bundle-windows-x86-20140321\sdk\platform-tools
- BZOJ 1878: [SDOI2009]HH的项链 离线树状数组
1878: [SDOI2009]HH的项链 Time Limit: 1 Sec Memory Limit: 256 MB 题目连接 http://www.lydsy.com/JudgeOnline/p ...