对文件进行词频统计,是一个大数据领域的hello word级别的应用,来看下实现有多简单: 1 Linux单机处理 egrep -o "\b[[:alpha:]]+\b" test_word.log|sort|uniq -c|sort -rn|head -10 2 Spark分布式处理(Scala) val sparkConf = new SparkConf() val sc = new SparkContext(sparkConf) sc.textFile("test_wo…

目录 摘要 算法关键 红黑树 稳定排序 代码框架 .h文件: .cpp文件 频率统计器的实现 接口设计与实现 接口设计 核心功能词频统计器流程 效果 单元测试 性能分析 性能分析图 问题发现 解决方案 异常处理 PSP表格记录 感想 基于sketch在大数据下的词频统计设计 引言 背景 解决方案 总结 参考文献: Github项目地址 摘要 本词频统计器包括行数统计.字符数统计.单词数统计.词频统计功能.基于红8黑树算法和稳定排序实现,其中红黑树算法为本词频统计器提供良好的效率.提供性能下限保证…

核心枚举 public enum ServerState { LOOKING, FOLLOWING, LEADING, OBSERVING; } zookeeper服务器状态:刚启动LOOKING,follower是FOLLOWING,leader是LEADING,observer是OBSERVING: public enum LearnerType { PARTICIPANT, OBSERVER; } 简单来说,zookeeper启动的核心类是QuorumPeerMain,启动之后会加载配置,…

一 简介 Shuffle,简而言之,就是对数据进行重新分区,其中会涉及大量的网络io和磁盘io,为什么需要shuffle,以词频统计reduceByKey过程为例, serverA:partition1: (hello, 1), (word, 1)serverB:partition2: (hello, 2) shuffle之后: serverA:partition1: (hello, 1), (hello, 2)serverB:partition2: (word, 1) 最后才能得到结果: (h…

一 简介 spark核心是RDD,官方文档地址:https://spark.apache.org/docs/latest/rdd-programming-guide.html#resilient-distributed-datasets-rdds官方描述如下:重点是可容错,可并行处理 Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant colle…

Flink 1.7 官方:https://flink.apache.org/ 一 简介 Apache Flink is an open source platform for distributed stream and batch data processing. Flink’s core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for…

Fortunately, Elasticsearch provides a very comprehensive and powerful REST API that you can use to interact with your cluster. Among the few things that can be done with the API are as follows: Check your cluster, node, and index health, status, and…

impala2.12 官方:http://impala.apache.org/ 一 简介 Apache Impala is the open source, native analytic database for Apache Hadoop. Impala is shipped by Cloudera, MapR, Oracle, and Amazon. impala是hadoop上的开源分析性数据库:C++和java语言开发: Do BI-style Queries on Hadoop Im…

tpc 官方:http://www.tpc.org/ 一 简介 The TPC is a non-profit corporation founded to define transaction processing and database benchmarks and to disseminate objective, verifiable TPC performance data to the industry. TPC(The Transaction Processing Perform…

1 下载 https://www.mongodb.com/download-center/community 比如: https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-4.0.9.tgz 2 连接 # cd $MONGODB_HOME# bin/mongo master:27017/database_name 3 查询 1 count >db.getCollection('table_name').find({}).count() 2 g…