计算均值mean的MapReduce程序Computing mean with MapReduce
In this post we'll see how to compute the mean of the max temperatures of every month for the city of Milan.
The temperature data is taken from http://archivio-meteo.distile.it/tabelle-dati-archivio-meteo/, but since the data are shown in tabular form, we had to sniff the HTTP conversation to see that the data come from this URL and are in JSON format.
Using Jackson, we could transform this JSON into a format simpler to use with Hadoop: CSV. The result of conversion is this:
01012000,-4.0,5.0
02012000,-5.0,5.1
03012000,-5.0,7.7
04012000,-3.0,9.7
...
If you're curious to see how we transformed it, take a look at the source code.
Let's look at the mapper class for this job:
public static class MeanMapper extends Mapper<Object, Text, Text, SumCount> { private final int DATE = 0;
private final int MIN = 1;
private final int MAX = 2; private Map<Text, List<Double>> maxMap = new HashMap<>(); @Override
public void map(Object key, Text value, Context context) throws IOException, InterruptedException { // gets the fields of the CSV line
String[] values = value.toString().split((",")); // defensive check
if (values.length != 3) {
return;
} // gets date and max temperature
String date = values[DATE];
Text month = new Text(date.substring(2));
Double max = Double.parseDouble(values[MAX]); // if not present, put this month into the map
if (!maxMap.containsKey(month)) {
maxMap.put(month, new ArrayList<Double>());
} // adds the max temperature for this day to the list of temperatures
maxMap.get(month).add(max);
} @Override
protected void cleanup(Context context) throws IOException, InterruptedException { // loops over the months collected in the map() method
for (Text month: maxMap.keySet()) { List<Double> temperatures = maxMap.get(month); // computes the sum of the max temperatures for this month
Double sum = 0d;
for (Double max: temperatures) {
sum += max;
} // emits the month as the key and a SumCount as the value
context.write(month, new SumCount(sum, temperatures.size()));
}
}
}
How we've seen in the last posts (about optimization and combiners), in the mapper we first put values into a map, and when the input is over, we loop over the keys to sum the values and to emit them. Note that we use the SumCount class, which is a utility class that wraps the two values we need to compute a mean: the sum of all the values and the number of values.
A common error in this kind of computation is making the mapper directly emit the mean; let's see what it can happen if we suppose to have a dataset like this:
01012000,0,10.0
02012000,0,20.0
03012000,0,2.0
04012000,0,4.0
05012000,0,3.0
and two mappers, which will receive the first two and the last three lines respectively. The first mapper will compute a mean of 15.0, given from (10.0 + 20.0) / 2. The second will compute a mean of 3.0, given from (2.0 + 4.0 + 3.0) / 3. When the reducer receive this two values, it sums them together and divide by two, so that the mean will be: 9.0, given from (15.0 + 3.0) / 2. But the correct mean for the values in this example is 7.8, which is given from (10.0 + 20.0 + 4.0 + 2.0 + 3.0) / 5.
This error is due to the fact that any mapper can receive any number of lines, so the value it will emit is only a part of the information needed to compute a mean.
If instead of emitting the mean we emit the sum of the values and the number of values, we can overcome the problem. In the example we saw before, the first mapper will emit the pair (30.0, 2) and the second (9.0, 3); if we sum the values and divide it by the sum of the numbers, we obtain the right result.
Let's get back to our job and look at the reducer:
public static class MeanReducer extends Reducer<text, sumcount,="" text,="" doublewritable=""> { private Map<text, sumcount=""> sumCountMap = new HashMap<>(); @Override
public void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException { SumCount totalSumCount = new SumCount(); // loops over all the SumCount objects received for this month (the "key" param)
for (SumCount sumCount : values) { // sums all of them
totalSumCount.addSumCount(sumCount);
} // puts the resulting SumCount into a map
sumCountMap.put(new Text(key), totalSumCount);
} @Override
protected void cleanup(Context context) throws IOException, InterruptedException { // loops over the months collected in the reduce() method
for (Text month: sumCountMap.keySet()) { double sum = sumCountMap.get(month).getSum().get();
int count = sumCountMap.get(month).getCount().get(); // emits the month and the mean of the max temperatures for the month
context.write(month, new DoubleWritable(sum/count));
}
}
}
The reducer is simpler because it has just to retrieve all the SumCount objects emitted from the reducers and add them together. After receiving the input, it loops over the map of the SumCount objects and emits the month and the mean.
from: http://andreaiacono.blogspot.com/2014/04/computing-mean-with-mapreduce.html
计算均值mean的MapReduce程序Computing mean with MapReduce的更多相关文章
- 用Python语言写Hadoop MapReduce程序Writing an Hadoop MapReduce Program in Python
In this tutorial I will describe how to write a simple MapReduce program for Hadoop in the Python pr ...
- 一起学Hadoop——使用IDEA编写第一个MapReduce程序(Java和Python)
上一篇我们学习了MapReduce的原理,今天我们使用代码来加深对MapReduce原理的理解. wordcount是Hadoop入门的经典例子,我们也不能免俗,也使用这个例子作为学习Hadoop的第 ...
- HDFS设计思路,HDFS使用,查看集群状态,HDFS,HDFS上传文件,HDFS下载文件,yarn web管理界面信息查看,运行一个mapreduce程序,mapreduce的demo
26 集群使用初步 HDFS的设计思路 l 设计思想 分而治之:将大文件.大批量文件,分布式存放在大量服务器上,以便于采取分而治之的方式对海量数据进行运算分析: l 在大数据系统中作用: 为各类分布式 ...
- [python]使用python实现Hadoop MapReduce程序:计算一组数据的均值和方差
这是参照<机器学习实战>中第15章“大数据与MapReduce”的内容,因为作者写作时hadoop版本和现在的版本相差很大,所以在Hadoop上运行python写的MapReduce程序时 ...
- 怎样通过Java程序提交yarn的mapreduce计算任务
因为项目需求,须要通过Java程序提交Yarn的MapReduce的计算任务.与一般的通过Jar包提交MapReduce任务不同,通过程序提交MapReduce任务须要有点小变动.详见下面代码. 下面 ...
- 简单的java Hadoop MapReduce程序(计算平均成绩)从打包到提交及运行
[TOC] 简单的java Hadoop MapReduce程序(计算平均成绩)从打包到提交及运行 程序源码 import java.io.IOException; import java.util. ...
- mapreduce程序编写(WordCount)
折腾了半天.终于编写成功了第一个自己的mapreduce程序,并通过打jar包的方式运行起来了. 运行环境: windows 64bit eclipse 64bit jdk6.0 64bit 一.工程 ...
- 基于Hbase数据的Mapreduce程序环境开发
一.实验目标 编写Mapreduce程序,以Hbase表数据为Map输入源,计算结果输出到HDFS或者Hbase表中. 在非CDH5的Hadoop集群环境中,将编写好的Mapreduce程序整个工程打 ...
- 从零开始学习Hadoop--第2章 第一个MapReduce程序
1.Hadoop从头说 1.1 Google是一家做搜索的公司 做搜索是技术难度很高的活.首先要存储很多的数据,要把全球的大部分网页都抓下来,可想而知存储量有多大.然后,要能快速检索网页,用户输入几个 ...
随机推荐
- string类总结
头文件: <string> 初始化: string str(s1); string str("value"); , 'c'); 读写 //输入未知数目的string对象 ...
- 易普优APS与国外知名高级计划排程系统对比
众所周知软件执行效率受制于硬件性能,市面上的APS产品多为单机版本,企业要应用好APS,保证紧急插单.计划下发全程无忧,用户电脑硬件性能是不容忽视的一大瓶颈.APS的直接用户是车间管理人员.计划员,而 ...
- Caffe训练AlexNet网络模型——问题二
训练时,出现Check failed:error == cudaSuccess (2 vs. 0) out of memory,并且accruary = 0,如下图所示: 解决方法:将train_va ...
- web到service简单原理例子
这是目前的理解 附上服务端源码 package com.lsw.server; import java.io.*; import java.net.*; import java.util.HashMa ...
- Picasso:开启大前端的未来
“道生一,一生二,二生三,三生万物.” —— <道德经> Picasso是大众点评移动研发团队自研的高性能跨平台动态化框架,经过两年多的孕育和发展,目前在美团多个事业群已经实现了大规模的应 ...
- 回顾2014 Java发生的5件大事
回顾2014 Java发生的5件大事 1.2月1日:RedMonk分析师确认并宣布Java是最受欢迎和多样化的语言! 2014年,Java生态圈伴随着引擎的轰鸣起步,随着FOSDEM年会的Free J ...
- Swift2.0语言教程之类的属性
Swift2.0语言教程之类的属性 类 虽然函数可以简化代码,但是当一个程序中出现成百上千的函数和变量时,代码还是会显得很混乱.为此,人们又引入了新的类型——类.它是人们构建代码所用的一种通用.灵活的 ...
- BZOJ 5059: 前鬼后鬼的守护 可并堆 左偏树 数学
https://www.lydsy.com/JudgeOnline/problem.php?id=5059 题意:将原序列{ai}改为一个递增序列{ai1}并且使得abs(ai-ai1)的和最小. 如 ...
- 【Python3】【树形dp】uva10253 Series-Parallel Networks
设“共n个叶子,且每个非叶节点至少有两个子节点”的树的数量为f[n],再乘2就是本题答案. 设状态d(i,j)表示每棵子树最多包含i个叶子.一共有j个叶子的树的个数.于是f(n)=d(n-1,n).假 ...
- 热爱编程,热爱生活,快速适应IT新技术
现在新技术层出不穷,我们没有那么多时间去深入学习每一门.对于大多数技术,我们只需要搞懂概念部分,从宏观上了解一下,决定我们要不要深入地去学习它.有了这些了解,就能轻松地与别人聊天了,也可以为以后技术选 ...