MapReduce编程系列

1、项目名称：

2、程序代码：

package com.sort;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.util.GenericOptionsParser;

public class Sort {

    //map将输入中的value化成IntWritable类型，作为输出的key

    public static class Map extends Mapper<Object, Text , IntWritable, IntWritable>{

        public static IntWritable data = new IntWritable();

        public void map(Object key , Text value, Context context) throws IOException,InterruptedException{

            System.out.println("Mapper.................");

            System.out.println("key:"+key+"  value:"+value);

            String line = value.toString();

            data.set(Integer.parseInt(line));

            context.write(data, new IntWritable(1));

            System.out.println("data:"+data+" context:"+context);

        }

    }

    //reduce将输入的key复制到输出的value上，然后根据输入的value-list中元素的个数决定key的输出次数

    //用全局linenum来代表key的位次

    public static class Reduce extends Reducer<IntWritable , IntWritable, IntWritable, IntWritable >{

        public static IntWritable linenum = new IntWritable(1);

        public void reduce(IntWritable key, Iterable<IntWritable> values , Context context)throws IOException,InterruptedException{

            System.out.println("Reducer.................");

            System.out.println("key:"+key+"  value:"+values);

            for(IntWritable val : values){

                context.write(linenum, key);

                System.out.println("linenum:" + linenum +"  key:"+key+" context:"+context);

                linenum = new IntWritable(linenum.get()+1);

            }

        }

    }

    public static void main(String [] args) throws Exception{

        Configuration conf = new Configuration();

        String [] otherArgs = new GenericOptionsParser(conf,args).getRemainingArgs();

        if(otherArgs.length != 2){

            System.out.println("Usage: sort<in><out>");

            System.exit(2);

        }

        Job job = new Job(conf,"sort");

        job.setJarByClass(Sort.class);

        job.setMapperClass(Map.class);

        job.setReducerClass(Reduce.class);

        job.setOutputKeyClass(IntWritable.class);

        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, new Path(otherArgs[0]));

        FileOutputFormat.setOutputPath(job,new Path(otherArgs[1]));

        System.exit(job.waitForCompletion(true)? 0 : 1);

    }

}

3、测试数据：

file1：

2
32
654
32
15
756
65223

file2：

5956
22
650
92

file3：

26
54
6

4、运行过程：

14/09/21 17:44:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/09/21 17:44:27 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
14/09/21 17:44:28 INFO input.FileInputFormat: Total input paths to process : 3
14/09/21 17:44:28 WARN snappy.LoadSnappy: Snappy native library not loaded
14/09/21 17:44:28 INFO mapred.JobClient: Running job: job_local_0001
14/09/21 17:44:28 INFO util.ProcessTree: setsid exited with exit code 0
14/09/21 17:44:28 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@365f3cec
14/09/21 17:44:28 INFO mapred.MapTask: io.sort.mb = 100
14/09/21 17:44:28 INFO mapred.MapTask: data buffer = 79691776/99614720
14/09/21 17:44:28 INFO mapred.MapTask: record buffer = 262144/327680
Mapper.................
key:0 value:2
data:2 context:org.apache.hadoop.mapreduce.Mapper$Context@40804be
Mapper.................
key:2 value:32
data:32 context:org.apache.hadoop.mapreduce.Mapper$Context@40804be
Mapper.................
key:5 value:654
data:654 context:org.apache.hadoop.mapreduce.Mapper$Context@40804be
Mapper.................
key:9 value:32
data:32 context:org.apache.hadoop.mapreduce.Mapper$Context@40804be
Mapper.................
key:12 value:15
data:15 context:org.apache.hadoop.mapreduce.Mapper$Context@40804be
Mapper.................
key:15 value:756
data:756 context:org.apache.hadoop.mapreduce.Mapper$Context@40804be
Mapper.................
key:19 value:65223
data:65223 context:org.apache.hadoop.mapreduce.Mapper$Context@40804be
14/09/21 17:44:28 INFO mapred.MapTask: Starting flush of map output
14/09/21 17:44:28 INFO mapred.MapTask: Finished spill 0
14/09/21 17:44:28 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
14/09/21 17:44:29 INFO mapred.JobClient: map 0% reduce 0%
14/09/21 17:44:31 INFO mapred.LocalJobRunner:
14/09/21 17:44:31 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
14/09/21 17:44:31 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5c72877c
14/09/21 17:44:31 INFO mapred.MapTask: io.sort.mb = 100
14/09/21 17:44:31 INFO mapred.MapTask: data buffer = 79691776/99614720
14/09/21 17:44:31 INFO mapred.MapTask: record buffer = 262144/327680
Mapper.................
key:0 value:5956
data:5956 context:org.apache.hadoop.mapreduce.Mapper$Context@5c0134fb
Mapper.................
key:5 value:22
data:22 context:org.apache.hadoop.mapreduce.Mapper$Context@5c0134fb
Mapper.................
key:8 value:650
data:650 context:org.apache.hadoop.mapreduce.Mapper$Context@5c0134fb
Mapper.................
key:12 value:92
data:92 context:org.apache.hadoop.mapreduce.Mapper$Context@5c0134fb
14/09/21 17:44:31 INFO mapred.MapTask: Starting flush of map output
14/09/21 17:44:31 INFO mapred.MapTask: Finished spill 0
14/09/21 17:44:31 INFO mapred.Task: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
14/09/21 17:44:32 INFO mapred.JobClient: map 100% reduce 0%
14/09/21 17:44:34 INFO mapred.LocalJobRunner:
14/09/21 17:44:34 INFO mapred.Task: Task 'attempt_local_0001_m_000001_0' done.
14/09/21 17:44:34 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5c88c5d3
14/09/21 17:44:34 INFO mapred.MapTask: io.sort.mb = 100
14/09/21 17:44:34 INFO mapred.MapTask: data buffer = 79691776/99614720
14/09/21 17:44:34 INFO mapred.MapTask: record buffer = 262144/327680
Mapper.................
key:0 value:26
data:26 context:org.apache.hadoop.mapreduce.Mapper$Context@36a05d78
Mapper.................
key:3 value:54
data:54 context:org.apache.hadoop.mapreduce.Mapper$Context@36a05d78
Mapper.................
key:6 value:6
data:6 context:org.apache.hadoop.mapreduce.Mapper$Context@36a05d78
14/09/21 17:44:34 INFO mapred.MapTask: Starting flush of map output
14/09/21 17:44:34 INFO mapred.MapTask: Finished spill 0
14/09/21 17:44:34 INFO mapred.Task: Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
14/09/21 17:44:37 INFO mapred.LocalJobRunner:
14/09/21 17:44:37 INFO mapred.Task: Task 'attempt_local_0001_m_000002_0' done.
14/09/21 17:44:37 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3c521e5d
14/09/21 17:44:37 INFO mapred.LocalJobRunner:
14/09/21 17:44:37 INFO mapred.Merger: Merging 3 sorted segments
14/09/21 17:44:37 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 146 bytes
14/09/21 17:44:37 INFO mapred.LocalJobRunner:
Reducer.................
key:2 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:1 key:2 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:6 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:2 key:6 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:15 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:3 key:15 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:22 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:4 key:22 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:26 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:5 key:26 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:32 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:6 key:32 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
linenum:7 key:32 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:54 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:8 key:54 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:92 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:9 key:92 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:650 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:10 key:650 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:654 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:11 key:654 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:756 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:12 key:756 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:5956 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:13 key:5956 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
Reducer.................
key:65223 value:org.apache.hadoop.mapreduce.ReduceContext$ValueIterable@38839cf7
linenum:14 key:65223 context:org.apache.hadoop.mapreduce.Reducer$Context@23475bbf
14/09/21 17:44:37 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
14/09/21 17:44:37 INFO mapred.LocalJobRunner:
14/09/21 17:44:37 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
14/09/21 17:44:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://localhost:9000/user/hadoop/sort_output
14/09/21 17:44:40 INFO mapred.LocalJobRunner: reduce > reduce
14/09/21 17:44:40 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
14/09/21 17:44:41 INFO mapred.JobClient: map 100% reduce 100%
14/09/21 17:44:41 INFO mapred.JobClient: Job complete: job_local_0001
14/09/21 17:44:41 INFO mapred.JobClient: Counters: 22
14/09/21 17:44:41 INFO mapred.JobClient:   Map-Reduce Framework
14/09/21 17:44:41 INFO mapred.JobClient:     Spilled Records=28
14/09/21 17:44:41 INFO mapred.JobClient:     Map output materialized bytes=158
14/09/21 17:44:41 INFO mapred.JobClient:     Reduce input records=14
14/09/21 17:44:41 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
14/09/21 17:44:41 INFO mapred.JobClient:     Map input records=14
14/09/21 17:44:41 INFO mapred.JobClient:     SPLIT_RAW_BYTES=345
14/09/21 17:44:41 INFO mapred.JobClient:     Map output bytes=112
14/09/21 17:44:41 INFO mapred.JobClient:     Reduce shuffle bytes=0
14/09/21 17:44:41 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
14/09/21 17:44:41 INFO mapred.JobClient:     Reduce input groups=13
14/09/21 17:44:41 INFO mapred.JobClient:     Combine output records=0
14/09/21 17:44:41 INFO mapred.JobClient:     Reduce output records=14
14/09/21 17:44:41 INFO mapred.JobClient:     Map output records=14
14/09/21 17:44:41 INFO mapred.JobClient:     Combine input records=0
14/09/21 17:44:41 INFO mapred.JobClient:     CPU time spent (ms)=0
14/09/21 17:44:41 INFO mapred.JobClient:     Total committed heap usage (bytes)=1325400064
14/09/21 17:44:41 INFO mapred.JobClient:   File Input Format Counters
14/09/21 17:44:41 INFO mapred.JobClient:     Bytes Read=48
14/09/21 17:44:41 INFO mapred.JobClient:   FileSystemCounters
14/09/21 17:44:41 INFO mapred.JobClient:     HDFS_BYTES_READ=161
14/09/21 17:44:41 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=162878
14/09/21 17:44:41 INFO mapred.JobClient:     FILE_BYTES_READ=3682
14/09/21 17:44:41 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=81
14/09/21 17:44:41 INFO mapred.JobClient:   File Output Format Counters
14/09/21 17:44:41 INFO mapred.JobClient:     Bytes Written=81

5、运行结果：

1   2
2   6
3   15
4   22
5   26
6   32
7   32
8   54
9   92
10   650
11   654
12   756
13   5956
14   65223

MapReduce编程系列 — 4：排序的更多相关文章

【原创】MapReduce编程系列之二元排序
普通排序实现普通排序的实现利用了按姓名的排序,调用了默认的对key的HashPartition函数来实现数据的分组.partition操作之后写入磁盘时会对数据进行排序操作(对一个分区内的数据作排序 ...
MapReduce编程：数字排序
问题描述将乱序数字按照升序排序. 思路描述按照mapreduce的默认排序,依次输出key值. 代码 package org.apache.hadoop.examples; import java ...
MapReduce编程系列 — 6：多表关联
1.项目名称: 2.程序代码: 版本一(详细版): package com.mtjoin; import java.io.IOException; import java.util.Iterator; ...
MapReduce编程系列 — 5：单表关联
1.项目名称: 2.项目数据: chile parentTom LucyTom JackJone LucyJone JackLucy MaryLucy Ben ...
MapReduce编程系列 — 3：数据去重
1.项目名称: 2.程序代码: package com.dedup; import java.io.IOException; import org.apache.hadoop.conf.Configu ...
MapReduce编程系列 — 2：计算平均分
1.项目名称: 2.程序代码: package com.averagescorecount; import java.io.IOException; import java.util.Iterator ...
MapReduce编程系列 — 1：计算单词
1.代码: package com.mrdemo; import java.io.IOException; import java.util.StringTokenizer; import org.a ...
【原创】MapReduce编程系列之表连接
问题描述需要连接的表如下:其中左边是child,右边是parent,我们要做的是找出grandchild和grandparent的对应关系,为此需要进行表的连接. Tom Lucy Tom Jim ...
MapReduce 编程系列九 Reducer数目
本篇介绍怎样控制reduce的数目.前面观察结果文件,都会发现通常是以part-r-00000 形式出现多个文件,事实上这个reducer的数目有关系.reducer数目多,结果文件数目就多. 在初始 ...

随机推荐

Poj 2583 Series Determination
1.Link: http://poj.org/problem?id=2583 2.Content: Series Determination Time Limit: 1000MS Memory L ...
[java学习笔记]java语言核心----面向对象之static关键字
static关键字用处用于修饰成员变量和成员函数被修饰后的成员具有以下特点: 随着类的加载而加载优先于对象存在被所有对象所共享可以直接被类名调用使用注意静态方法只能访问静态成员:非静态方 ...
nginx 配置文件
#基于ip设置 server{ listen 80; server_name 192.168.116.129; location /{ root /usr/etc/ngin/html/ip; inde ...
Unity学习笔记(2):注册映射
在上一篇文章中(认识Unity)中概要介绍了Unity和Ioc,本节主要介绍IoC中的注册映射,并使用代码和配置文件两种方式进行说明. 定义依赖注入相关信息定义ILogger接口 public in ...
HTML中href的链接刷新页面问题
在上一篇随笔中说到了html()方法不能一直改变标签的值的问题,当单击完成时,回调函数返回的值瞬间就没有了,今天突然想到了,我单击的是链接啊,就算链接到本界面上,也要进行刷新,页面一刷新,显示的值自然 ...
C语言控制语句总结（if else for switch while break continue)
一.if语句 1表达式 if(条件表达式) 语句注: (1)条件表达式,一般为逻辑表达式或关系表达式,但也可以是任何数值类型,如整型.实型.字符型.指针型数据等. (2)语句,由于是C语言的语句,而 ...
WPF xmal绑定数据，当显示数据过长用省略号代替的方法
有时候会遇到这种情况,用了数据绑定显示的数据太长时,如何让过长的数据显示规定的长度,多余的用省略号代替呢,自己写了个简单的小例子和大家分享一下^_^,我也是学习WPF不久,这是我第一次写博客,有问题还 ...
【发问】代表ODBC、Ibatis 发问 Hibernate、Linq、Entity、JPA
分页: 多表关联查询: 多表操作临时表: 存储过程式长语句 : Hibernate与iBATIS的比较博客分类: db iBATISHibernateSQL数据结构ORM 1.出身 hibern ...
制作复选框(Toggle)
怎样判断是否应当使用复选框复选框,就是对一个选项做上一个标记,表示这个选项已经被选中了.在游戏中,复选框一般用来做一些选项的控制,这种选项一般都只有两种答案:是和否.例如,单击一下开启音乐的复选框, ...
mybatis的parameterType使用map实现真正的sql随意写
在dao层给map赋值纠正一下应该把dd作为传入而不是sbiId; sqlMap中的parameterType="java.util.Map", 就 OK package com ...

MapReduce编程系列 — 4：排序

MapReduce编程系列 — 4：排序的更多相关文章

随机推荐

热门专题