MapReduce架构

主从结构

　　主节点：JobTracker（一个）

　　从节点：TaskTrackers（多个）

JobTracker：

　　接收客户提交的计算任务

　　把计算任务分配给TaskTrackers执行

　　监控TaskTracker执行情况

TaskTrackers：

　　执行JobTracker分配的计算任务

---

MapReduce计算模型

　　在Hadoop中，每个MapReduce任务都被初始化为一个Job，每个Job分为两个阶段：Map、Reduce。这两个阶段分别用两个函数表示 ：Map、Reduce

　　Map函数接收一个<key,value>形式的输入，产生同样形式的中间输出。Hadoop将所有相同key的value集合到一起传递给Reduce函数

　　Reduce函数接收一个<key,(list of value)>形式的的呼入，然后对value集合进行处理输出结果。Reduce的输出也是<key,value>的形式

练习：

输入文本

姓名 分数

多个文本，内容行如上述，统计每个人的平均分

Map

 package org.zln.scorecount;

 import org.apache.hadoop.io.IntWritable;
 import org.apache.hadoop.io.LongWritable;
 import org.apache.hadoop.io.Text;
 import org.apache.hadoop.mapreduce.Mapper;

 import java.io.IOException;
 import java.util.StringTokenizer;

 /**
  * Created by sherry on 15-7-12.
  */
 public class ScoreMap extends Mapper<LongWritable,Text,Text,IntWritable> {

     @Override
     protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
         String line = value.toString();//将纯文本的数据转化为string
         StringTokenizer tokenizer = new StringTokenizer(line,"\n");//切割
         while (tokenizer.hasMoreTokens()){
             StringTokenizer tokenizerLine = new StringTokenizer(tokenizer.nextToken());
             String strName = tokenizerLine.nextToken();//姓名
             String strScore = tokenizerLine.nextToken();//成绩

             Text name = new Text(strName);
             int scoreInt = Integer.parseInt(strScore);
             context.write(name,new IntWritable(scoreInt));//输出姓名：成绩

         }
     }
 }

Reduce

 package org.zln.scorecount;

 import org.apache.hadoop.io.IntWritable;
 import org.apache.hadoop.io.Text;
 import org.apache.hadoop.mapreduce.Reducer;

 import java.io.IOException;
 import java.util.Iterator;

 /**
  * Created by sherry on 15-7-12.
  */
 public class ScoreReduce extends Reducer<Text,IntWritable,Text,IntWritable> {
     @Override
     protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
         int sum = 0;
         int count = 0;
         Iterator<IntWritable> intWritableIterator = values.iterator();
         while (intWritableIterator.hasNext()){
             sum += intWritableIterator.next().get();//总分
             count++;//平均分
         }
         int avg = sum/count;
         context.write(key,new IntWritable(avg));
     }
 }

Main

 package org.zln.scorecount;

 import org.apache.hadoop.conf.Configured;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.io.IntWritable;
 import org.apache.hadoop.io.Text;
 import org.apache.hadoop.mapreduce.Job;
 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
 import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
 import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
 import org.apache.hadoop.util.Tool;
 import org.apache.hadoop.util.ToolRunner;

 /**
  * Created by sherry on 15-7-12.
  */
 public class ScoreMain extends Configured implements Tool{
     public int run(String[] args) throws Exception {
         Job job = new Job(getConf());
         job.setJarByClass(ScoreMain.class);
         job.setJobName("ScoreCount");

         job.setOutputKeyClass(Text.class);
         job.setOutputValueClass(IntWritable.class);

         job.setMapperClass(ScoreMap.class);
         job.setReducerClass(ScoreReduce.class);

         job.setInputFormatClass(TextInputFormat.class);
         job.setOutputFormatClass(TextOutputFormat.class);

         FileInputFormat.setInputPaths(job, new Path(args[0]));
         FileOutputFormat.setOutputPath(job, new Path(args[1]));

         boolean success = job.waitForCompletion(true);
         return success?0:1;
     }

     //统计平均分
     public static void main(String[] args) throws Exception {
         int ret = ToolRunner.run(new ScoreMain(), args);
         System.exit(ret);
     }
 }

---

---

我们的Map与Reduce都继承了父类，并复写了map或reduce方法

父类中 还有 三个方法未作处理

setup：启动map/reduce后首先调用

cleanup：最后调用

run：每次调用的时候都会执行