hadoop map（分片）数量确定

之前学习hadoop的时候，一直希望可以调试hadoop源码，可是一直没找到有效的方法，今天在调试矩阵乘法的时候发现了调试的方法，所以在这里记录下来。

1）事情的起因是想在一个Job里设置map的数量(虽然最终的map数量是由分片决定的)，在hadoop1.2.1之前，设置方法是：

job.setNumMapTasks()

不过，hadoop1.2.1没有了这个方法，只保留了设置reduce数量的方法。继续搜索资料，发现有同学提供了另外一种方法，就是使用configuration设置，设置方式如下：

conf.set("mapred.map.tasks",5);//设置5个map

按照上述方法设置之后，还是没有什么效果,控制分片数量的代码如下():

goalSize=totalSize/(numSplits==0？1：numSplits)
//totalSize是输入数据文件的大小，numSplits是用户设置的map数量，就是按照用户自己
//的意愿，每个分片的大小应该是goalSize
minSize=Math.max(job.getLong("mapred.min.split.size",1),minSplitSize)
//hadoop1.2.1中mapred-default.xml文件中mapred.min.split.size=0，所以job.getLong("mapred.min.split.size",1)=0，而minSplitSize是InputSplit中的一个数据成员，在File//Split中值为1.所以minSize=1，其目的就是得到配置中的最小值。
splitSize=Math.max(minSize,Math.min(goalSize,blockSize))
//真正的分片大小就是取按照用户设置的map数量计算出的goalSize和块大小blockSize中最小值(这是为了是分片不会大于一个块大小，有利于本地化计算)，并且又比minSize大的值。

其实，这是hadoop1.2.1之前的生成分片的方式，所以即使设置了map数量也不会有什么实际效果。

2）新版API(hadoop1.2.1)中计算分片的代码如下所示：

  public List<InputSplit> getSplits(JobContext job) throws IOException {
         long minSize = Math.max(this.getFormatMinSplitSize(), getMinSplitSize(job));
         long maxSize = getMaxSplitSize(job);
         ArrayList splits = new ArrayList();
         List files = this.listStatus(job);
         Iterator i$ = files.iterator();

         while(true) {
             while(i$.hasNext()) {
                 FileStatus file = (FileStatus)i$.next();
                 Path path = file.getPath();
                 FileSystem fs = path.getFileSystem(job.getConfiguration());
                 long length = file.getLen();
                 BlockLocation[] blkLocations = fs.getFileBlockLocations(file, 0L, length);
                 if(length != 0L && this.isSplitable(job, path)) {
                     long blockSize = file.getBlockSize();
                     long splitSize = this.computeSplitSize(blockSize, minSize, maxSize);

                     long bytesRemaining;
                     for(bytesRemaining = length; (double)bytesRemaining / (double)splitSize > 1.1D; bytesRemaining -= splitSize) {
                         int blkIndex = this.getBlockIndex(blkLocations, length - bytesRemaining);
                         splits.add(new FileSplit(path, length - bytesRemaining, splitSize, blkLocations[blkIndex].getHosts()));
                     }

                     if(bytesRemaining != 0L) {
                         splits.add(new FileSplit(path, length - bytesRemaining, bytesRemaining, blkLocations[blkLocations.length - 1].getHosts()));
                     }
                 } else if(length != 0L) {
                     splits.add(new FileSplit(path, 0L, length, blkLocations[0].getHosts()));
                 } else {
                     splits.add(new FileSplit(path, 0L, length, new String[0]));
                 }
             }

             job.getConfiguration().setLong("mapreduce.input.num.files", (long)files.size());
             LOG.debug("Total # of splits: " + splits.size());
             return splits;
         }
     }

第17行使用computeSplitSize(blockSize,minSize,maxsize)计算分片大小。

a.minSize通过以下方式计算：

 long minSize = Math.max(this.getFormatMinSplitSize(), getMinSplitSize(job))

而getFormatMinSplitSize()：

protected long getFormatMinSplitSize() {
        return 1L;
    }

而getMinSplitSize(job):

 public static long getMinSplitSize(JobContext job) {
        return job.getConfiguration().getLong("mapred.min.split.size", 1L);
    }

没有设置“mapred.min.split.size”的默认值是0。

所以，不设置“mapred.min.split.size”的话，就使用方法的默认值1代替，而“mapred.min.split.size”的默认值是0，所以minSize的值就是1

b.再看maxSize的计算方式：

   long maxSize = getMaxSplitSize(job);

而getMaxSplitSize():

public static long getMaxSplitSize(JobContext context) {
        return context.getConfiguration().getLong("mapred.max.split.size", 9223372036854775807L);
    }

没有设置"mapred.max.split.size"的话，就使用方法的默认值 9223372036854775807，而"mapred.max.split.size"并没有默认值，所以maxSize= 9223372036854775807；

c.我们已经能够计算出minSize=1，maxSize= 9223372036854775807，接下来计算分片大小：

 long splitSize = this.computeSplitSize(blockSize, minSize, maxSize);

protected long computeSplitSize(long blockSize, long minSize, long maxSize) {
        return Math.max(minSize, Math.min(maxSize, blockSize));
    }

显然，分片大小是就是maxSize和blockSize的较小值(minSize=1),那么我们就可以通过设置"mapred.max.split.size"来控制map的数量，只要设置值比物理块小就可以了。使用configuration对象的设置方法如下：

conf.set("mapred.max.split.size",2000000)//单位是字节，物理块是16M

3）可以设置map数量的矩阵乘法代码如下所示：

 /**
  * Created with IntelliJ IDEA.
  * User: hadoop
  * Date: 16-3-14
  * Time: 下午3:13
  * To change this template use File | Settings | File Templates.
  */
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileSystem;
 import java.io.IOException;
 import java.net.URI;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.io.*;
 import org.apache.hadoop.io.DoubleWritable;
 import org.apache.hadoop.io.Writable;
 import org.apache.hadoop.mapreduce.InputSplit;
 import org.apache.hadoop.mapreduce.Job;
 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
 import org.apache.hadoop.mapreduce.lib.input.FileSplit;
 import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
 import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
 import org.apache.hadoop.mapreduce.Reducer;
 import org.apache.hadoop.mapreduce.Mapper;
 import org.apache.hadoop.filecache.DistributedCache;
 import org.apache.hadoop.util.ReflectionUtils;

 public class MutiDoubleInputMatrixProduct {

     public static void initDoubleArrayWritable(int length,DoubleWritable[] doubleArrayWritable){
         for (int i=0;i<length;++i){
             doubleArrayWritable[i]=new DoubleWritable(0.0);
         }
     }

     public static  class MyMapper extends Mapper<IntWritable,DoubleArrayWritable,IntWritable,DoubleArrayWritable>{
         public DoubleArrayWritable map_value=new DoubleArrayWritable();
         public  double[][] leftMatrix=null;/******************************************/
         //public Object obValue=null;
         public DoubleWritable[] arraySum=null;
         public DoubleWritable[] tempColumnArrayDoubleWritable=null;
         public DoubleWritable[] tempRowArrayDoubleWritable=null;
         public double sum=0;
         public double uValue;
         public int leftMatrixRowNum;
         public int leftMatrixColumnNum;
         public void setup(Context context) throws IOException {
             Configuration conf=context.getConfiguration();
             leftMatrixRowNum=conf.getInt("leftMatrixRowNum",10);
             leftMatrixColumnNum=conf.getInt("leftMatrixColumnNum",10);
             leftMatrix=new double[leftMatrixRowNum][leftMatrixColumnNum];
             uValue=(double)(context.getConfiguration().getFloat("u",1.0f));
             tempRowArrayDoubleWritable=new DoubleWritable[leftMatrixColumnNum];
             initDoubleArrayWritable(leftMatrixColumnNum,tempRowArrayDoubleWritable);
             tempColumnArrayDoubleWritable=new DoubleWritable[leftMatrixRowNum];
             initDoubleArrayWritable(leftMatrixRowNum,tempColumnArrayDoubleWritable);
             System.out.println("map setup() start!");
             //URI[] cacheFiles=DistributedCache.getCacheFiles(context.getConfiguration());
             Path[] cacheFiles=DistributedCache.getLocalCacheFiles(conf);
             String localCacheFile="file://"+cacheFiles[0].toString();
             //URI[] cacheFiles=DistributedCache.getCacheFiles(conf);
             //DistributedCache.
             System.out.println("local path is:"+cacheFiles[0].toString());
             // URI[] cacheFiles=DistributedCache.getCacheFiles(context.getConfiguration());
             FileSystem fs =FileSystem.get(URI.create(localCacheFile), conf);
             SequenceFile.Reader reader=null;
             reader=new SequenceFile.Reader(fs,new Path(localCacheFile),conf);
             IntWritable key= (IntWritable)ReflectionUtils.newInstance(reader.getKeyClass(),conf);
             DoubleArrayWritable value= (DoubleArrayWritable)ReflectionUtils.newInstance(reader.getValueClass(),conf);
             //int valueLength=0;
             int rowIndex=0;
             int index;
             while (reader.next(key,value)){
                 index=-1;
                 for (Writable val:value.get()){ //ArrayWritable类的get方法返回Writable[]数组
                     tempRowArrayDoubleWritable[++index].set(((DoubleWritable)val).get());
                 }
                 //obValue=value.toArray();
                 rowIndex=key.get();
                 leftMatrix[rowIndex]=new double[leftMatrixColumnNum];
                 //this.leftMatrix=new double[valueLength][Integer.parseInt(context.getConfiguration().get("leftMatrixColumnNum"))];
                 for (int i=0;i<leftMatrixColumnNum;++i){
                     //leftMatrix[rowIndex][i]=Double.parseDouble(Array.get(obValue, i).toString());
                     //leftMatrix[rowIndex][i]=Array.getDouble(obValue, i);
                     leftMatrix[rowIndex][i]= tempRowArrayDoubleWritable[i].get();
                 }

             }
             arraySum=new DoubleWritable[leftMatrix.length];
             initDoubleArrayWritable(leftMatrix.length,arraySum);
         }
         public void map(IntWritable key,DoubleArrayWritable value,Context context) throws IOException, InterruptedException {
             //obValue=value.toArray();
             InputSplit inputSplit=context.getInputSplit();
             String fileName=((FileSplit)inputSplit).getPath().getName();
             if (fileName.startsWith("FB")) {
                 context.write(key,value);
             }
             else{
                 int ii=-1;
                 for(Writable val:value.get()){
                     tempColumnArrayDoubleWritable[++ii].set(((DoubleWritable)val).get());
                 }
                 //arraySum=new DoubleWritable[this.leftMatrix.length];
                 for (int i=0;i<this.leftMatrix.length;++i){
                     sum=0;
                     for (int j=0;j<this.leftMatrix[0].length;++j){
                         //sum+= this.leftMatrix[i][j]*Double.parseDouble(Array.get(obValue,j).toString())*(double)(context.getConfiguration().getFloat("u",1f));
                         //sum+= this.leftMatrix[i][j]*Array.getDouble(obValue,j)*uValue;
                         sum+= this.leftMatrix[i][j]*tempColumnArrayDoubleWritable[j].get()*uValue;
                     }
                     arraySum[i].set(sum);
                     //arraySum[i].set(sum);
                 }
                 map_value.set(arraySum);
                 context.write(key,map_value);
             }
         }
     }
     public static class MyReducer extends Reducer<IntWritable,DoubleArrayWritable,IntWritable,DoubleArrayWritable>{
         public DoubleWritable[] sum=null;
         // public Object obValue=null;
         public DoubleArrayWritable valueArrayWritable=new DoubleArrayWritable();
         public DoubleWritable[] tempColumnArrayDoubleWritable=null;
         private int leftMatrixRowNum;

         public void setup(Context context){
             //leftMatrixColumnNum=context.getConfiguration().getInt("leftMatrixColumnNum",100);
             leftMatrixRowNum=context.getConfiguration().getInt("leftMatrixRowNum",100);
             sum=new DoubleWritable[leftMatrixRowNum];
             initDoubleArrayWritable(leftMatrixRowNum,sum);
             //tempRowArrayDoubleWritable=new DoubleWritable[leftMatrixColumnNum];
             tempColumnArrayDoubleWritable=new DoubleWritable[leftMatrixRowNum];
             initDoubleArrayWritable(leftMatrixRowNum,tempColumnArrayDoubleWritable);
         }
         //如果矩阵的计算已经在map中完成了，貌似可以不使用reduce，如果不创建reduce类，MR框架仍然会调用一个默认的reduce，只是这个reduce什么也不做
         //但是，不使用reduce的话，map直接写文件，有多少个map就会产生多少个结果文件。这里使用reduce是为了将结果矩阵存储在一个文件中。
         public void reduce(IntWritable key,Iterable<DoubleArrayWritable>value,Context context) throws IOException, InterruptedException {
             //int valueLength=0;
             for(DoubleArrayWritable doubleValue:value){
                 int index=-1;
                 for (Writable val:doubleValue.get()){
                     tempColumnArrayDoubleWritable[++index].set(((DoubleWritable)val).get());
                 }
                 //valueLength=Array.getLength(obValue);
                 /*
                 for (int i=0;i<leftMatrixRowNum;++i){
                     //sum[i]=new DoubleWritable(Double.parseDouble(Array.get(obValue,i).toString())+sum[i].get());
                     //sum[i]=new DoubleWritable(Array.getDouble(obValue,i)+sum[i].get());
                     sum[i].set(tempColumnArrayDoubleWritable[i].get()+sum[i].get());
                 }
                 */
             }
             //valueArrayWritable.set(sum);
             valueArrayWritable.set(tempColumnArrayDoubleWritable);
             context.write(key,valueArrayWritable);
             /*
             for (int i=0;i<sum.length;++i){
                 sum[i].set(0.0);
             }
             */

         }
     }

     public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
         String uri=args[3];
         String outUri=args[4];
         String cachePath=args[2];
         HDFSOperator.deleteDir(outUri);
         Configuration conf=new Configuration();
         DistributedCache.addCacheFile(URI.create(cachePath),conf);//添加分布式缓存
         /**************************************************/
         //FileSystem fs=FileSystem.get(URI.create(uri),conf);
         //fs.delete(new Path(outUri),true);
         /*********************************************************/
         conf.setInt("leftMatrixColumnNum",Integer.parseInt(args[0]));
         conf.setInt("leftMatrixRowNum",Integer.parseInt(args[1]));
         conf.setFloat("u",1.0f);
         //conf.set("mapred.map.tasks",args[5]);
         //int mxSplitSize=Integer.valueOf(args[5])
         conf.set("mapred.max.split.size",args[5]);//hadoop1.2.1中并没有setNumMapTasks方法，只能通过这种方式控制计算分片的大小来控制map数量
         conf.set("mapred.jar","MutiDoubleInputMatrixProduct.jar");
         Job job=new Job(conf,"MatrixProdcut");
         job.setJarByClass(MutiDoubleInputMatrixProduct.class);
         job.setInputFormatClass(SequenceFileInputFormat.class);
         job.setOutputFormatClass(SequenceFileOutputFormat.class);
         job.setMapperClass(MyMapper.class);
         job.setReducerClass(MyReducer.class);
         job.setMapOutputKeyClass(IntWritable.class);
         job.setMapOutputValueClass(DoubleArrayWritable.class);
         job.setOutputKeyClass(IntWritable.class);
         job.setOutputValueClass(DoubleArrayWritable.class);
         FileInputFormat.setInputPaths(job, new Path(uri));
         FileOutputFormat.setOutputPath(job,new Path(outUri));
         System.exit(job.waitForCompletion(true)?0:1);
     }

 }
 class DoubleArrayWritable extends ArrayWritable {
     public DoubleArrayWritable(){
         super(DoubleWritable.class);
     }
 /*
     public String toString(){
         StringBuilder sb=new StringBuilder();
         for (Writable val:get()){
             DoubleWritable doubleWritable=(DoubleWritable)val;
             sb.append(doubleWritable.get());
             sb.append(",");
         }
         sb.deleteCharAt(sb.length()-1);
         return sb.toString();
     }
 */
 }

 class HDFSOperator{
     public static boolean deleteDir(String dir)throws IOException{
         Configuration conf=new Configuration();
         FileSystem fs =FileSystem.get(conf);
         boolean result=fs.delete(new Path(dir),true);
         System.out.println("sOutput delete");
         fs.close();
         return result;
     }
 }

4）接下来说说如何断点调试hadoop源码，这里以计算文件分片的源码为例来说明。

a.首先找到FileInputFormat类，这个类就在hadoop-core-1.2.1.jar中，我们需要将这个jar包添加到工程中，如下所示：

虽然这是编译之后的类文件，也就是字节码，但是仍然可以像java源码一样，断点调试，这里我们分别在getSplits()方法和computeSplitSize()方法中添加两个断点，然后使用IDEA在本地直接以Debug方式运行我们的MapReduce程序，结果如下所示：

命中断点，并且我们可以查看相关的变量值。