Map Reduce Application(Join)

We are going to explain how join works in MR , we will focus on reduce side join and map side join.

Reduce Side Join

Assuming we have 2 datasets , one is user information(id, name...) , the other is comments made by users(user id, content, date...). We want to join the 2 datasets to select the username and comment they posted. So, this is a typical join example.  You can implement all types of join including innter join/outer join/full outer join... As the name indicates, the join is done in reducer.

• We use 2/n mappers for each dataset(table in RDBMS). So, we set this with code below.

   MultipleInputs.addInputPath(job,filePath,TextInputFormat.class,UserMapper.class)
   MultipleInputs.addInputPath(job,filePath,TextInputFormat.class,CommentsMapper.class)
  3 ....
  4 MultipleInputs.addInputPath(job,filePath,TextInputFormat.class,OtherMapper.class)
  ....

• In each mapper, we just need to output the key/value pairs as the job is most done in reducer.  In reduce function, when it iterators the values for a given key, reduce function needs to know the value is from which dataset to perform the join. Reducer itself may not be able to distinguish which value is from which mapper(UserMapper or CommentsMapper) for a given key. So, in the map function, we have a chance to mark the value like prefix the value with the mapper name  something like that.

   outkey.set(userId);
   //mark this value so reduce function knows
   outvalue.set("UserMapper"+value.toString);
   context.write(outkey,outvalue)

• In reducer, we get the join type from configuration, perform the join.  there can be multiple reducers and with multiple threads.

   public void setup(Context context){
       joinType = context.getConfiguration().get("joinType");
   }
   public void reduce(Text text, Iterable<Text> values, Context context)
           throws Exception {
           listUser.clear();
           listComments.clear();
           for (Text t: values){
               if(isFromUserMapper(t)){
                   listUser.add(realContent(t));
               }else if (isFromCommentsMapper(t)){
                   listUser.add(realContent(t));
               }
           }
           doJoin(context);
   }
   private void doJoin(Context context) throws Exception{
       if (joinType.equals("inner")){
           if(both are not empty){
               for (Text user:listUser){
                   for (Text comm: listComments){
                       context.write(user,comm);
                   }
               }
           }
       }else if (){
       }.....
   }

In reducer side join, all data will be sent to reducer side, so, the overall network bandwith is required.

Map Side Join/Replicated Join

As the name indicates , the join operation is done in map side . So, there is no reducer.   It is very suitable for join datasets which has only 1 large dataset and others are small dataset and can be read into small memory in a single machine. It is faster than reduce side join (as no reduce phase, no intermediate output, no network transfer)

We still use the sample example that is to join user(small) and comments(large) datasets. How to implement it?

• Set the number of reduce to 0.

job.setNumReduceTasks(0);

• Add the small datasets to hadoop distribute cache.The first one is deprecated.

 DistributedCache.addCacheFile(new Path(args[]).toUri(),job.getConfiguration)
 job.addCacheFile(new Path(filename).toUri());

• In mapper setup function, get the cache by code below. The first one is deprecated. Read the file and put the the key / value in an instance variable like HashMap. This is single thread, so it is safe.

Path[] localPaths = context.getLocalCacheFiles();
URI[] uris = context.getCacheFiles()

• In the mapper function, since, you have the entire user data set in the HashMap, you can try to get the key(comes from the split of comment dataset) from the HashMap. If it exists, you get a match. Because only one split of comments dataset goes into each mapper task, you can only perform an inner join or a left outer join.

What is Hadoop Distributed Cache?

"DistributedCache is a facility provided by the Map-Reduce framework to cache files needed by applications. Once you cache a file for your job, hadoop framework will make it available on(or broadcast to) each and every data nodes (in file system, not in memory) where you map/reduce tasks are running. Then you can access the cache file as local file in your Mapper Or Reducer job. Now you can easily read the cache file and populate some collection (e.g Array, Hashmap etc.) in your code"  The cache will be removed once the job is done as they are temporary files.

The size of the cache can be configured in mapred-site.xml.

How to use Distributed Cache(the API has changed)?

• Add cache in driver.
  Note the # sign in the URI. Before it, you specify the absolute data path in HDFS. After it, you set a name(symlink) to specify the local file path in your mapper/reducer.

     job.addCacheFile(new URI("/user/ricky/user.txt#user"));
     job.addCacheFile(new URI("/user/ricky/org.txt#org"));

     return job.waitForCompletion(true) ? 0 : 1;

• Read cache in your task(mapper/reduce), probably in setup function.

 @Override
 protected void setup(
         Mapper<LongWritable, Text, Text, Text>.Context context)
         throws IOException, InterruptedException {
     if (context.getCacheFiles() != null
             && context.getCacheFiles().length > 0) {

         File some_file = new File("user");
         File other_file = new File("org");
     }
     super.setup(context);
 }

Reference:

https://www.youtube.com/user/pramodnarayana/videos

https://stackoverflow.com/questions/19678412/number-of-mappers-and-reducers-what-it-means