LAB1 partIII

partIII

实现分发MapReduce 任务，实现 schedule() 方法在 mapreduce/schedule.go。

schedule()函数的职责是把任务分给可用的worker.

schedule()通过信道registerChan参数读取注册的worker.

worker与schedule()之间的顺序是没有确定的。

schedule()告诉worker 执行远程函数调用Worker.DoTask.

远程调用的参数 DoTaskArgs定义在 mapreduce/common_rpc.go

File 参数只有 mapTask 才用到.

rpc 使用call() function 定义在 mapreduce/common_rpc.go

测试命令

go test -run TestParallel

会执行两边，TestParallelBasic and TestParallelCheck;

master@master:~/study/6.824/src/mapreduce$ go test -run TestParallel

2019/03/22 14:34:40 rpc.Register: method "CleanupFiles" has 1 input parameters; needs exactly three

2019/03/22 14:34:40 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/22 14:34:40 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

2019/03/22 14:34:40 rpc.Register: method "Wait" has 1 input parameters; needs exactly three

2019/03/22 14:34:40 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/22 14:34:40 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

/var/tmp/824-1000/mr20412-master: Starting Map/Reduce task test

2019/03/22 14:34:40 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/22 14:34:40 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

Schedule: 20 mapPhase tasks (10 I/Os)

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #0 on file 824-mrinput-0.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #1 on file 824-mrinput-1.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker1: mapPhase task #1 done

/var/tmp/824-1000/mr20412-worker0: mapPhase task #0 done

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #2 on file 824-mrinput-2.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #3 on file 824-mrinput-3.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #3 done

/var/tmp/824-1000/mr20412-worker1: mapPhase task #2 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #4 on file 824-mrinput-4.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #5 on file 824-mrinput-5.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #4 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #6 on file 824-mrinput-6.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker1: mapPhase task #5 done

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #7 on file 824-mrinput-7.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #6 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #8 on file 824-mrinput-8.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker1: mapPhase task #7 done

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #9 on file 824-mrinput-9.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #8 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #10 on file 824-mrinput-10.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker1: mapPhase task #9 done

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #11 on file 824-mrinput-11.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #10 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #12 on file 824-mrinput-12.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker1: mapPhase task #11 done

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #13 on file 824-mrinput-13.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #12 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #14 on file 824-mrinput-14.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker1: mapPhase task #13 done

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #15 on file 824-mrinput-15.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #14 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #16 on file 824-mrinput-16.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker1: mapPhase task #15 done

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #17 on file 824-mrinput-17.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #16 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #18 on file 824-mrinput-18.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker1: mapPhase task #17 done

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #19 on file 824-mrinput-19.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #18 done

/var/tmp/824-1000/mr20412-worker1: mapPhase task #19 done

Schedule: mapPhase done

Schedule: 10 reducePhase tasks (20 I/Os)

/var/tmp/824-1000/mr20412-worker1: given reducePhase task #1 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: given reducePhase task #0 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker1: reducePhase task #1 done

/var/tmp/824-1000/mr20412-worker1: given reducePhase task #2 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: reducePhase task #0 done

/var/tmp/824-1000/mr20412-worker0: given reducePhase task #3 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker1: reducePhase task #2 done

/var/tmp/824-1000/mr20412-worker1: given reducePhase task #4 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: reducePhase task #3 done

/var/tmp/824-1000/mr20412-worker0: given reducePhase task #5 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker1: reducePhase task #4 done

/var/tmp/824-1000/mr20412-worker1: given reducePhase task #6 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: reducePhase task #5 done

/var/tmp/824-1000/mr20412-worker0: given reducePhase task #7 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker1: reducePhase task #6 done

/var/tmp/824-1000/mr20412-worker1: given reducePhase task #8 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: reducePhase task #7 done

/var/tmp/824-1000/mr20412-worker0: given reducePhase task #9 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker1: reducePhase task #8 done

/var/tmp/824-1000/mr20412-worker0: reducePhase task #9 done

Schedule: reducePhase done

Merge: read mrtmp.test-res-0

Merge: read mrtmp.test-res-1

Merge: read mrtmp.test-res-2

Merge: read mrtmp.test-res-3

Merge: read mrtmp.test-res-4

Merge: read mrtmp.test-res-5

Merge: read mrtmp.test-res-6

Merge: read mrtmp.test-res-7

Merge: read mrtmp.test-res-8

Merge: read mrtmp.test-res-9

/var/tmp/824-1000/mr20412-master: Map/Reduce task completed

2019/03/22 14:34:41 rpc.Register: method "CleanupFiles" has 1 input parameters; needs exactly three

2019/03/22 14:34:41 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/22 14:34:41 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

2019/03/22 14:34:41 rpc.Register: method "Wait" has 1 input parameters; needs exactly three

2019/03/22 14:34:41 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/22 14:34:41 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

/var/tmp/824-1000/mr20412-master: Starting Map/Reduce task test

Schedule: 20 mapPhase tasks (10 I/Os)

2019/03/22 14:34:41 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/22 14:34:41 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

/var/tmp/824-1000/mr20412-worker1: given mapPhase task #0 on file 824-mrinput-0.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #1 on file 824-mrinput-1.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #1 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #2 on file 824-mrinput-2.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #2 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #3 on file 824-mrinput-3.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #3 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #4 on file 824-mrinput-4.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #4 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #5 on file 824-mrinput-5.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #5 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #6 on file 824-mrinput-6.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #6 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #7 on file 824-mrinput-7.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #7 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #8 on file 824-mrinput-8.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #8 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #9 on file 824-mrinput-9.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #9 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #10 on file 824-mrinput-10.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #10 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #11 on file 824-mrinput-11.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #11 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #12 on file 824-mrinput-12.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #12 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #13 on file 824-mrinput-13.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #13 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #14 on file 824-mrinput-14.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #14 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #15 on file 824-mrinput-15.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #15 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #16 on file 824-mrinput-16.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #16 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #17 on file 824-mrinput-17.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #17 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #18 on file 824-mrinput-18.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #18 done

/var/tmp/824-1000/mr20412-worker0: given mapPhase task #19 on file 824-mrinput-19.txt (nios: 10)

/var/tmp/824-1000/mr20412-worker0: mapPhase task #19 done

/var/tmp/824-1000/mr20412-worker1: mapPhase task #0 done

Schedule: mapPhase done

Schedule: 10 reducePhase tasks (20 I/Os)

/var/tmp/824-1000/mr20412-worker1: given reducePhase task #1 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: given reducePhase task #0 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: reducePhase task #0 done

/var/tmp/824-1000/mr20412-worker1: reducePhase task #1 done

/var/tmp/824-1000/mr20412-worker0: given reducePhase task #2 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker1: given reducePhase task #3 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: reducePhase task #2 done

/var/tmp/824-1000/mr20412-worker0: given reducePhase task #4 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker1: reducePhase task #3 done

/var/tmp/824-1000/mr20412-worker1: given reducePhase task #5 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: reducePhase task #4 done

/var/tmp/824-1000/mr20412-worker0: given reducePhase task #6 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker1: reducePhase task #5 done

/var/tmp/824-1000/mr20412-worker1: given reducePhase task #7 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: reducePhase task #6 done

/var/tmp/824-1000/mr20412-worker0: given reducePhase task #8 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker1: reducePhase task #7 done

/var/tmp/824-1000/mr20412-worker1: given reducePhase task #9 on file (nios: 20)

/var/tmp/824-1000/mr20412-worker0: reducePhase task #8 done

/var/tmp/824-1000/mr20412-worker1: reducePhase task #9 done

Schedule: reducePhase done

Merge: read mrtmp.test-res-0

Merge: read mrtmp.test-res-1

Merge: read mrtmp.test-res-2

Merge: read mrtmp.test-res-3

Merge: read mrtmp.test-res-4

Merge: read mrtmp.test-res-5

Merge: read mrtmp.test-res-6

Merge: read mrtmp.test-res-7

Merge: read mrtmp.test-res-8

Merge: read mrtmp.test-res-9

/var/tmp/824-1000/mr20412-master: Map/Reduce task completed

PASS

ok _/home/master/study/6.824/src/mapreduce 2.053s

schedule.go

package mapreduce

import (

   "fmt"

   "sync"

   "strings"

   "time"

)

//

// schedule() starts and waits for all tasks in the given phase (mapPhase

// or reducePhase). the mapFiles argument holds the names of the files that

// are the inputs to the map phase, one per map task. nReduce is the

// number of reduce tasks. the registerChan argument yields a stream

// of registered workers; each item is the worker's RPC address,

// suitable for passing to call(). registerChan will yield all

// existing registered workers (if any) and new ones as they register.

//

func schedule(jobName string, mapFiles []string, nReduce int, phase jobPhase, registerChan chan string) {

   var ntasks int

   var n_other int // number of inputs (for reduce) or outputs (for map)

   switch phase {

   case mapPhase:

      ntasks = len(mapFiles)

      n_other = nReduce

   case reducePhase:

      ntasks = nReduce

      n_other = len(mapFiles)

   }

   fmt.Printf("Schedule: %v %v tasks (%d I/Os)\n", ntasks, phase, n_other)

   // All ntasks tasks have to be scheduled on workers. Once all tasks

   // have completed successfully, schedule() should return.

   //

   // Your code here (Part III, Part IV).

   //

   usableWorkers := make(chan string)

   waitGroup := new(sync.WaitGroup)

   i :=

   for  {

      var work string;

      select {

         case work  =<- usableWorkers:

            //fmt.Printf("usable worker: %v \n", work)

         case work =<- registerChan:

            //fmt.Printf("usable worker: %v \n", work)

      //default:

         //fmt.Printf("not usable worker wait for next loop ...\n")

      }

      if  strings.TrimSpace(work) != "" {

         waitGroup.Add()

         go func(index int) {

            rqArs := new(DoTaskArgs)

            rqArs.JobName=jobName

            if  phase == mapPhase {

               rqArs.File  = mapFiles[index]    // only for map, the input file

            }

            rqArs.Phase=phase   // are we in mapPhase or reducePhase?

            rqArs.TaskNumber=index      // this task's index in the current phase

            // NumOtherPhase is the total number of tasks in other phase; mappers

            // need this to compute the number of output bins, and reducers needs

            // this to know how many input files to collect.

            rqArs.NumOtherPhase=n_other

                defer func(){

                   waitGroup.Done()

                   usableWorkers <- work

                   }()

            call(work, "Worker.DoTask",

               rqArs, nil)

         }(i)

         i++

      } else {

         time.After(time.Second)

      }

      if i == ntasks {

         break

      }

   }

   waitGroup.Wait()

   fmt.Printf("Schedule: %v done\n", phase)

}

LAB1 partIII的更多相关文章

6.828 lab1 bootload
MIT6.828 lab1地址:http://pdos.csail.mit.edu/6.828/2014/labs/lab1/ 第一个练习,主要是让我们熟悉汇编,嗯,没什么好说的. Part 1: P ...
Machine Learning #Lab1# Linear Regression
Machine Learning Lab1 打算把Andrew Ng教授的#Machine Learning#相关的6个实验一一实现了贴出来- 预计时间长度战线会拉的比較长(毕竟JOS的7级浮屠还没搞 ...
ucore lab1 bootloader学习笔记
---恢复内容开始--- 开机流程回忆以Intel 80386为例,计算机加电后,CPU从物理地址0xFFFFFFF0(由初始化的CS:EIP确定,此时CS和IP的值分别是0xF000和0xFFF0 ...
LAB1 partV
partV 创建文档反向索引.word -> document 与前面做的单词统计类似,这个是单词与文档位置的映射关系. mapF 文档解析相同,返回信息不同而已. reduceF 返回归约 ...
6.824 LAB1 环境搭建
MIT 6.824 LAB1 环境搭建 vmware 虚拟机 linux ubuntu server 安装 go 官方安装步骤: 下载此压缩包并提取到 /usr/local 目录,在 /usr/l ...
软件测试：lab1.Junit and Eclemma
软件测试:lab1.Junit and Eclemma Task: Install Junit(4.12), Hamcrest(1.3) with Eclipse Install Eclemma wi ...
MIT 6.824 lab1:mapreduce
这是 MIT 6.824 课程 lab1 的学习总结,记录我在学习过程中的收获和踩的坑. 我的实验环境是 windows 10,所以对lab的code 做了一些环境上的修改,如果你仅仅对code 感兴 ...
清华大学OS操作系统实验lab1练习知识点汇总
lab1知识点汇总还是有很多问题,但是我觉得我需要在查看更多资料后回来再理解,学这个也学了一周了,看了大量的资料...还是它们自己的80386手册和lab的指导手册觉得最准确,现在我就把这部分知识做 ...
JOS lab1 part2 分析
lab1的Exercise 2就是让我们熟悉gdb的si操作,并知道BIOS的几条指令在做什么就够了,所以我们也会尽可能的去分析每一行代码. 首先进入到6.8282/lab这个目录下,输入指令make ...

随机推荐

python写mapReduce初步
最近在学了python了,从mapReduce开始 ,话不多说了,直接上代码了哈 map阶段,map.py文件 import sys # 标准输入 # 在终端的话,就需要这样了 cat a.txt | ...
JAVA高级-面试题总结
最近面试了一些公司,针对面试中遇到的问题在此记录,提升自己,造福大家一.java源码相关 ArrayList创建和add等各种api使用原理 HashMap 的创建,put原理,和HashTable ...
L2-002 链表去重（25 分)
L2-002 链表去重 (25 分) 给定一个带整数键值的链表 L,你需要把其中绝对值重复的键值结点删掉.即对每个键值 K,只有第一个绝对值等于 K 的结点被保留.同时,所有被删除的结点须被保存在 ...
pycharm介绍
PyCharm是一种Python IDE(集成开发软件),带有一整套可以帮助用户在使用Python语言开发时提高其效率的工具. 1.1 PyCharm主界面 PyCharm包含4个主要区分,分别为 ...
java Boolean和boolean的区别
Boolean b1=new Boolean("false"); Boolean b2=new Boolean("tRue"); Boolean b3=new ...
41.找出所有和为S的连续正数序列
小明很喜欢数学,有一天他在做数学作业时,要求计算出9~16的和, 他马上就写出了正确答案是100.但是他并不满足于此,他在想究竟有多少种连续的正数序列的和为100(至少包括两个数). 没多久,他就得到 ...
python类的语法和底层实现
语法: class 类名: name = “egon” # 类属性 def __init__(self): self.age = 18 # 对象属性 self.__sex = "fe ...
服务器A制定计划任务，BAT脚本自动备份oracle数据文件，拷贝至服务器B的共享目录。
运行环境:windows server 2008 R2 目的:在数据库服务器A进行数据库自动备份,并且保留5天. 为了安全,需要在web应用服务器B进行数据库的冗余备份,建立双保险.(保留15天) A ...
MySQL命令行登陆，远程登陆MySQL 的方法
https://www.cnblogs.com/lvk618/p/3522321.html 1.MySQL自带工具的存放路径: D:\Program Files\MySQL\MySQL Server ...
python函数之可迭代对象、迭代器的判断
怎么判断一个对象是可迭代对象还是迭代器例子 from collections import Iterable, Iterator lst = ['Today is Wednesday', 'Tomo ...

LAB1 partIII

LAB1 partIII的更多相关文章

随机推荐

热门专题