LAB1 partIV

PartIV

实现处理worker 失败情况。

worker 处理失败，master 应该重新分配该任务给其他的worker 处理。

rpc 失败情况复杂，可能worker 结果回应丢失了，也有可能 worker还在处理，但是

master rpc 超时了。考虑两个worker都拿到相同的task 且都有结果文件的情况，

要确保输出任务的原子性。当前这个程序框架无法保证任务输出的原子性,只考虑失败的worker没任何输出。

测试命令

$ go test -run Failure

master@master:~/study/6.824/src/mapreduce$

master@master:~/study/6.824/src/mapreduce$ go test -run Failure

2019/03/24 08:59:25 rpc.Register: method "CleanupFiles" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Wait" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

/var/tmp/824-1000/mr2499-master: Starting Map/Reduce task test

2019/03/24 08:59:25 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

Schedule: 20 mapPhase tasks (10 I/Os)

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #0 on file 824-mrinput-0.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #1 on file 824-mrinput-1.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #1 done

/var/tmp/824-1000/mr2499-worker1: mapPhase task #0 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #3 on file 824-mrinput-3.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #2 on file 824-mrinput-2.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #2 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #4 on file 824-mrinput-4.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #3 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #5 on file 824-mrinput-5.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #5 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #6 on file 824-mrinput-6.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #4 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #7 on file 824-mrinput-7.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #6 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #8 on file 824-mrinput-8.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #7 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #9 on file 824-mrinput-9.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #9 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #10 on file 824-mrinput-10.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #8 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #11 on file 824-mrinput-11.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #10 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #12 on file 824-mrinput-12.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #11 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #13 on file 824-mrinput-13.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #12 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #14 on file 824-mrinput-14.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #13 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #15 on file 824-mrinput-15.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #14 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #16 on file 824-mrinput-16.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #15 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #17 on file 824-mrinput-17.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #16 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #18 on file 824-mrinput-18.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #17 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #19 on file 824-mrinput-19.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #18 done

/var/tmp/824-1000/mr2499-worker1: mapPhase task #19 done

Schedule: mapPhase done

Schedule: 10 reducePhase tasks (20 I/Os)

/var/tmp/824-1000/mr2499-worker1: given reducePhase task #1 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker1: reducePhase task #1 done

/var/tmp/824-1000/mr2499-worker1: given reducePhase task #2 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker1: reducePhase task #2 done

/var/tmp/824-1000/mr2499-worker1: given reducePhase task #3 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker1: reducePhase task #3 done

/var/tmp/824-1000/mr2499-worker1: given reducePhase task #4 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker1: reducePhase task #4 done

/var/tmp/824-1000/mr2499-worker1: given reducePhase task #5 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker1: reducePhase task #5 done

/var/tmp/824-1000/mr2499-worker1: given reducePhase task #6 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker1: reducePhase task #6 done

/var/tmp/824-1000/mr2499-worker1: given reducePhase task #7 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker1: reducePhase task #7 done

/var/tmp/824-1000/mr2499-worker1: given reducePhase task #8 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker1: reducePhase task #8 done

/var/tmp/824-1000/mr2499-worker1: given reducePhase task #9 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker1: reducePhase task #9 done

/var/tmp/824-1000/mr2499-worker1: given reducePhase task #0 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker1: reducePhase task #0 done

Schedule: reducePhase done

Master: RPC /var/tmp/824-1000/mr2499-worker0 shutdown error

Merge: read mrtmp.test-res-0

Merge: read mrtmp.test-res-1

Merge: read mrtmp.test-res-2

Merge: read mrtmp.test-res-3

Merge: read mrtmp.test-res-4

Merge: read mrtmp.test-res-5

Merge: read mrtmp.test-res-6

Merge: read mrtmp.test-res-7

Merge: read mrtmp.test-res-8

Merge: read mrtmp.test-res-9

/var/tmp/824-1000/mr2499-master: Map/Reduce task completed

2019/03/24 08:59:25 rpc.Register: method "CleanupFiles" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Wait" has 1 input parameters; needs exactly three

/var/tmp/824-1000/mr2499-master: Starting Map/Reduce task test

Schedule: 20 mapPhase tasks (10 I/Os)

2019/03/24 08:59:25 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/24 08:59:25 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #0 on file 824-mrinput-0.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #1 on file 824-mrinput-1.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #0 done

/var/tmp/824-1000/mr2499-worker0: mapPhase task #1 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #2 on file 824-mrinput-2.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #3 on file 824-mrinput-3.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #2 done

/var/tmp/824-1000/mr2499-worker0: mapPhase task #3 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #4 on file 824-mrinput-4.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #5 on file 824-mrinput-5.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #4 done

/var/tmp/824-1000/mr2499-worker0: mapPhase task #5 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #6 on file 824-mrinput-6.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #7 on file 824-mrinput-7.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #7 done

/var/tmp/824-1000/mr2499-worker0: mapPhase task #6 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #8 on file 824-mrinput-8.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #9 on file 824-mrinput-9.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #8 done

/var/tmp/824-1000/mr2499-worker0: mapPhase task #9 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #10 on file 824-mrinput-10.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #11 on file 824-mrinput-11.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #10 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #12 on file 824-mrinput-12.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #11 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #13 on file 824-mrinput-13.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #12 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #14 on file 824-mrinput-14.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #13 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #15 on file 824-mrinput-15.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #14 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #16 on file 824-mrinput-16.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #15 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #17 on file 824-mrinput-17.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #16 done

/var/tmp/824-1000/mr2499-worker1: given mapPhase task #18 on file 824-mrinput-18.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker0: mapPhase task #17 done

/var/tmp/824-1000/mr2499-worker0: given mapPhase task #19 on file 824-mrinput-19.txt (nios: 10)

/var/tmp/824-1000/mr2499-worker1: mapPhase task #18 done

/var/tmp/824-1000/mr2499-worker0: mapPhase task #19 done

Schedule: mapPhase done

Schedule: 10 reducePhase tasks (20 I/Os)

2019/03/24 08:59:26 rpc.Register: method "Lock" has 1 input parameters; needs exactly three

2019/03/24 08:59:26 rpc.Register: method "Unlock" has 1 input parameters; needs exactly three

/var/tmp/824-1000/mr2499-worker3: given reducePhase task #2 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker2: given reducePhase task #3 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker3: reducePhase task #2 done

/var/tmp/824-1000/mr2499-worker2: reducePhase task #3 done

/var/tmp/824-1000/mr2499-worker3: given reducePhase task #5 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker2: given reducePhase task #4 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker3: reducePhase task #5 done

/var/tmp/824-1000/mr2499-worker3: given reducePhase task #6 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker2: reducePhase task #4 done

/var/tmp/824-1000/mr2499-worker2: given reducePhase task #7 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker3: reducePhase task #6 done

/var/tmp/824-1000/mr2499-worker3: given reducePhase task #8 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker2: reducePhase task #7 done

/var/tmp/824-1000/mr2499-worker2: given reducePhase task #9 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker3: reducePhase task #8 done

/var/tmp/824-1000/mr2499-worker3: given reducePhase task #0 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker2: reducePhase task #9 done

/var/tmp/824-1000/mr2499-worker2: given reducePhase task #1 on file (nios: 20)

/var/tmp/824-1000/mr2499-worker3: reducePhase task #0 done

/var/tmp/824-1000/mr2499-worker2: reducePhase task #1 done

Schedule: reducePhase done

Master: RPC /var/tmp/824-1000/mr2499-worker1 shutdown error

Master: RPC /var/tmp/824-1000/mr2499-worker0 shutdown error

Merge: read mrtmp.test-res-0

Merge: read mrtmp.test-res-1

Merge: read mrtmp.test-res-2

Merge: read mrtmp.test-res-3

Merge: read mrtmp.test-res-4

Merge: read mrtmp.test-res-5

Merge: read mrtmp.test-res-6

Merge: read mrtmp.test-res-7

Merge: read mrtmp.test-res-8

Merge: read mrtmp.test-res-9

/var/tmp/824-1000/mr2499-master: Map/Reduce task completed

PASS

ok _/home/master/study/6.824/src/mapreduce 2.845s

master@master:~/study/6.824/src/mapreduce$

schedule.go

package mapreduce

import (
   "fmt"
   "strings"
   "sync"
   "time"
)

//
// schedule() starts and waits for all tasks in the given phase (mapPhase
// or reducePhase). the mapFiles argument holds the names of the files that
// are the inputs to the map phase, one per map task. nReduce is the
// number of reduce tasks. the registerChan argument yields a stream
// of registered workers; each item is the worker's RPC address,
// suitable for passing to call(). registerChan will yield all
// existing registered workers (if any) and new ones as they register.
//
func schedule(jobName string, mapFiles []string, nReduce int, phase jobPhase, registerChan chan string) {
   var ntasks int
   var n_other int // number of inputs (for reduce) or outputs (for map)
   switch phase {
   case mapPhase:
      ntasks = len(mapFiles)
      n_other = nReduce
   case reducePhase:
      ntasks = nReduce
      n_other = len(mapFiles)
   }

   fmt.Printf("Schedule: %v %v tasks (%d I/Os)\n", ntasks, phase, n_other)

   // All ntasks tasks have to be scheduled on workers. Once all tasks
   // have completed successfully, schedule() should return.
   //
   // Your code here (Part III, Part IV).
   //

   //没有任务直接返回
   if ntasks <= 0 {
      return
   }
   //完成标志，应该取值为负数，不能与任务下标重叠
   finishFlag := -2
   //创建可工作worker队列
   usableWorkers := make(chan string)
   //创建任务队列
   waitTaskChan := make(chan int, ntasks)
   //初始化任务队列
   for k:=0; k<ntasks; k++ {
      waitTaskChan <- k
   }
   //创建任务完成计数跟踪线程通知主线程处理结果，主线线程堵塞获取待处理任务
   handleCount := sync.WaitGroup{}
   handleCount.Add(ntasks)
   go func() {
      handleCount.Wait()
      waitTaskChan <- finishFlag
   }()

   //获取待处理任务
   for taskIndex := range waitTaskChan {
      // 标志为处理所有的任务处理完成

      if taskIndex == finishFlag {
         break;
      }
      //两个信道怎么堵塞读取：worker怎么改为不为自旋的方式？
      for {
         var work string;
         //如果有待处理的任务，则获取可用的worker
         select {
         //应该先读新注册的worker.
         case work =<- registerChan:
         case work  =<- usableWorkers:
         }
         if  strings.TrimSpace(work) != "" {
            //创建处理任务线程
            go func(index int) {
               rqArs := new(DoTaskArgs)
               rqArs.JobName=jobName
               if  phase == mapPhase {
                  rqArs.File  = mapFiles[index]
               }
               rqArs.Phase=phase
               rqArs.TaskNumber=index
               rqArs.NumOtherPhase=n_other
               handleResult := false
               defer func(){
                  if handleResult {
                     //worker 处理成功，则完成计数减一，worker返回可用的队列
                     handleCount.Done()
                     usableWorkers <- work
                  } else {
                     //usableWorkers <- work //处理失败的 worker 不应该放回可用worker队列？
                     waitTaskChan <- index
                  }
               }()
               handleResult = call(work, "Worker.DoTask",
                  rqArs, nil)
            }(taskIndex)
            break
         } else {
            //释放cpu
            time.After(time.Second)
         }
      }
   }
   fmt.Printf("Schedule: %v done\n", phase)
}

LAB1 partIV的更多相关文章

6.828 lab1 bootload
MIT6.828 lab1地址:http://pdos.csail.mit.edu/6.828/2014/labs/lab1/ 第一个练习,主要是让我们熟悉汇编,嗯,没什么好说的. Part 1: P ...
Machine Learning #Lab1# Linear Regression
Machine Learning Lab1 打算把Andrew Ng教授的#Machine Learning#相关的6个实验一一实现了贴出来- 预计时间长度战线会拉的比較长(毕竟JOS的7级浮屠还没搞 ...
ucore lab1 bootloader学习笔记
---恢复内容开始--- 开机流程回忆以Intel 80386为例,计算机加电后,CPU从物理地址0xFFFFFFF0(由初始化的CS:EIP确定,此时CS和IP的值分别是0xF000和0xFFF0 ...
LAB1 partV
partV 创建文档反向索引.word -> document 与前面做的单词统计类似,这个是单词与文档位置的映射关系. mapF 文档解析相同,返回信息不同而已. reduceF 返回归约 ...
6.824 LAB1 环境搭建
MIT 6.824 LAB1 环境搭建 vmware 虚拟机 linux ubuntu server 安装 go 官方安装步骤: 下载此压缩包并提取到 /usr/local 目录,在 /usr/l ...
软件测试：lab1.Junit and Eclemma
软件测试:lab1.Junit and Eclemma Task: Install Junit(4.12), Hamcrest(1.3) with Eclipse Install Eclemma wi ...
MIT 6.824 lab1:mapreduce
这是 MIT 6.824 课程 lab1 的学习总结,记录我在学习过程中的收获和踩的坑. 我的实验环境是 windows 10,所以对lab的code 做了一些环境上的修改,如果你仅仅对code 感兴 ...
清华大学OS操作系统实验lab1练习知识点汇总
lab1知识点汇总还是有很多问题,但是我觉得我需要在查看更多资料后回来再理解,学这个也学了一周了,看了大量的资料...还是它们自己的80386手册和lab的指导手册觉得最准确,现在我就把这部分知识做 ...
JOS lab1 part2 分析
lab1的Exercise 2就是让我们熟悉gdb的si操作,并知道BIOS的几条指令在做什么就够了,所以我们也会尽可能的去分析每一行代码. 首先进入到6.8282/lab这个目录下,输入指令make ...

随机推荐

String与StringBuffer之间的转换
来源:http://www.oschina.net/code/snippet_2261089_47352 package demo; /* String与StringBuffer之间的转换 * Str ...
fabric一些资料网站，长期更新
nodejs的入门教程 http://nqdeng.github.io/7-days-nodejs/#1.2 http://blog.fens.me/series-nodejs/ fabric的知识博 ...
【Coucurrency-CountDownLatch】-20161203-0002
简介 java异步任务相关的工具.主要用在某些线程需要等到其他线程完成某些操作后才能执行的场景. 等待线程需要显示的调用wait方法,表示线程当前挂起,需要等到countdownLatch到0才执行. ...
RN集成echarts4图表组件react-native-secharts(转载)
一个webview封装的图表组件.基于百度echarts4,相比native-echarts有echarts自带对象支持,例如渐变色等,用法与官网相同用法. echarts version 4.2.0 ...
如何在运行时(Runtime)获得泛型的真正类型
前言由于Java 的类型擦除机制,在编译时泛型都被转为了Object,例如List<String>经过编译之后将变为类型 List.可以通过以下的方式再运行时获得泛型的真正类型泛型如何 ...
【Linux】comm指令
语法 comm [-123][--help][--version][第1个文件][第2个文件] 参数 -1 不显示只在第1个文件里出现过的列 -2 不显示只在第2个文件里出现过的列 -3 不显示只在第 ...
linux终端窗口字体缩放快捷键
环境:ubuntu16.04, 打开终端,有时候log输出一行显示不下 ‘ctrl’ + ‘-’字体缩小,一行显示更多的内容 ‘ctrl’ + ‘shift’ + ‘+’字体变大
http-server让你在任何目录下都可以创建web服务
在做前端页面开发,或者预览时,如果借助于Apache.Tomcat.nginx等预览页面,每次需要将所需预览的页面移动到对应的文件夹下,且还需要考虑是否删除相关目录原有的文件,显然比较麻烦. 那么有没 ...
vue+axios跨域解决方法
通过这种方法也可以解决跨域的问题. 使用http-proxy-middleware 代理解决(项目使用vue-cli脚手架搭建) 例如请求的url:“http://f.apiplus.cn/bj11x ...
SpringSecurity身份验证基础入门
对于没有访问权限的用户需要转到登录表单页面.要实现访问控制的方法多种多样,可以通过Aop.拦截器实现,也可以通过框架实现(如:Apache Shiro.Spring Security). pom.xm ...

LAB1 partIV

LAB1 partIV的更多相关文章

随机推荐

热门专题