6.824Lab1实验指导书：MapReduce
Introduction
In this lab you'll build a MapReduce system. You'll implement a worker process that calls application Map and Reduce functions and handles reading and writing files, and a coordinator process that hands out tasks to workers and copes with failed workers. You'll be building something similar to the . (Note: the lab us "coordinator" instead of the paper's "master".)
ginnie
本lab要实现MR，详细见论⽂
We supply you with a simple quential mapreduce implementation in src/ It runs the maps and reduces one at a time, in a single process. We also provide you with a couple of MapReduce applications: word-count in , and a text indexer in You can run word count quentially as follows:
肃杀
提供了 mr-quential，以及⼀些mr应⽤
$ cd ~/6.824
$ cd src/main
$ go build -race -buildmode=plugin ../
$ rm mr-out*
$ go run - wc.so pg*.txt
$ more mr-out-0
A 509
ABOUT 2
ACT 8
...
↑ 这样运⾏mr-quential
(Note: If you don't compile with -race, you won't be able to run with -race)
Feel free to borrow code You should also have a look at to e what MapReduce application code looks like.
↑可以借鉴的东西
pilot>viso
Your Job ()
Your job is to implement a distributed MapReduce, consisting of two programs, the coordinator and the worker. There will be just one coordinator process, and one or more worker process executing in parallel. In a real system the workers would run on a bunch of different machines, but for this lab you'll run them all on a single machine. The workers will talk to the coordinator via RPC. Each worker process will ask the coordinator for a task, read the task's input from one or more files, execute the task, and write the task's output to one or more files. The coordinator should notice if a worker hasn't completed its task in a reasonable amount of time (for this lab, u ten conds), and give the same task to a different worker.
实现⼀个分布式的MR，master和worker。本lab中所有的都运⾏在⼀台机器上。worker和master通过RPC通信。本lab中，如果worker 没有在10s内响应master，master就应该将对应的任务分配给其他worker
We have given you a little code to start you off. The "main" routines for the coordinator and worker are
in and ; don't change the files. You should put your implementation in , , and
↑改哪些代码，不改哪些代码
Here's how to run your code on the word-count MapReduce application. First, make sure the word-count plugin is freshly built:
$ go build -race -buildmode=plugin ../
In the main directory, run the coordinator.
$ rm mr-out*
$ go run - pg-*.txt
The pg-*.txt arguments are the input files; each file corresponds to one "split", and is the input to one Map task. The -race flags runs go with its race detector.
↑每个pg**⽂件代表⼀个输⼊的split，对应⼀个map任务的输⼊，在本实验中推荐使⽤-race进⾏build和run In one or more other windows, run some workers:
（可启动多个worker）
$ go run - wc.so
When the workers and coordinator have finished, look at the output in mr-out-*. When you've completed the lab, the sorted union of the output files should match the quential output, like this:
master和worker退出之后，就可以检查最终输出了
$ cat mr-out-* | sort | more
A 509
ABOUT 2
ACT 8
英语六级真题下载
...
We supply you with a test script in main/test-mr.sh. The tests check that the wc and indexer MapReduce applications produce the correct output when given files as input. The test
s also check that your implementation runs the Map and Reduce tasks in parallel, and that your implementation recovers from workers that crash while running tasks.
测试这些：
1.是否和mr-quential产⽣⼀样的结果
2.worker是否并⾏执⾏
3.如果worker crash，你的实现能否recover（就是指即使worker crash，要可以得出正确结果吧）
If you run the test script now, it will hang becau the coordinator never finishes:
初始代码，因为Done永远返回fal，所以master永远不会退出
$ cd ~/6.824/src/main
$ bash test-mr.sh
*** Starting wc test.
You can change ret := fal to true in the Done function in so that the coordinator exits
immediately. Then:
如果将Done改为直接返回true，那么master马上退出
$ bash test-mr.sh
*** Starting wc test.
sort: No such file or directory
cmp: EOF on mr-wc-all
--- wc output is not the same
--- wc test: FAIL上海夜校
$
The test script expects to e output in files named mr-out-X, one for each reduce task. The empty implementations of and don't produce tho files (or do much of anything el), so the test fails.
因为此时还没有产⽣输出⽂件，所以不会通过测试
When you've finished, the test script output should look like this:
$ bash test-mr.sh
*** Starting wc test.stayhereforever
--- wc test: PASS
*** Starting indexer test.
--- indexer test: PASS
*** Starting map parallelism test.
--- map parallelism test: PASS
*** Starting reduce parallelism test.
--- reduce parallelism test: PASS
*** Starting crash test.
powerbook--- crash test: PASS
*** PASSED ALL TESTS
$
You'll also e some errors from the Go RPC package that look like
可能会有这种提⽰，不⽤在意
2019/12/16 13:27:09 rpc.Register: method "Done" has 1 input parameters; needs exactly three
Ignore the messages; registering the coordinator as an checks if all its methods are suitable for RPCs (have 3 inputs); we know that Done is not called via RPC.
↑留意下这⼀句
A few rules:
map阶段将中间键值对通过partition函数分到nReduce个部分，nReduce是由上述路径传播：mrmaster->master
worker应该将第X个reduce任务的输出放到名为mr-out-X的⽂件中
mr-out-X中每⾏对应⼀个reduce函数的输出（实现中是怎样的？√）
格式同mr-quential
只能修改这三个⽂件，其他的如果修改了，测试之前要改回去
worker应该将map的输出放在当前⽬录下，之后其将被当做reduce的输⼊
当MR整个完成时，Done返回true，此时master退出
当全部⼯作完成时，worker应该退出。⼀种简单的办法是利⽤call的返回值：如果worker不能联系上master，那就可以假设master因为全部任务已完成⽽退出了，那么worker也可以退出了。当然你的实现⾥，也可以是master向worker发⼀个“请退出”的信息3dmax学习班
Hints
可以这样开始：
⾸先让worker发⼀个RPC请求给master，请求任务。然后让master返回⼀个还没开始的map任务的输⼊⽂件的⽂件名（是怎样的流程？）
然后让worker来读这个⽂件，并且调⽤map函数，同mr-quential
map和reduce函数是在so⽂件⾥，动态加载
advice
如果修改了这个⽂件夹⾥的，那么你可能也需要rebuild那些MR应⽤
