First: Please make sure you've read the instructions for part 1 before starting part 2
Writing a bunch of code is useless if it doesn't work. In the second part of this project we will be focusing on testing our memoizer to make sure it works as expected. You should expect to have significantly more test code than application code by the end of this. While it may seem like the memoizer is a fairly simple program, it turns out that there are a surprising number of ways it can fail, especially if we can't just give up when a service we depend on fails.
Important: This part of the project does not support windows. You will only be able to run the provided tests on osx or linux. It is strongly recommended that you do most of your work on hive.
This project relies on some newer features of Go. Before continuing, make sure that you are using Go version 1.10 or higher. To do this, run:
You should see a number at least as big as 1.10. This is the default on the hive machines, but depending on how you installed Go, it might be older. Once you've made sure you have a recent version of Go, you'll need to download some helper code.
Update the proj5 package
We have made some modifications and additions to the proj5 repo, please update to the latest version:
go get -u github.com/61c-teach/sp18-proj5
Take a look inside, you'll see that there is a new file called err.go. This file contains a specialized error type (as opposed to the generic "error" that MnistResp currently passes) called MemErr. Of particular interest here is the use of memoizer-specific error condition codes (e.g. MemErr_serErr). These types let our memoizer be more specific about what went wrong, and allows upstream code to programatically handle different conditions. Without these, a human might need to look at the error descriptions to figure out what happened which could take hours instead of nanoseconds. This MemErrCause type acts much like an enum in C (see this link for more details on golang-style enums).
You should create this type using the "CreateMemErr" method only and read its cause using "GetErrCause" only. See test_helpers.go for an example of using GetErrCause. In test_helpers.go, the "validateResp" function uses GetErrCause to find the cause of resp.Err. What is interesting here is that resp.Err is of type "error" not "MemErr". This is a form of polymorphism (like inheritance in Java). Don't worry though, as long as your memoizer only produces errors of type MemErr, and you only ever read causes using GetErrCause, you shouldn't have to worry too much about it.
Hint: If you get errors that are similar to "panic: interface conversion: error is *errors.errorString, not proj5.MemErr", you're probably returning an error instead of a MemErr somewhere in your memoizer.
Get the memoizer_testing package
We have provided a new testing harness to make it a bit easier for you to write tests. You'll need to download this tarball and unpack it in your $GOPATH/src/bitbucket.org/USER/proj5-xxx directory. You're project directory should look someting like:
-proj5-xxx/ -data/ -proj5-testing/ -data/ -mock_test.go -run_buggy.py -test_framework.go -memoizer.go -memoizer_test.go -env.sh ...
Hint: the "wget" command downloads a file from a url into the current directory. You can use it to easily get the tarballs into your hive machines. Just "cd" to the desired directory and run "wget https://inst.eecs.berkeley.edu/~cs61c/sp18/projs/05-02/proj5-testing.tar"
We'll have more on how to use this framework later. For now, cd into the proj5-testing directory and change the "memoizer" import path in test_framework.go to point to your proj5-xxx directory. The new import path should look something like:
package memoizer_testing import ( "testing" "time" //memoizer "github.com/61c-teach/proj5-impls/memoizer_good" memoizer "bitbucket.org/YOUR_USER/proj5-xxx ...
This tells the test framework to use the implementation of memoizer
located in "$GOPATH/src/bitbucket.org/YOUR_USER/proj5-xxx" for its tests.
Later, we will install a few more versions of memoizer for you to test, you
can try them out by changing this import path again. Check if you've done
this correctly by running
go test. You should see tests
running, but you might not be passing them all right now. This is fine for
now, you'll fix the errors later.
Get the reference memoizer implementations
To aid you in testing, we are providing you with pre-compiled versions of our staff memoizer solution. One of these solutions is fully correct (as far as we know), but the rest have a specific bug introduced. You will need to write tests that discover these bugs. For now, let's just get them installed and make sure they run.
Download the proj5-impls.tar file and place it in your $GOPATH directory. Very Important: You must place proj5-impls.tar in your $GOPATH directory and have your CWD (current working directory) be in $GOPATH for the following to work. To ensure you are set up for the next step, run the following:
cd $GOPATH ls
You should see something like:
bin pkg proj5-impls.tar src
Now from within your $GOPATH directory, unpack the tar file:
tar -xf proj5-impls.tar
You should now have the binary package installed. Check your src/github.com/61c-teach/ directory, there should be an entry for proj5-impls/.
Note: What just happened? The proj5-impls.tar file implements an "overlay". It packages up a whole directory tree that matches the directory structure of your Go workspace. By unpacking it, tar copies files far down in the directory structure, creating any intermediate directories as needed. The reason we ask you to be so careful is that these types of tarballs can cause all sorts of problems if you accidently unpack them in a directory structure that they were not expecting. This technique is used a lot in the linux world to install packages and perform system upgrades.
Now lets test one of them out! Go back to your proj5-testing directory and replace the import for memoizer (in test_framework.go) with "github.com/61c-teach/proj5-impls/memoizer_classID" and run "go test" again. You should see this test failing. This is good! The implementation of Memoizer in memoizer_classID has a bug where it returns a bad message ID to the client if the classifier gives it a bad message ID. Now try changing the import to "github.com/61c-teach/proj5-impls/memoizer_classErr" and run "go test" again. This one passes. This is bad! memoizer_classErr has a bug where the memoizer reports the wrong sort of MemErrCause when the classifier returns an error, but our tests aren't good enough to catch it. You're main task in part-2 will be to write better tests that catch all these bugs.
Using Mocks for Testing
If you take a look at the errors in "Formal Memoizer Requirements" below, you'll notice that most of the errors involve what the memoizer should do if one of the other services misbehaves somehow. The problem is that (we hope) we gave you fairly reliable implementations for them, they don't really cause the sorts of problems that you're supposed to handle. How can you test your code's response to a buggy classifier without a buggy classifier? The answer is that you just need to write your own classifier, with blackjack and...ahem, I mean...with specific bugs and predictable behavior. This technique of creating a fake version of some dependency is called "Mocking" ("mock" is another word for fake).
Take a look in mock_test.go. You'll see that it defines a bunch of
functions called things like mockClassifierGood and others called
checkFullMock. The first implement the mock services (mockClassifierGood
acts like a classifier that doesn't cause any errors, and always classifies
images using the
lblIm function). The second function is a checker.
Checkers pretend to be the client of your service, they send it requests
and make sure that the responses they get are correct. checkFullMock
ensures that the memoizer behaves correctly when it's given known good
caches and classifiers (where the classifiers use lblIm to label images).
The test framework handles the running of mocks and checkers for you, just
follow the lead of the TestMocks function.
Note: For the curious, the t.Run(label, func) function runs a subtest named "label" using the function "func". Subtests get reported independently when you do "go test -v" and can fail independently. In this case, func is actually a "closure" which is a one-off function (like python's "lambda") that calls into our framework to launch your mocks and checker. The runMockTest function spins up your mocks and whichever memoizer its import path is currently pointing to as goroutines, and then runs the checker with the handle of the memoizer.
run_buggy.py: Evaluating your tests
Once you've written some mocks, the first step is to make sure your memoizer passes your tests, but how do you know you've written enough tests? In the real-world, this is a very hard problem, but we've gotchu. We've provided you with a bunch of slightly broken implementations of the memoizer in proj5-impls. All but Memoizer_good ought to fail a good test suite, but most of them pass the starter code we gave you.
To run these tests, you can change the import path for the memoizer in test_framework.go to point to one. To run them all, you can run the run_buggy.py python script which will run them all at once and report which passed/failed.
Warning: I tried to make the run_buggy.py script fairly robust, but there are no guarantees. We strongly suggest that you commit often, and don't make any modifications (other than changing the import path) to test_framework.go.
Deep Dive into a Mock: Classifier BadID
We've included a mock for you to help inspire your own designs. Open up mock_test.go and search for where the test called "ClassBadID" is run. This test uses the following mocks (search for them lower down in mock_test.go):
- mockClassifierBadID: This classifier acts just like
mockClassifierGood most of the time, but on the
whenFail'th request (
whenFailis a constant defined at the top of the file), it returns a bogus ID (see the line that has
if reqCount == whenFail), then it goes back to returning the correct ID
- mockCacheGood: This cache behaves normally, but it performs a few extra checks on IDs to make sure your memoizer is using it correctly.
- checkClassBadId: This checker makes sure that the memoizer handles a bad ID from the classifier correctly. It does this by:
- Sending the first
whenFail-1images to your memoizer and checking them using the proj5.CheckImages routine that you've seen before. Notice how it computes the "expects" array using the
lblImfunction instead of looking at the real labels. Unlike the real classifier, we know what the label should be for each image so we can double check that the memoizer hasn't messed with them. These should work since the classifier behaves normally for the first
- Sending the
whenFail'th message directly instead of using
proj5.CheckImage. This is because this request is supposed to fail (our mock classifier returns a bogus ID on this request). We then check that the response from the memoizer contains an error, and that the error cause is
MemErr_serCorrupt(indicating that one of the memoizer's services corrupted a request).
- Retrying the request. Our mock is designed to respond normally to
all requests after the
whenFail'th, so this ought to succeed. We can use proj5.CheckImage for requests that we expect to succeed.
You are encouraged to use this as a starting point for writing more unit
tests. For example, what else could the classifier do on the
whenFail'th request? How you might write a version of the
cache that fails on the
whenFail'th request? How can you
control which messages go to the cache vs the classifier?
Code Coverage: Do my tests test my program?
You've now written some tests that were targeted to the memoizer specification. But how can you be sure that your tests cover all the different behaviors your code might exhibit? Might there be some bug lurking in your program? Have you tested all the different parts? To answer these questions, we need the concept of code coverage. Code coverage refers to what percentage of your program actually ran during the tests. For instance, if you have an if/else statement, you would want to have one test that tries the TRUE case, and another that tests the FALSE case.
Go provides some excellent tools on measuring code coverage. We recommend reading through this blog entry and trying out the examples before continuing.
Since our tester is in a different package than our memoizer code (notice that mock_test.go is in package proj5_testing), we'll need to use a slightly different coverage command. First, make sure that test_framework.go is importing your implementation of the memoizer. Now run the following commands from the proj5-testing directory:
go test -coverprofile=coverage.out -coverpkg "bitbucket.org/USER/proj5-xxx" go tool cover -func=coverage.out
The first command runs your code with profiling enabled and saves the log of the run in coverage.out. The "go tool cover" command reads this file and presents the information in interesting ways. The "-func" version prints a simple coverage summary. A much more interesting way is to use "-html=coverage.out", this will open an html page with a visual representation of the coverage.
Hint: If you're working remotely (e.g. on the hive machines) and you still want to see the html output, you can use the command "go tool cover -html=coverage.out -o coverage.html". This will save the output in a file called coverage.html that you can then scp to you local machine for viewing.
Depending on your code and the tests you've written, you might have very high code-coverage right now. While this is good, it doesn't necessarily mean that your tests are perfect, it just means that they test the functionality that you have right now. For instance, a test that does nothing but fail immediately would have 100% code coverage! As you add checks in your memoizer to deal with bugs, you should re-run coverage to make sure your test covers the new code you added.
Requirements and Scoring
The points will be divided 70/30 between part 1 and part 2.
Part 1 Requirements (70pts)To get full points on part 1, you must pass both the original unit tests as well as the new mock tests that we provided to you. There will not be any hidden tests.
Part 2 Requirements (30pts)
There will be three main components to your score on this part of equal weight (10pts each):
- Catching Known Bugs: Your mocks should catch the bugs in the provided proj5-impls (except Memoizer_good which, hopefully, doesn't have any bugs). You must catch all the bugs for full points, but you'll only lose one or two points for missing some.
- Passing Tests: Your memoizer should pass all of your own tests, and most of our tests. Our tests are sufficient to catch all the bugs in proj5-impls, but don't test much beyond that. If you catch all those bugs, and pass your own tests, you'll get full points on this part.
- Code Coverage: Your tests should acheive 100% code coverage on your memoizer. Coverage of instructor-provided helper functions is not necessary.
Formal Memoizer Requirements
To help you get a sense for what bugs to look for, and to understand what bugs the buggy implementations are experiencing, we provide here a formal list or requirements. The memoizer:
- Should exit and close its respQ when the client closes the reqQ
- Should close cache and classifier reqQs when it exits
- May not send duplicate IDs to the classifier
- May not send duplicate IDs for reads to the cache (writes are OK)
- Bad behavior from the cache should be ignored (use classifier instead). Bad behavior from the cache includes (but isn't limited to):
- Bad message IDs
- Crashed. You may assume that the cache will close its respQ if it crashes.
- Bad behavior from Classifier should only cause an error if the cache does not have the requested label (or has crashed/erred). In this case, the memoizer should return an error of type MemErr to the client with the appropriate error cause. Bad behavior from the classifier includes (but isn't limited to):
- Bad message IDs (cause=MemErr_serCorrupt)
- Arbitrary error messages (cause=MemErr_serErr, the MemErr should also contain the original error from the classifier in its serErr field)
- Crashed. You may assume that the classifier will close its respQ if it crashes. (cause=MemErr_serCrash)
- All errors reported by the memoizer (in the MnistResp.Err field) must be of type MemErr and must have the correct MemErrCause (see the comments in err.go for error codes and their meanings). Note that the "serErr" field of MemErr should be set to "nil" except when the cause is MemErr_serErr (it should be set to the error returned by the classifier in this case).
Note: We allow you to assume that a crashed service will politely close its respQ before crashing. In reality, failures can be much more subtle. Sometimes a service could just stop responding forever, other times it could just stop for a while and then come back, other times it just runs extra slow. We encourage you to think about how you might deal with those sorts of crashes (but you won't need to in this project).
Parts 1 and 2 will be submitted together (as one project) so the submission instructions from part 1 still apply. However, be sure to commit the proj5-testing directory to your repository.