CS61C Spring 2018 Project 5: Microservices in Go - Part 2: The testening


TA: Nathan Pemberton
Parts 1 and 2 both Due 4/30 @ 23:59:59

Overview

First: Please make sure you've read the instructions for part 1 before starting part 2

Writing a bunch of code is useless if it doesn't work. In the second part of this project we will be focusing on testing our memoizer to make sure it works as expected. You should expect to have significantly more test code than application code by the end of this. While it may seem like the memoizer is a fairly simple program, it turns out that there are a surprising number of ways it can fail, especially if we can't just give up when a service we depend on fails.

Setup

Important: This part of the project does not support windows. You will only be able to run the provided tests on osx or linux. It is strongly recommended that you do most of your work on hive.

This project relies on some newer features of Go. Before continuing, make sure that you are using Go version 1.10 or higher. To do this, run:

go version

You should see a number at least as big as 1.10. This is the default on the hive machines, but depending on how you installed Go, it might be older. Once you've made sure you have a recent version of Go, you'll need to download some helper code.

Update the proj5 package

We have made some modifications and additions to the proj5 repo, please update to the latest version:

go get -u github.com/61c-teach/sp18-proj5

Take a look inside, you'll see that there is a new file called err.go. This file contains a specialized error type (as opposed to the generic "error" that MnistResp currently passes) called MemErr. Of particular interest here is the use of memoizer-specific error condition codes (e.g. MemErr_serErr). These types let our memoizer be more specific about what went wrong, and allows upstream code to programatically handle different conditions. Without these, a human might need to look at the error descriptions to figure out what happened which could take hours instead of nanoseconds. This MemErrCause type acts much like an enum in C (see this link for more details on golang-style enums).

You should create this type using the "CreateMemErr" method only and read its cause using "GetErrCause" only. See test_helpers.go for an example of using GetErrCause. In test_helpers.go, the "validateResp" function uses GetErrCause to find the cause of resp.Err. What is interesting here is that resp.Err is of type "error" not "MemErr". This is a form of polymorphism (like inheritance in Java). Don't worry though, as long as your memoizer only produces errors of type MemErr, and you only ever read causes using GetErrCause, you shouldn't have to worry too much about it.

Hint: If you get errors that are similar to "panic: interface conversion: error is *errors.errorString, not proj5.MemErr", you're probably returning an error instead of a MemErr somewhere in your memoizer.

Get the memoizer_testing package

We have provided a new testing harness to make it a bit easier for you to write tests. You'll need to download this tarball and unpack it in your $GOPATH/src/bitbucket.org/USER/proj5-xxx directory. You're project directory should look someting like:

-proj5-xxx/
  -data/
  -proj5-testing/
      -data/
      -mock_test.go
      -run_buggy.py
      -test_framework.go
  -memoizer.go
  -memoizer_test.go
  -env.sh
  ...

Hint: the "wget" command downloads a file from a url into the current directory. You can use it to easily get the tarballs into your hive machines. Just "cd" to the desired directory and run "wget https://inst.eecs.berkeley.edu/~cs61c/sp18/projs/05-02/proj5-testing.tar"

We'll have more on how to use this framework later. For now, cd into the proj5-testing directory and change the "memoizer" import path in test_framework.go to point to your proj5-xxx directory. The new import path should look something like:

test_framework.go

package memoizer_testing

import (
  "testing"
  "time"

  //memoizer "github.com/61c-teach/proj5-impls/memoizer_good"
  memoizer "bitbucket.org/YOUR_USER/proj5-xxx
  ...

This tells the test framework to use the implementation of memoizer located in "$GOPATH/src/bitbucket.org/YOUR_USER/proj5-xxx" for its tests. Later, we will install a few more versions of memoizer for you to test, you can try them out by changing this import path again. Check if you've done this correctly by running go test. You should see tests running, but you might not be passing them all right now. This is fine for now, you'll fix the errors later.

Get the reference memoizer implementations

To aid you in testing, we are providing you with pre-compiled versions of our staff memoizer solution. One of these solutions is fully correct (as far as we know), but the rest have a specific bug introduced. You will need to write tests that discover these bugs. For now, let's just get them installed and make sure they run.

Download the proj5-impls.tar file and place it in your $GOPATH directory. Very Important: You must place proj5-impls.tar in your $GOPATH directory and have your CWD (current working directory) be in $GOPATH for the following to work. To ensure you are set up for the next step, run the following:

cd $GOPATH
ls

You should see something like:

bin  pkg proj5-impls.tar src

Now from within your $GOPATH directory, unpack the tar file:

tar -xf proj5-impls.tar

You should now have the binary package installed. Check your src/github.com/61c-teach/ directory, there should be an entry for proj5-impls/.

Note: What just happened? The proj5-impls.tar file implements an "overlay". It packages up a whole directory tree that matches the directory structure of your Go workspace. By unpacking it, tar copies files far down in the directory structure, creating any intermediate directories as needed. The reason we ask you to be so careful is that these types of tarballs can cause all sorts of problems if you accidently unpack them in a directory structure that they were not expecting. This technique is used a lot in the linux world to install packages and perform system upgrades.

Now lets test one of them out! Go back to your proj5-testing directory and replace the import for memoizer (in test_framework.go) with "github.com/61c-teach/proj5-impls/memoizer_classID" and run "go test" again. You should see this test failing. This is good! The implementation of Memoizer in memoizer_classID has a bug where it returns a bad message ID to the client if the classifier gives it a bad message ID. Now try changing the import to "github.com/61c-teach/proj5-impls/memoizer_classErr" and run "go test" again. This one passes. This is bad! memoizer_classErr has a bug where the memoizer reports the wrong sort of MemErrCause when the classifier returns an error, but our tests aren't good enough to catch it. You're main task in part-2 will be to write better tests that catch all these bugs.

Using Mocks for Testing

If you take a look at the errors in "Formal Memoizer Requirements" below, you'll notice that most of the errors involve what the memoizer should do if one of the other services misbehaves somehow. The problem is that (we hope) we gave you fairly reliable implementations for them, they don't really cause the sorts of problems that you're supposed to handle. How can you test your code's response to a buggy classifier without a buggy classifier? The answer is that you just need to write your own classifier, with blackjack and...ahem, I mean...with specific bugs and predictable behavior. This technique of creating a fake version of some dependency is called "Mocking" ("mock" is another word for fake).

Take a look in mock_test.go. You'll see that it defines a bunch of functions called things like mockClassifierGood and others called checkFullMock. The first implement the mock services (mockClassifierGood acts like a classifier that doesn't cause any errors, and always classifies images using the lblIm function). The second function is a checker. Checkers pretend to be the client of your service, they send it requests and make sure that the responses they get are correct. checkFullMock ensures that the memoizer behaves correctly when it's given known good caches and classifiers (where the classifiers use lblIm to label images). The test framework handles the running of mocks and checkers for you, just follow the lead of the TestMocks function.

Note: For the curious, the t.Run(label, func) function runs a subtest named "label" using the function "func". Subtests get reported independently when you do "go test -v" and can fail independently. In this case, func is actually a "closure" which is a one-off function (like python's "lambda") that calls into our framework to launch your mocks and checker. The runMockTest function spins up your mocks and whichever memoizer its import path is currently pointing to as goroutines, and then runs the checker with the handle of the memoizer.

run_buggy.py: Evaluating your tests

Once you've written some mocks, the first step is to make sure your memoizer passes your tests, but how do you know you've written enough tests? In the real-world, this is a very hard problem, but we've gotchu. We've provided you with a bunch of slightly broken implementations of the memoizer in proj5-impls. All but Memoizer_good ought to fail a good test suite, but most of them pass the starter code we gave you.

To run these tests, you can change the import path for the memoizer in test_framework.go to point to one. To run them all, you can run the run_buggy.py python script which will run them all at once and report which passed/failed.

Warning: I tried to make the run_buggy.py script fairly robust, but there are no guarantees. We strongly suggest that you commit often, and don't make any modifications (other than changing the import path) to test_framework.go.

Deep Dive into a Mock: Classifier BadID

We've included a mock for you to help inspire your own designs. Open up mock_test.go and search for where the test called "ClassBadID" is run. This test uses the following mocks (search for them lower down in mock_test.go):

You are encouraged to use this as a starting point for writing more unit tests. For example, what else could the classifier do on the whenFail'th request? How you might write a version of the cache that fails on the whenFail'th request? How can you control which messages go to the cache vs the classifier?

Code Coverage: Do my tests test my program?

You've now written some tests that were targeted to the memoizer specification. But how can you be sure that your tests cover all the different behaviors your code might exhibit? Might there be some bug lurking in your program? Have you tested all the different parts? To answer these questions, we need the concept of code coverage. Code coverage refers to what percentage of your program actually ran during the tests. For instance, if you have an if/else statement, you would want to have one test that tries the TRUE case, and another that tests the FALSE case.

Go provides some excellent tools on measuring code coverage. We recommend reading through this blog entry and trying out the examples before continuing.

Since our tester is in a different package than our memoizer code (notice that mock_test.go is in package proj5_testing), we'll need to use a slightly different coverage command. First, make sure that test_framework.go is importing your implementation of the memoizer. Now run the following commands from the proj5-testing directory:

go test -coverprofile=coverage.out -coverpkg "bitbucket.org/USER/proj5-xxx"
go tool cover -func=coverage.out

The first command runs your code with profiling enabled and saves the log of the run in coverage.out. The "go tool cover" command reads this file and presents the information in interesting ways. The "-func" version prints a simple coverage summary. A much more interesting way is to use "-html=coverage.out", this will open an html page with a visual representation of the coverage.

Hint: If you're working remotely (e.g. on the hive machines) and you still want to see the html output, you can use the command "go tool cover -html=coverage.out -o coverage.html". This will save the output in a file called coverage.html that you can then scp to you local machine for viewing.

Depending on your code and the tests you've written, you might have very high code-coverage right now. While this is good, it doesn't necessarily mean that your tests are perfect, it just means that they test the functionality that you have right now. For instance, a test that does nothing but fail immediately would have 100% code coverage! As you add checks in your memoizer to deal with bugs, you should re-run coverage to make sure your test covers the new code you added.

Requirements and Scoring

The points will be divided 70/30 between part 1 and part 2.

Part 1 Requirements (70pts)

To get full points on part 1, you must pass both the original unit tests as well as the new mock tests that we provided to you. There will not be any hidden tests.

Part 2 Requirements (30pts)

There will be three main components to your score on this part of equal weight (10pts each):

Formal Memoizer Requirements

To help you get a sense for what bugs to look for, and to understand what bugs the buggy implementations are experiencing, we provide here a formal list or requirements. The memoizer:

Note: We allow you to assume that a crashed service will politely close its respQ before crashing. In reality, failures can be much more subtle. Sometimes a service could just stop responding forever, other times it could just stop for a while and then come back, other times it just runs extra slow. We encourage you to think about how you might deal with those sorts of crashes (but you won't need to in this project).

Submission

Parts 1 and 2 will be submitted together (as one project) so the submission instructions from part 1 still apply. However, be sure to commit the proj5-testing directory to your repository.