Test Phase of Projects

 This phase of your project involves testing your main hypothesis, with a particular focus on testing the metrics for success that you defined in your implementation writeup. A good test, whether or not it involves users, must have a logical connection between your hypothesis, your test plan, and your analysis. A typical hypothesis might be:
Hypothesis: Jen can be trained to speak more slowly based on class participation
How would you test this? You could do an empirical study with quantitative results
Test plan: We will all yawn and put our heads on our arms whenever Jen speaks quickly. We will look very awake when she speaks slowly. One person will record everything she says and the times when people look sleepy/awake for later analysis
A quantitative analysis would look for numerical evidence that Jen did or did not start to speak more slowly as time went on, or that Jen slowed down whenever people started to look sleepy. The audio recordings would be analyzed to measure speaking rate, class reaction timestamps would be used to index in to this, and so on.

The same hypothesis could be tested more qualitatively. For example:
Test plan: We will all yawn and put our heads on our arms whenever Jen speaks quickly. We will look very awake when she speaks slowly. We will interview Jen both before and afterwards to find out more about her awareness of her own speaking speed, what she does to slow down, and how these changes impacted her, whether she noticed them at all, and so on. We will also fill out questionnaires about Jen's speaking speed.
A qualitative analysis would then compare Jen's answers in both interviews to see if the changes made by the class had any impact. Class member questionnaires would also be analyzed, as the class might subjectively  feel there is a speed difference whether or not there actually is one (perhaps better concentration/attention makes Jen's speech seem slower or faster).

There are many subtelties to test design which many of you have already dealt with in this class, and I will not go into all of those details here. A few of those issues are specific to project types and described below.

1. Design/Prototype/Eval

This is the "classic" project type. At this stage, you have selected some prototype to focus on and built it with the test in mind. At this point you are doing a classic study in the style described above to see if the prototype solves the problem you set out to solve.

2. Tools and Techniques

A tool/technique can be tested in different ways, depending on how it should generalize. In general you should show
  1. You can do something with your tool at least as well as you could do it before
  2. You can do a range of things with your tool that is at least as large as the range of things handled by the tools you are trying to improve upon
You should also show that you can do either (1) or (2) better than before (i.e. you can do something old more easily with your tool, or you can do even more things than past tools could support).

These statements can be proved programmatically to some extent, or can be shown through experimental studies more similar to the one described above. All of this depends in part upon the tool.

3. Methods/Models

A method/model model project, at this stage, should have some solid evidence that the method or model works as expected. At this point, similar to a tool or technique, you want to show the power of the method/model. This might mean comparing it to other methods/models, or using it to analyze a task or application that had not been previously analyzed in that fashion. The end result is proof that the method or model can be applied to a new domain or can be used to do something old in a better fashion.

Writeup

The final writeup of this phase is not due until the end of the semester, and is described in more detail in the final paper What you should turn in on Friday, April 19 is your proposed user study plan. This should be fairly short, and should document web page.how you will test your hypothesis. The reason you are required to turn this in is so that we can help you to debug the design of your test before you begin the actual study. As a result, your writeup should
10 pts state the hypothesis & metrics for success (from implementation writeup)
10 pts describe the experimental method (subjects, tasks, etc)
10 pts describe how you plan to do the analysis

If you have already written such a document, for example for CPHS, please turn it in to me again with any modifications you have made since included.

You will recieve comments on these writeups by Monday April 22nd at the latest so that they can be of help to you as you actually conduct the user study. I suggest you start recruiting subjects even sooner.