Test Phase of Projects
This phase of your project involves testing your main hypothesis,
with a particular focus on testing the metrics for success that you defined
in your implementation writeup. A good test, whether or not it involves users,
must have a logical connection between your hypothesis, your test
plan, and your analysis. A typical hypothesis might be:
Hypothesis: Jen can be trained to speak more slowly based on class participation
How would you test this? You could do an empirical study with quantitative
results
Test plan: We will all yawn and put our heads on our arms whenever Jen
speaks quickly. We will look very awake when she speaks slowly. One person
will record everything she says and the times when people look sleepy/awake
for later analysis
A quantitative analysis would look for numerical evidence that Jen did
or did not start to speak more slowly as time went on, or that Jen slowed
down whenever people started to look sleepy. The audio recordings would be
analyzed to measure speaking rate, class reaction timestamps would be used
to index in to this, and so on.
The same hypothesis could be tested more qualitatively. For example:
Test plan: We will all yawn and put our heads on our arms whenever Jen
speaks quickly. We will look very awake when she speaks slowly. We
will interview Jen both before and afterwards to find out more about her awareness
of her own speaking speed, what she does to slow down, and how these changes
impacted her, whether she noticed them at all, and so on. We will
also fill out questionnaires about Jen's speaking speed.
A qualitative analysis would then compare Jen's answers in both interviews
to see if the changes made by the class had any impact. Class member questionnaires
would also be analyzed, as the class might subjectively feel there is
a speed difference whether or not there actually is one (perhaps better concentration/attention
makes Jen's speech seem slower or faster).
There are many subtelties to test design which many of you have already
dealt with in this class, and I will not go into all of those details here.
A few of those issues are specific to project types and described below.
1. Design/Prototype/Eval
This is the "classic" project type. At this stage, you have selected some
prototype to focus on and built it with the test in mind. At this point you
are doing a classic study in the style described above to see if the prototype
solves the problem you set out to solve.
2. Tools and Techniques
A tool/technique can be tested in different ways, depending on how it should
generalize. In general you should show
- You can do something with your tool at least as well as you could
do it before
- You can do a range of things with your tool that is at least as large
as the range of things handled by the tools you are trying to improve upon
You should also show that you can do either (1) or (2) better than before
(i.e. you can do something old more easily with your tool, or you can do even
more things than past tools could support).
These statements can be proved programmatically to some extent, or can be
shown through experimental studies more similar to the one described above.
All of this depends in part upon the tool.
3. Methods/Models
A method/model model project, at this stage, should have some solid evidence
that the method or model works as expected. At this point, similar to a tool
or technique, you want to show the power of the method/model. This might mean
comparing it to other methods/models, or using it to analyze a task or application
that had not been previously analyzed in that fashion. The end result is
proof that the method or model can be applied to a new domain or can be used
to do something old in a better fashion.
Writeup
The final writeup of this phase is not due until the end of the semester,
and is described in more detail in the final paper
What you should turn in on Friday, April 19 is your proposed user study
plan. This should be fairly short, and should document web page.how you
will test your hypothesis. The reason you are required to turn this in is
so that we can help you to debug the design of your test before you begin
the actual study. As a result, your writeup should
10 pts state the hypothesis & metrics for success (from implementation
writeup)
10 pts describe the experimental method (subjects, tasks, etc)
10 pts describe how you plan to do the analysis
If you have already written such a document, for example for CPHS, please
turn it in to me again with any modifications you have made since included.
You will recieve comments on these writeups by Monday April 22nd at the
latest so that they can be of help to you as you actually conduct the user
study. I suggest you start recruiting subjects even sooner.