The first several parts of this assignment involves the use of David Bailey's
model for learning hand-motion verbs. Please read O2
(Bailey et al.) in your reader before starting this assignment. You should also
allow yourself plenty of time in the lab to complete the assignment.
- You may work together with a partner for this assignment to use the
VerbLearn program in the lab, but you should answer questions separately and
submit individual assignments. (Please be sure to indicate your partner's
name on your assignment.)
- Please check back to this page periodically for possible updates to the
assignment.
- See here for general information
about the Soda labs, and here
for information about using the program elsewhere. Note that the assignment
works best on the designated Soda machines.
- See here for some basic UNIX commands.
- See below for some suggestions about what to do if you
are having problems getting the program to run properly.
The last part of the assignment requires you to do some simple calculations
involved in the model merging algorithm. You may wish to do this part of the
assignment first to get a better theoretical grasp of how VerbLearn works.
To get started on the assignment, you will need to get and install a copy of the
program in your directory. To do this you should:
- Log in to a PC machine running Solaris in Room 277 Soda. (See the note
above about running the system elsewhere). If you have trouble with the
display in 277, try from one of the SunRays machines (e.g. Room 275) as you
might have for assignment 6 and ssh to a Solaris-X86 server (e.g. po.eecs,
torus.cs).
At this point you may need to open a terminal window if there isn't one
already there. If you don't know how to do this, ask someone for help. (A
command like "xterm" or "Terminal" may be available in
a menu that appears when you click on the desktop with the right mouse
button.)
- Copy the file vl.tar.gz
to your directory: (or click on the link to download and save into current
working directory)
cp ~cs182/programs/vl.tar.gz . (don't forget the period at
the end)
- Uncompress and install the program:
gunzip vl.tar.gz
tar -xvf vl.tar (be patient -- this may take a little while)
This should result in a directory called ntlsystem appearing in
your directory.
- Type cd ntlystem to enter the directory. (Make sure to run
the program from this directory.)
This directory contains a directory called classes (which
contains the Java classes needed to run the program); a properties
directory (which has some system properties used by the program); and a data
directory (which has data used by the program. Do not erase these files!
- Type ./vl to start the program.
In a few moments you should see a window pop up, with a menubar across the
top.
WARNINGS: The program is somewhat slow and potentially
buggy (apologies!). It is recommended that you stick to the instructions below.
A few specific warnings:
- The system occasionally prints out comments to the window from which
you're running it; many of these are non-fatal and can be safely ignored
(including, for instance, warnings about older Java classes being used). If
something less ignorable arises, however, you may wish to kill the program.
You can do this by killing the entire VerbLearn window (in whatever way is
usual for your window manager).
- Some of the buttons/windows are not functional. For the most part they
shouldn't be dangerous, but clicking on them may produce additional
error/warnings in your terminal window.
- Give the system a few moments to react when you select a menu item.
- If the program hangs or fails in some other way, you can kill that copy of
the program and run another one.
In this section you will familiarize yourself with the system by re-enacting the
English verb training based on a pre-labeled set of "scenarios"
(action descriptions):
Viewing the data
- Open the scenario set. Select "Load Scenario Set..." from
the "File" menu, and choose "oldData.sset" in the file
dialog box that pops up. If you do not see this file there, you might want
to browse using the box; it should be in the path ($HOME)/vl/NTLSystem/data/scenario.sets/oldData.sset.
- Examine the scenario set. Select "Open Scenario Viewer"
from the "Scenario Set" menu. A window labeled
"ScenarioViewer" should pop up.
- Along the left edge of the screen is a list of scenarios with names
"sc001" through "sc199"; you can click on any of
these to see a particular scenario (using the scrollbar as necessary).
(Note that the < and > buttons at the bottom of the screen can
also be used to browse forward and backward, but it's probably best to
stick to the scenario list. (The << and >> buttons in the
bottom region don't work and should be ignored.)
- The main portion of the screen is split into two panels labeled
"Motor" and "World", which display the motor and
world feature structures (or fstructs), respectively, for the
currently selected scenario (whose name is also shown at the top). Each
of these is shown with its highest probability value, where
probabilities are indicated within parentheses. You can examine the
alternate values for these features by clicking on the pull-down menus
(for instance, the "accel" feature allows high, medium, low
and zero as acceleration values). Note that these are all actual
observed scenarios, so the values are all deterministic (1 for the
observed value, 0 for the others).
- The "Scenario Label" section of the screen displays the
label (verb or action name) given to this scenario by an
English-speaking informant.
Notes:
- The "Copy & Add" and "Delete Selected" buttons
(used to clone scenarios into new example scenarios and remove
scenarios, respectively) are not needed for this assignment. You should
avoid using these, since changes to the training data may interfere with
the standard training run procedures. (Or, if you experiment with these,
be careful to reload the scenario set before doing this part of the
assignment.)
- You may also wish to examine the windows that come up when you click
the "Edit" button next to the features, or the
"Add/Remove Features" button associated with each fstruct.
- You can also view the features of the scenario set by selecting
"Open Scenario Set Options" from the "Scenario Set"
menu. The "Generate Scenarios" window is not currently
functional.
Learning a lexicon
- Prepare to learn a lexicon. Select "Train" from the
"Lexicon" menu. A new window should pop up. Examine the (default)
training parameters displayed in the "Parameters" section of the
screen. Explanations for some of these parameters pop up when you put your
mouse on them; see here for
some additional descriptions. Leave these as is for now, but you will
experiment with them later.
- Name the lexicon to be created. Type a name into the
"Lexicon" text field. (Make sure you do this, or you'll have to
repeat your training session.)
- Choose a data set to train on. You can leave this as the default
"66-33-shuffled" dataset to use the standard 200-scenario training
set, but "original-50" and "second-50" are smaller
datasets available in the pull-down menu. (If you are on a slower machine
you may find these useful for doing shorter training runs, but the full
training set produces better results.)
- Learn a lexicon by training on the scenario set. Click the
"Train" button. Training will occur, spewing output to the
"Training log" window, which you may scroll through to inspect
after training finishes. This output is a running commentary on the model
merging algorithm in action:
- First, each scenario is incorporated into the lexicon as a new
word sense (with a slightly smoothed-out probability distribution).
- Then, for each word, similar word senses are merged into new
word senses. The information displayed in the training log describes the
effect of merging on the lexicon and the reason for stopping.
Note that you may wish to save your training log in a text file to examine
later. But please do NOT turn this log in!!
Evaluating the learned lexicon
- Inspect the trained lexicon. Select your newly created lexicon from
the "Open Lexicon" submenu of the "Lexicon" menu. If you
do not see a list of words at the top of the screen, you may need to enlarge
your window.
Click on individual verbs to see the word senses learned for each, and
click on individual word senses to see the actual learned models. Note
that the learned word senses resemble the training scenarios, except that
each feature is associated with a probability for each possible value.
- Test the trained lexicon's "Recognize" performance.
Select the "Recognize" command under the "Testing" menu.
- You will need to select a parameter file (use the "default"
parameters) and a dataset (typically you should choose the same name you
used in training, so that the test set is chosen to consist of novel
instances not in the training data).
- You can also select whether the results are given in
"Detail" or not, and whether you test on all scenarios or just
on a selected one.
- After you click on the button that performs the test, results of the
test should appear in the log window. For each test scenario, the
program should have produced the highest probability label.
- Test the trained lexicon's "Obey" performance. Select the
"Obey" command under the "Testing" menu. Follow
guidelines for "Recognize" test. Here, the program should produce
the most likely scenario for the given label (and world state) as well as
the corresponding recognized label.
Questions
- Pick a label for which the program learned at least 2 word senses.
- Show the learned senses and point out the main differences between
them. Write a sentence for each sense in which the sense is used
unambiguously.
- Do the word senses correspond to different senses in your dialect?
(That is, do you think VerbLearn learned a sense distinction that is
cognitively real for you?) Why/why not?
- What kinds of errors are in the recognition/obey tests? Do some errors
seem more plausible/expected than others?
Now that you are familiar with the program functions, you should experiment
with varying the default training parameters to see how the resulting lexicon
and senses differ from the original experiment. Follow the instructions above
for learning again, with the following changes:
- Pick one of the following parameters; the relevant descriptions from the parameters
guide are repeated here:
- minMerge: Minimum similarity required between two senses if they are
to be merged.
- modelPriorWeight: Magnitude of preference for merging relative to
preserving likelihood.
(Note: this is like the alpha parameter from the Part 4 handout.)
- batchSize: Number of new instances of a word to accumulate between
merging episodes.
(Note: the Part 4 handout uses on "online" version of the
algorithm, in which data instances are incorporated one at a time.)
- trainingPasses: Replicate training data this many times.
- minLabel: Minimum P(word | action) required to emit word.
- minObey: Minimum P(sense | worldstate) require to use sense in setting
linking Fstruct.
(Note: linking Fstruct refers to either Motor or World feature
structure.)
- Vary your chosen parameter over several training runs. Make sure you
choose a new lexicon name for each training run to allow you to compare them
later. You may wish to save each training log to examine. (Again, no need to
submit these logs!)
Questions
- Which parameter did you choose to vary? Over what range?
- How did the learned lexicons and/or test performance on
"Recognize" or "Obey" differ?
- Try to explain your results in terms of the model merging algorithm. (It
may be helpful to do Part 4 of the assignment first.)
Based on what you know about the system, you will now consider how you would
go about teaching the system a few new verbs, which may require a different set
of features than those already built in to the program. You may need to add some
features, and perhaps remove a few.
The words you should consider are crack and nudge, as well as one
other verb of your choice that requires one of the following features:
- intention
- object type, shape or weight
- orientation (such as in Regier's Mixtec "cii" example)
- For each of your three words:
- Describe any features you would need to change or add to the current
set of motor and world features to accommodate the word.
- Show some examples of scenarios you would add for each word sense.
Make sure the examples cover a reasonable span of the semantic range of
your new words. Describe in general the range of variation of feature
values for your added scenarios.
- Should the system be able to learn the new word? Why or why not?
- Now consider the following verbs:
throw (X at Y), give (X to Y), juggle (the ball),
aim (the ball), slice (the apple), dodge (a punch)
Discuss whether the system (which its original features) would be able to
learn these verbs. If not, what features would have to be added to the
system for it to learn the verb?
See handout: [PDF]
on the model merging algorithm and how it applies to the problem of learning
simple grammars. You should answer the 3 questions in the handout on the
calculations involved in the algorithm. (Note: the PDF has incorrect dates. Sorry. Ignore them.)
Reader: O2 (Bailey et
al.)
You may also wish to see the computational version of this assignment
for some additional links related to model merging and
right-regular grammars.
FYI: the computational and non-computational versions of the assignment use variations of the
same model merging handout, ending either in a final set of calculations
(non-comp) or in an implementation (comp).
- If you are having UNIX-based problems (involving log in, disk quota,
windows, path, etc.), you should seek help from the Cory Instructional
Account people.
- If you can't get the program running AT ALL, you may want to work with
someone else who can get it running on his/her instructional account.
- Note that if you are having system problems, you can still do most of the
assignment.
- As usual, make sure to describe the problems you encountered in enough
detail that someone trying to correct the problem or alert other students to
the problem will know what to do.