The first several parts of this assignment involves the use of David Bailey's model for learning hand-motion verbs. Please read O2 (Bailey et al.) in your reader before starting this assignment. You should also allow yourself plenty of time in the lab to complete the assignment.

• You may work together with a partner for this assignment to use the VerbLearn program in the lab, but you should answer questions separately and submit individual assignments. (Please be sure to indicate your partner's name on your assignment.)
• Please check back to this page periodically for possible updates to the assignment.
• See here for general information about the Soda labs, and here for information about using the program elsewhere. Note that the assignment works best on the designated Soda machines.
• See here for some basic UNIX commands.
• See below for some suggestions about what to do if you are having problems getting the program to run properly.

1. Log in to a PC machine running Solaris in Room 277 Soda. (See the note above about running the system elsewhere). If you have trouble with the display in 277, try from one of the SunRays machines (e.g. Room 275) as you might have for assignment 6 and ssh to a Solaris-X86 server (e.g. po.eecs, torus.cs).
   

   At this point you may need to open a terminal window if there isn't one already there. If you don't know how to do this, ask someone for help. (A command like "xterm" or "Terminal" may be available in a menu that appears when you click on the desktop with the right mouse button.)

2. Copy the file vl.tar.gz to your directory: (or click on the link to download and save into current working directory)
   cp ~cs182/programs/vl.tar.gz . (don't forget the period at the end)
3. Uncompress and install the program:
   gunzip vl.tar.gz
   tar -xvf vl.tar (be patient -- this may take a little while)
   This should result in a directory called ntlsystem appearing in your directory.
4. Type cd ntlystem to enter the directory. (Make sure to run the program from this directory.)

   This directory contains a directory called classes (which contains the Java classes needed to run the program); a properties directory (which has some system properties used by the program); and a data directory (which has data used by the program. Do not erase these files!

5. Type vl to start the program. (If you get a Command not found error, you might try ./vl.)
   In a few moments you should see a window pop up, with a menubar across the top.
   

• The system occasionally prints out comments to the window from which you're running it; many of these are non-fatal and can be safely ignored (including, for instance, warnings about older Java classes being used). If something less ignorable arises, however, you may wish to kill the program. You can do this by killing the entire VerbLearn window (in whatever way is usual for your window manager).
• Some of the buttons/windows are not functional. For the most part they shouldn't be dangerous, but clicking on them may produce additional error/warnings in your terminal window.
• Give the system a few moments to react when you select a menu item.
• If the program hangs or fails in some other way, you can kill that copy of the program and run another one.

Viewing the data

• Open the scenario set. Select "Load Scenario Set..." from the "File" menu, and choose "oldData.sset" in the file dialog box that pops up. If you do not see this file there, you might want to browse using the box; it should be in the path ($HOME)/vl/NTLSystem/data/scenario.sets/oldData.sset.
• Examine the scenario set. Select "Open Scenario Viewer" from the "Scenario Set" menu. A window labeled "ScenarioViewer" should pop up.
  • Along the left edge of the screen is a list of scenarios with names "sc001" through "sc199"; you can click on any of these to see a particular scenario (using the scrollbar as necessary). (Note that the < and > buttons at the bottom of the screen can also be used to browse forward and backward, but it's probably best to stick to the scenario list. (The << and >> buttons in the bottom region don't work and should be ignored.)
  • The main portion of the screen is split into two panels labeled "Motor" and "World", which display the motor and world feature structures (or fstructs), respectively, for the currently selected scenario (whose name is also shown at the top). Each of these is shown with its highest probability value, where probabilities are indicated within parentheses. You can examine the alternate values for these features by clicking on the pull-down menus (for instance, the "accel" feature allows high, medium, low and zero as acceleration values). Note that these are all actual observed scenarios, so the values are all deterministic (1 for the observed value, 0 for the others).
  • The "Scenario Label" section of the screen displays the label (verb or action name) given to this scenario by an English-speaking informant.
  Notes:
  • The "Copy & Add" and "Delete Selected" buttons (used to clone scenarios into new example scenarios and remove scenarios, respectively) are not needed for this assignment. You should avoid using these, since changes to the training data may interfere with the standard training run procedures. (Or, if you experiment with these, be careful to reload the scenario set before doing this part of the assignment.)
  • You may also wish to examine the windows that come up when you click the "Edit" button next to the features, or the "Add/Remove Features" button associated with each fstruct.
  • You can also view the features of the scenario set by selecting "Open Scenario Set Options" from the "Scenario Set" menu. The "Generate Scenarios" window is not currently functional.

Learning a lexicon

• Prepare to learn a lexicon. Select "Train" from the "Lexicon" menu. A new window should pop up. Examine the (default) training parameters displayed in the "Parameters" section of the screen. Explanations for some of these parameters pop up when you put your mouse on them; see here for some additional descriptions. Leave these as is for now, but you will experiment with them later.
• Name the lexicon to be created. Type a name into the "Lexicon" text field. (Make sure you do this, or you'll have to repeat your training session.)
• Choose a data set to train on. You can leave this as the default "66-33-shuffled" dataset to use the standard 200-scenario training set, but "original-50" and "second-50" are smaller datasets available in the pull-down menu. (If you are on a slower machine you may find these useful for doing shorter training runs, but the full training set produces better results.)
• Learn a lexicon by training on the scenario set. Click the "Train" button. Training will occur, spewing output to the "Training log" window, which you may scroll through to inspect after training finishes. This output is a running commentary on the model merging algorithm in action:
  1. First, each scenario is incorporated into the lexicon as a new word sense (with a slightly smoothed-out probability distribution).
  2. Then, for each word, similar word senses are merged into new word senses. The information displayed in the training log describes the effect of merging on the lexicon and the reason for stopping.
  Note that you may wish to save your training log in a text file to examine later. But please do NOT turn this log in!!

Evaluating the learned lexicon

• Inspect the trained lexicon. Select your newly created lexicon from the "Open Lexicon" submenu of the "Lexicon" menu. If you do not see a list of words at the top of the screen, you may need to enlarge your window.

  Click on individual verbs to see the word senses learned for each, and click on individual word senses to see the actual learned models. Note that the learned word senses resemble the training scenarios, except that each feature is associated with a probability for each possible value.

• Test the trained lexicon's "Recognize" performance. Select the "Recognize" command under the "Testing" menu.
  • You will need to select a parameter file (use the "default" parameters) and a dataset (typically you should choose the same name you used in training, so that the test set is chosen to consist of novel instances not in the training data).
  • You can also select whether the results are given in "Detail" or not, and whether you test on all scenarios or just on a selected one.
  • After you click on the button that performs the test, results of the test should appear in the log window. For each test scenario, the program should have produced the highest probability label.
• Test the trained lexicon's "Obey" performance. Select the "Obey" command under the "Testing" menu. Follow guidelines for "Recognize" test. Here, the program should produce the most likely scenario for the given label (and world state) as well as the corresponding recognized label.

Questions

1. Pick a label for which the program learned at least 2 word senses.
   1. Show the learned senses and point out the main differences between them. Write a sentence for each sense in which the sense is used unambiguously.
   2. Do the word senses correspond to different senses in your dialect? (That is, do you think VerbLearn learned a sense distinction that is cognitively real for you?) Why/why not?
2. What kinds of errors are in the recognition/obey tests? Do some errors seem more plausible/expected than others?

Now that you are familiar with the program functions, you should experiment with varying the default training parameters to see how the resulting lexicon and senses differ from the original experiment. Follow the instructions above for learning again, with the following changes:

1. Pick one of the following parameters; the relevant descriptions from the parameters guide are repeated here:
   • minMerge: Minimum similarity required between two senses if they are to be merged.
   • modelPriorWeight: Magnitude of preference for merging relative to preserving likelihood.
     (Note: this is like the alpha parameter from the Part 4 handout.)
   • batchSize: Number of new instances of a word to accumulate between merging episodes.
     (Note: the Part 4 handout uses on "online" version of the algorithm, in which data instances are incorporated one at a time.)
   • trainingPasses: Replicate training data this many times.
   • minLabel: Minimum P(word | action) required to emit word.
   • minObey: Minimum P(sense | worldstate) require to use sense in setting linking Fstruct.
     (Note: linking Fstruct refers to either Motor or World feature structure.)
2. Vary your chosen parameter over several training runs. Make sure you choose a new lexicon name for each training run to allow you to compare them later. You may wish to save each training log to examine. (Again, no need to submit these logs!)

Questions

1. Which parameter did you choose to vary? Over what range?
2. How did the learned lexicons and/or test performance on "Recognize" or "Obey" differ?
3. Try to explain your results in terms of the model merging algorithm. (It may be helpful to do Part 4 of the assignment first.)

Based on what you know about the system, you will now consider how you would go about teaching the system a few new verbs, which may require a different set of features than those already built in to the program. You may need to add some features, and perhaps remove a few.

The words you should consider are break and tap, as well as one other verb of your choice that requires one of the following features:

• intention
• object type, shape or weight
• orientation (such as in Regier's Mixtec "cii" example)

1. For each of your three words:
   1. Describe any features you would need to change or add to the current set of motor and world features to accommodate the word.
   2. Show some examples of scenarios you would add for each word sense. Make sure the examples cover a reasonable span of the semantic range of your new words. Describe in general the range of variation of feature values for your added scenarios.
   3. Should the system be able to learn the new word? Why or why not?
2. Now consider the following verbs:

   throw (X at Y), accept (X from Y), juggle (the ball), aim (the ball), peel (the apple), block (a punch)

   Discuss whether the system (which its original features) would be able to learn these verbs. If not, what features would have to be added to the system for it to learn the verb?

You may also wish to see the computational version of this assignment for some additional links related to model merging and right-regular grammars.

FYI: the computational and non-computational versions of the assignment use variations of the same model merging handout, ending either in a final set of calculations (non-comp) or in an implementation (comp).

• If you are having UNIX-based problems (involving log in, disk quota, windows, path, etc.), you should seek help from the Cory Instructional Account people.
• If you can't get the program running AT ALL, you may want to work with someone else who can get it running on his/her instructional account.
• Note that if you are having system problems, you can still do most of the assignment.
• As usual, make sure to describe the problems you encountered in enough detail that someone trying to correct the problem or alert other students to the problem will know what to do.