INTRODUCTION

IMPORTANT NOTE: Make sure you understand the backpropagation algorithm (as covered in lecture, section, the handout, and course readings) BEFORE you begin coding. A good, simple walkthrough of backpropagation can be found here.

Part 1:    Due Thursday, February 21st, by 11:59pm - this part will not be accepted more than 2 days late introduction    -    part 1    -    part 2

Implementing backpropagation: For this part you will complete a simple neural network with two input nodes and a single output node. The provided code already sets up a three-node net for you, however you are tasked with supporting feedforward activation and backpropagation learning. You should make the following assumptions: Input and output patterns will be contained in an int[][] (an array of patterns, each of which is an array of integers). The output unit uses the sigmoid activation function and should also have a bias. The network has some training parameters that you may wish to vary during experimentation as appropriate. Be sure to include both learningRate and momentum in your weight updates. Weights should be updated in one of two ways: on a per-pattern basis (stochastic) or on a per-epoch basis (batch). You should implement both; your network will be told which to do via the trainType parameter. Copy the starter code (zip). Take some time to examine the Net, Unit and TesterPart1 classes they define respectively. Each contains a number of methods, most of which are marked as TODO. You are required to implement the methods in Net.java and Unit.java. The third file, TesterPart1.java, is provided to help you test and debug the project, as well as interfacing with the autograder. Note: all code must work with Java 1.5 (i.e. don't use new stuff from version 6.) Note that the starter code includes two JAR files. One JAR file includes a standard unit test library that I use for running tests. The other includes an auto-checker that will check the results of your code. Study the provided code and understand how the network and data structures are being used. Net.java already creates and links 4 units in the appropriate fashion: 2 input nodes, an output node, and a bias node. (Click on the figure to enlarge.) When testing how your network learns, the autograder will inspect the data structures of your Units, so please use them! Also, note that the order of the weights is important: the bias unit weight should always be last. Implement the backpropagation algorithm. There are many functions that need to be filled in. If you don't know where to start, try completing the functions in the same order that the test file checks them: in Unit.java: initialize(): Randomize all incoming weights to values chosen uniformly between -1 and 1 (the values of NetParams.minWeight and NetParams.maxWeight). Also, construct data structures for weights and weight changes. computeActivation(): Apply sigmoid function to weighted sum of inputs computeError(): Compute error and delta for the output node computeStochasticWeightChange(): Calculate the current weight change in the stochastic case accumulateWeightChange(): Add in the current weight change in the batch case resetAccumulators(): Reset the accumulators in the batch case computeBatchWeightChange(): Calculate the total weight change in the batch case updateWeights(): Update changes to weights for this pattern computeWeightChangeMomentum(): Calculate momentum term in weight updates in Net.java: feedforward(): Present pattern and compute activations for rest of net computeError(): Present all patterns to network and calculate current error train(): Train the net according to the current training parameters. computeError(): Present all patterns to network and calculate current error Test and Comment your Program. Make sure to comment your code clearly. If your code is wrong and your comments are specific enough (don't write a book) at least we will know your intent. You may be graded in part on the basis of your comments! I belive in code readability. There are two easy ways to test your program after you have compiled all of your code ( % javac -classpath=$CLASSPATH:.:a3-p1-tests.jar:junit.jar -g *.java ). Call TesterPart1 with % java TesterPart1 to run a battery of tests on your various functions. Although they can't guarantee your functions are correct, they will at least make sure you are not making a silly error. If you fail one of the earlier tests, it will also cause you to fail later tests. Train your network with various parameters, making sure sure that changing the momentum, learning rate, etc... has the expected effect. Make the call: % java TesterPart1 training_file ne lr mom ec trainType where: training_file is the training file containing the function to be learned (either "and.data", "or.data", "xor.data", or "same.data")   -   of course xor and same won't be learned yet ne is the number of epochs lr is the learning rate mom is the momentum ec is the error criterion trainType is the training style, either Batch or Stochastic for example (with arbitrary numbers for ne, lr, mom, ec): % java TesterPart1 and.data 500 .1 .5 .1 Batch What to submit for Part 1 You should submit your assignment using the submit program available on the EECS instructional system; instructions for doing this can be found here. This assignment is a3-1. Be sure to include: Your completed code, including Net.java and Unit.java. Remember to submit any other code files you have created. Do not submit the .class files. Answers to these questions, in a file called answers.txt. Make sure this includes the usual information. Under what conditions is a network with hidden layers computationally identical to one without any hidden layers? How would you change Part 1 of this assignment? was it too difficult? too easy? (If relevant) What kinds of problems did you encounter? Describe any design problems you may have had. If your network doesn't work, where is the problem? If you weren't able to get everything working or ran out of time, explain your approach, what you know, and what you might do differently next time. Note: If you cheat, we will catch you. Don't cheat!

Part 2:    Due Thursday, February 28th, by 11:59pm introduction    -    part 1    -    part 2

Expanding the Network: In this part you will be expanding your neural network to include an arbitrary number of input, hidden, and output units. In the first part, four units were hard coded into the Net class: two input units, one output unit, and one bias unit. Now that you understand the back-propagation algorithm, it is time to add hidden layers. You must overhaul the internal structure and make important architectural decisions. Get the starter code here (zip). If you don't like your solution to part1, you might want to use the updated files, Net.java and Unit.java. The files will be updated the morning of Sunday, February 24th. However if you want the files earlier, send an e-mail to . Start by updating the Net.java constructor You may also need to update functions associated with network construction. Update feedforward activation. Once you've implemented feedforward, you will have a good idea if you've made the right decisions and have correctly set up your network. Update your backpropagation. Although most of the changes are topical (from explicit unit calls to iteration over an arbitrary number of units) the addition of a hidden layer may cause a few problems. Hidden units calculate error differently from output units. When there are hidden units, you must be careful about the order of units in which error is calculated. Update the train() function This is most likely where you will run into problems with order. Once again, test and comment your program. Make sure to comment your code clearly. If your code is wrong and your comments are specific enough (don't write a book) at least we will know your intent. You may be graded on the readability of your comments and code. There are two easy ways to test your program after you have compiled all of your code ( % javac -classpath=$CLASSPATH:.:a3-p2-tests.jar:junit.jar -g *.java ). you can call TesterPart2 with % java TesterPart2 to run a few very elementary tests on your various functions. These tests are much less sophisticated than for Part 1, since we know much less about exactly what data structures you are using. Instead of testing particular functions, we simply test to make sure that the network learns appropriately. To train your network with various parameters make the call: % java -classpath $CLASSPATH:.:a3-p2-tests.jar:junit.jar starterCode.TesterPart2 ne lr mom ec training_file hidden_layers nhid trainType where: training_file is the file containing the function to be learned (either "and.data", "or.data, "xor.data", "same.data", "auto8.data", or "multi.data") ne is the number of epochs lr is the learning rate mom is the momentum ec is the error criterion hidden_layers is the number of hidden layers nhid is the number of units in each hidden layer trainType is the training type, either Batch or Stochastic for example (with arbitrary numbers for ne, lr, mom, ec): % java -classpath $CLASSPATH:.:a3-p2-tests.jar:junit.jar starterCode.TesterPart2 same.data 500 .1 .5 .1 1 2 You can run a more complete test by running runTest2.sh, which will run your network on a set of different training examples. It will produce a bunch of output files, and also produce a tarball output2.tgz. You can then check that your output is correct by running ~cs182/sp07/assignments/a3/runCheck2.sh. This will both check that your output is correct and that your calculations are correct. Your program should now be able to correctly learn all 6 of these functions. What to submit for Part 2 You should submit your assignment using the submit program available on the EECS instructional system; instructions for doing this can be found here. Be sure to include: Your completed code, including Net.java and Unit.java. Remember to submit any other code files you have created. Do not submit the .class files. Answers to these questions, in a file called answers.txt. Make sure this includes the usual information. How would you change Part 2 of this assignment? was it too difficult? too easy? (If relevant) What kinds of problems did you encounter? Describe any design problems you may have had. If your network doesn't work, where is the problem? If you weren't able to get everything working or ran out of time, explain your approach, what you know, and what you might do differently next time. Note: If you cheat, we will catch you. Don't cheat!