IMPORTANT NOTE: Make sure you understand the backpropagation algorithm (as covered in lecture, section, the handout, and course readings) BEFORE you begin coding. A good simple walkthrough of backpropagation can be found here.

Part 1:    Due Wednesday, February 15th, by 11:59pm - this part will not be accepted more than 2 days late introduction    -    part 1    -    part 2

Implementing backpropagation: For this part you will complete a simple neural network with two input nodes and a single output node. The provided code already sets up a three-node net for you, however you are tasked with supporting feedforward activation and backpropagation learning. You should make the following assumptions: Input and output patterns will be contained in an int[][] (an array of patterns, each of which is an array of integers). The output unit uses the sigmoid activation function and should also have a bias. The network has some training parameters that you may wish to vary during experimentation as appropriate. Be sure to include both learningRate and momentum in your weight updates. Weights should be updated on a per-pattern basis (i.e., you needn't implement batch learning). Copy the files Net.java and Unit.java and TesterPart1.java. Take some time to examine the Net, Unit and TesterPart1 classes they define respectively. Each contains a number of methods, most of which are marked as ** to be filled in **. You are required to implement the methods in Net.java and Unit.java. The third file, TesterPart1.java, is provided to help you test and debug the project. The autograder will use the same format to create, train, and test your network (with more expansive test cases). Feel free to add additional test cases, however do not add any critical code in TesterPart1.java as it will not be submitted. Note: all code must work with Java 1.4.1 (i.e. don't use fancy functions from the 1.5.0 Beta.) Study the provided code and understand how the network and data structures are being used. < /tr> Net.java already creates and links 4 units in the appropriate fashion: 2 input nodes, an output node, and a bias node. (Click on the figure to enlarge.) When testing how your network learns, the autograder only makes 2 functional calls to your code. The first is Net n = new Net(data, targets); to create the network, followed by n.train(); to train the network. The functions setTrainingParameters(...) and toString() may also be called but both of these are already written for you. Notice that toString() assumes that outUnit.inWeights[0] is the weight from inUnit1 to outUnit, while outUnit.inWeights[2] is the weight from the bias to outUnit. Implement the backpropagation algorithm. There are 8 functions that need to be filled in. If you don't know where to start, try completing the functions in the same order that the test file checks them: in Unit.java: initialize(): Randomize all incoming weights computeActivation(): Apply sigmoid function to weighted sum of inputs computeError(): Compute error for the output node computeWeightChange(): Calculate the current weight change updateWeights(): Update changes to weights for this pattern in Net.java: feedforward(): Present pattern and compute activations for rest of net computeError(): Present all patterns to network and calculate current error train(): Train the net according to the current training parameters Test and Comment your Program. Make sure to comment your code clearly. If your code is wrong and your comments are specific enough (don't write a book) at least we will know your intent. There are two easy ways to test your program after you have compiled all of your code ( % javac -g *.java ). Call TesterPart1 with % java TesterPart1 to run a few very elementary tests on your various functions. Although they can't guarantee your functions are correct, they will at least make sure you are not making a silly error. If you fail one of the earlier tests, it will also cause you to fail later tests. Train your network with various parameters, making sure sure that changing the momentum, learning rate, etc... has the expected effect. Make the call: % java TesterPart1 ftbl ne lr mom ec where: ftbl is the function to be learned (either "AND", "OR" or "SAME")   -   of course same won't be learned yet ne is the number of epochs lr is the learning rate mom is the momentum ec is the error criterion for example (with arbitrary numbers for ne, lr, mom, ec): % java TesterPart1 AND 500 .1 .5 .1 What to submit for Part 1 You should submit your assignment using the submit program available on the EECS instructional system; instructions for doing this can be found here. This assignment is a3-1. Be sure to include: Your completed code, including Net.java and Unit.java. Remember to submit any other code files you have created. Do not submit the .class files. Answers to these questions, in a file called answers.txt. Make sure this includes the usual information. Under what conditions is a network with hidden layers computationally identical to one without any hidden layers? How would you change Part 1 of this assignment? was it too difficult? too easy? (If relevant) What kinds of problems did you encounter? Describe any design problems you may have had. If your network doesn't work, where is the problem? If you weren't able to get everything working or ran out of time, explain your approach, what you know, and what you might do differently next time. Note: If you cheat, we will catch you. Don't cheat!

Part 2:    Due Wednesday, February 22th, by 11:59pm introduction    -    part 1    -    part 2

Expanding the Network: In this part you will be expanding your neural network to include an arbitrary number of input, hidden, and output units. In the first part, four units were hard coded into the Net class: two input units, one output unit, and one bias unit. Now that you understand the back-propagation algorithm, it is time to add hidden layers. You must overhall the internal structure and make important architectural decisions. Copy the file TesterPart2.java. If you don't like your solution to part1, also copy the updated files Net.java and Unit.java. The files will be updated the morning of Friday, February 17th. However if you want the files earlier, send an e-mail to makin+cs182@eecs.berkeley.edu. Decide what data structures you will need to supplement the ones supplied. Issues you will need to consider: How are connections between units represented? Will you just use the knowledge that one layer is completely connected to the next layer, or will you explicitly add input and output connections as in Part 1? Are input/hidden/output units represented differently? How do you distinguish between them? Will you keep the bias Unit from part 1? or do you plan on giving each unit its own bias property? How will you make sure that the propagation of activation (feedforward) and propagation of error (backpropagation) are done in the right order? Start by updating the Net.java constructor To accommodate the additional flexibility the Net constructor will need to be passed 5 arguments instead of the original 2. The header for the new constructor follows: (note: you may want to overload the constructor instead of replacing it) You may assume that neither hLayers nor nHid will be 0. /** * Constructor for Net class. * * Create Net with the specified architecture and data. * * @param in Number of input units * @param hLayers Number of hidden layers * @param nHid Number of Hidden units per layer * @param out Number of output units * @param patterns Array of input patterns (each an array of int) * @param targets Array of output patterns (each an array of int) */ public Net (int in, int hLayers, int nHid, int out, int[][] patterns, int[][] targets) {    // ** to be filled in ** // } You may also need to update functions associated with network construction. It is at this step that the bulk of you architectural decisions must be made, including how to store the arbitrary number of input, hidden, and output nodes. It may be easier to start with only a single hidden layer. (If you end up only being able to implement one hidden layer you won't lose many points.) Update toString() function in Net.java This is a very important step, as this output may be a primary means of grading your project. Make sure to include the epoch number, current error, and the weights coming into each unit. Make this output as viewer friendly as possible. Update feedforward activation. Once you've implemented feedforward, you will have a good idea if you've made the right decisions and have correctly set up your network. Update your backpropagation. Although most of the changes are topical (from explicit unit calls to iteration over an arbitrary number of units) the addition of a hidden layer may cause a few problems. Hidden units calculate error differently from output units. When there are hidden units, you must be careful about the order of units in which error is calculated. Update the Train() function This is most likely where you will run into problems with order. Once Again, Test and Comment your Program. Make sure to comment your code clearly. If your code is wrong and your comments are specific enough (don't write a book) at least we will know your intent. There are two easy ways to test your program after you have compiled all of your code ( % javac -g *.java ). you can call TesterPart2 with % java TesterPart2 to run a few very elementary tests on your various functions. Although they can't guaranty your functions are correct, they will at least make sure you are not making a silly error. If you fail one of the earlier tests, it will also cause you to fail later tests. To train your network with various parameters make the call: % java TesterPart2 ftbl ne lr mom ec where: ftbl is the function to be learned (either "AND", "OR, "SAME", "AUTO8", or "MULTI") ne is the number of epochs lr is the learning rate mom is the momentum ec is the error criterion for example (with arbitrary numbers for ne, lr, mom, ec): % java TesterPart1 AND 500 .1 .5 .1 Your program should now be able to correctly learn all 5 of these functions. What to submit for Part 2 You should submit your assignment using the submit program available on the EECS instructional system; instructions for doing this can be found here. Be sure to include: Your completed code, including Net.java and Unit.java. Remember to submit any other code files you have created. Do not submit the .class files. Answers to these questions, in a file called answers.txt. Make sure this includes the usual information. How would you change Part 2 of this assignment? was it too difficult? too easy? (If relevant) What kinds of problems did you encounter? Describe any design problems you may have had. If your network doesn't work, where is the problem? If you weren't able to get everything working or ran out of time, explain your approach, what you know, and what you might do differently next time. Note: If you cheat, we will catch you. Don't cheat!