Introduction to Remote Testing View

8/29/2004

 

Remote Testing View is an eclipse plug-in developed to display the results of the testcases run on a remote server server. The basic idea is this, students not only check-in their source code but also the testcases into CVS. Every couple hours, the remote server will checkout the testcases and run them against both the student's project and the reference solution. The result will then be checked in to the students' CVS repository. When the students synchronize with CVS next time, they will get two data files that contain the results of the testcases. Remote Testing View will look into the tests/ directory and display the results in a view as part of eclipse. The frequency that the remote testing runs may vary as needed.

Install the plug-in

  1. Make you sure have Eclipse 3.0 installed
  2. Download the plugin
  3. Unzip it to the plugin to the eclipse directory
  4. There is already a plugins/ directory in the eclipse's installation path. Make sure edu.berkeley.cs164.plugins.remote_testing_1.0.0 is in there.
  5. Start Eclipse

Start the plug-in

  1. Click Windows menu
  2. Click Show View submenu
  3. Click Other
  4. Expand CS164
  5. Choose Remote Testing
  6. Click OK.

Read the result

Below is a screen-shot of the Remote Testing view.  This screen-shot will be used as an example as we walk through different features of the plug-in.

 

  1. Description of the columns
    1. File Name - the test case file name
    2. Coverage - percentage of branches exercised in the reference solution with the current testcase
    3. Passed -  represents whether this test case is considered passed. NOTE: If you suspect that the results of this column contradict the specification, please notify the GSIs.
    4. Your Result - a textual description of the result after running your project
    5. Reference Result - a textual description of the result after running the reference solution
    6. Your Output -  the output of your project
    7. Reference Output - the output of the reference solution

  1. Description of the rows
    1. There are three levels in the plugin. They are the top tests-level, testcase-level and the input-file-level.
    2. tests-level displays the overall result of the project. Notice the Coverage column of this level is NOT the sum of the coverage of each individual test case. Since we the coverage measures the percentage of the branches exercised and some branches may be exercised in more than one testcase, those branches will only factor into the coverage percentage once.
    3. testcase-level displays the result of that particular testcase file. (e.g. sample.decaf)
    4. input-file-level is not fully used in PA1 because there is no input to your decaf program. In PA1, a "No input" entry is created to indicate the result of running the parent testcase without an input file. For example, the ¡°No input¡± row under sample.decaf is representing the result of pretty printing and interpreting sample.decaf without external input. In PA1, the testcase level and input level may seem to show the same kind of information, the later PAs will show their difference
  2. Examples

a.      arithmetic.decaf

                                                              i.      This testcase exercised 32.14% of the branches in the reference solution.

                                                            ii.      Both the student solution and the reference solution FINISHED execution.

                                                          iii.      The "No Input" row says it failed.

                                                          iv.      That is because the student's output and the reference's output don't match.

b.      dividebyzero.decaf

                                                            v.      This testcase exercised 29.76% of the branches in the reference solution.

                                                          vi.      Just as above, both solutions FINISHED execution.

                                                        vii.      Since the student's output and the reference's output match.

                                                      viii.      This testcase is considered passed.

c.      illegal.decaf

                                                          ix.      8.33% of the branches are executed

                                                            x.      Both student solution and the reference solution recognized the decaf program contains SYNTAX ERROR

                                                          xi.      This testcase is considered passed regardless whether the output match or not.

  1. Buttons

You can right click in the plug-in to do some manipulation with the view. "Show All Testcase Results" will expand the tree to the level where all testcase files are shown but not the input files. This gives you a concise view to see how many of them passed. "Show All Input Results" will expand the tree to the input files level including the testcase files. This representation lets you see the output of both of your project and the reference solution. On the top right corner of the view, there is a "Refresh" button that you can use to refresh the view.