You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: artifact_documentation/README.md
+56-16
Original file line number
Diff line number
Diff line change
@@ -28,49 +28,89 @@ All the above listed artifacts, with the exception of the virtual machine file,
28
28
5. Find and select the downloaded virtual machine file (CausalTesting_Artifact.ova). Click **"Continue"**.
29
29
6. Leave all the settings as they are and click **"Import"**.
30
30
31
-
Once the virtual machine is imported, it will appear in your VirtualBox Manager as **CausalTesting_Artifact**. You can now start the virtual machine by clicking the green **"Start"** arrow at the top of the VirtualBox Manager (see screenshot below).
If the virtual machine loads and you find it is not an appropriate size, you can find the size that works for you by doing the following:
39
40
41
+
1. In your VirtualBox VM menu, go to **View > Virtual Screen 1 > ...**. You will see different scaling options; select the one that best suites your screen.
2. Your menu may look different if you are running a different operating system. However, if given percentages to re-scale they will have the same effect.
42
46
47
+
### Waking up the virtual machine
43
48
49
+
If you leave the virtual machine and return to a black screen or screen saver, press any key on your keyboard to wake up the virtual machine.
44
50
51
+
## Loading and navigating Eclipse
45
52
53
+
Once the virtual machine loads, Eclipse will open. In Eclipse, there will be 8 projects loaded on the left side in the Project Explorer. Seven of the projects are from the [Defects4J](https://github.com/rjust/defects4j) defect benchmark; these projects are labeled *Defect_0_Training*, *Defect_1*, *Defect_2*, *Defect_3*, *Defect_4*, *Defect_5*, and *Defect_6*. The eighth project is *Holmes*.
Once the virtual machine loads, Eclipse will open -- this process may take a few minutes. Once Eclipse opens, there will be 7 projects in the Project Explorer labeled *Defect 0- Defect 6*.
57
+
At the bottom of the window, the Tasks View is open with a list of TODOs. Each TODO maps to a failing test in its respective project that exposes a defect in that project's source code.
58
+
For example **TODO: Test 00 (Training)** maps to a failing test in the project *Defect_0_Training* that exposes a defect.
49
59
50
-
At the bottom of the window, the Tasks View is open with a list of TODOs. Each TODO is attached to the failing test for each defect. For example, **TODO: Test 00 (Training)** is attached to the failing test *test_toBoolean_String()* which exposes *Defect 0* in the *toBoolean()* method.
60
+
You may get a "Low Disk Space" warning -- you can click "Ignore". Running Holmes doesn't require a significant amount of memory.
51
61
52
-
## Running Holmes
62
+
## Holmes runtime
53
63
54
-
To see Holmes output for each defect:
64
+
The runtime for Holmes varies depending on a number of factors, such as input value, input difference threshold, and project size. The current version of Holmes works on tests for single parameter methods that take either a primitive type or String. To replicate Holmes' runtime when generating tests, do the following:
55
65
56
-
**1. Double-click a TODO in the Tasks View to get to the failing test that exposes the defect.**
57
-
Each TODO is followed by comments that specify what we asked participants to do and the method to highlight for executing Holmes.
66
+
1. Double-click the TODO labeled **TODO: Test 00 (Training)**. Within a few seconds, the file *BooleanUtilsTest.java* opens at the ```test_toBoolean_String()``` method. Inside the ```test_toBoolean_String()``` method is the failing test ```assertEquals(false, BooleanUtils.toBoolean("tru");```.
3. Right-click the highlighted method and click **"Run Holmes"** in the pop-up menu.
74
+
3. Right-click the highlighted method and click **"Run Holmes"** in the pop-up menu (shown below). The editor will automatically go to the top of the file and some dialog windows may pop up as Holmes generates and executes tests. This process will take a minute or two; to reduce the chances of Holmes or the virtual machine hanging, we selected a defect for which Holmes is quickly able to find similar passing tests.
**Test 00** runs the test generation portion of Holmes (which does not include execution traces, as this was not automated in the user study version of Holmes). If you are running Holmes on **Test 00**, the editor will automatically go to the top of the file as Holmes generates and executes tests. This process will take a minute or two; eventually the Holmes View will open with results of the execution.
78
+
Eventually, a view labeled "Holmes View" will open at the bottom of the screen with the results of the execution (as shown below).
**Test 01 - Test 06** show the pre-processed output that participants saw during the user study. If you are running Holmes on **Test 01 - Test 06**, after a few seconds, the Holmes View will open with the pre-processed results used in our user study. For each of these defects, the output includes the original failing test, generated passing tests, and generated faiing tests. You can access the execution trace for each test by clicking the **"See Execution Trace"** button under the test of interest.
82
+
Now let's see how we can use Causal Testing to debug.
83
+
84
+
## Debugging with Holmes
85
+
86
+
Holmes is a prototype implementation of our novel testing technique, Causal Testing. Causal Testing conducts causal experiments, which involves perturbing test inputs to find passing executions that are similar to the failing execution, to help developers understand why a given test is failing and how to fix it.
87
+
88
+
To see how you can use Causal Testing to debug a failing test, we first need to produce the output:
89
+
90
+
1. Double-click the TODO labled **TODO: Test 01**. This will open the file *StringEscapeUtilsTest.java* at the ```testEscapeJavaWithSlash()``` test method. Inside the test method is the following failing test:
91
+
92
+
```
93
+
String input = "String with a slash (/) in it";
94
+
final String expected = input;
95
+
final String actual = StringEscapeUtils.escapeJava(input);
96
+
97
+
assertEquals(expected, actual);
98
+
```
99
+
100
+
2. For this test, the method call being tested is ```escapeJava(input)```. Therefore, to invoke Holmes we want to double-click to highlight ```escapeJava```, as shown below.
3. Right click the highlighted method and select **Run Holmes** from the pop-up menu. The output will appear at the bottom of the screen in the "Holmes View"
105
+
106
+
<imgsrc="https://drive.google.com/uc?id=1SsHbQei8HjLDAuWMpthbxDubzzJiJhwW"alt="Run Holmes Test 02"width="700"height="500"/>
107
+
108
+
109
+
Now that we have the Causal Testing results, we can begin to debug the defect.
110
+
111
+
First, we can see that Holmes has provided three similar passing tests and three similar failing tests. Just from looking at the inputs to the tests that pass and the tests that fail we can see that, like the original failing test, all the additional failing tests include the ```/``` character while the tests that pass do not.
112
+
113
+
Second, we can see that each generated test has a button under it labeled "See Execution Trace". Clicking this button opens a minimized trace of the execution; clicking the button again hides the trace.
0 commit comments