From 8daff1b8ce8f73e97b0ba2e68bde7767b65d20ad Mon Sep 17 00:00:00 2001 From: Brittany Johnson Date: Thu, 23 Jan 2020 19:17:45 -0500 Subject: [PATCH 1/9] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 30fd3e4..cef7def 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ collection materials. Our artifact includes the following: 1. A virtual machine file (CausalTesting_Artifact.ova) that contains -the user study setup and version of Holmes used. +the user study setup and source code for the version of Holmes used. 2. The form participants completed as they worked through each defect in the user study (Cause-Identification-Form.pdf). @@ -29,4 +29,4 @@ debriefing following the study (Evaluation-Questionnaire.pdf) All the above listed artifacts, with the exception of the virtual machine file, are located in this repository. -The **INSTALL.md** file provides details on how to locate, set up, and run the virutal machine. \ No newline at end of file +The **INSTALL.md** file provides details on how to locate, set up, and run the virutal machine. From c26e7d95db27d77e955ff262ff731afab19f32ef Mon Sep 17 00:00:00 2001 From: Brittany Johnson Date: Thu, 23 Jan 2020 20:12:21 -0500 Subject: [PATCH 2/9] Update INSTALL.md --- INSTALL.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 3dd27bd..f2fb15d 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -22,23 +22,29 @@ Once the virtual machine is imported, it will appear in your VirtualBox Manager VirtualBox Manager -Once the virtual machine loads, Eclipse will open -- this process may take a few minutes. Once Eclipse opens, there will be 7 files opens labeled *Defect 0- Defect 6*. At the bottom of the window, the Tasks View is open with a list of TODOs. Each TODO is attached to the failing test for each defect. For example, **TODO: Test 00 (Training)** is associated with *Defect_0_Training*. +Once the virtual machine loads, Eclipse will open -- this process may take a few minutes. Once Eclipse opens, there will be 7 projects in the Project Explorer labeled *Defect 0- Defect 6*. +At the bottom of the window, the Tasks View is open with a list of TODOs. Each TODO is attached to the failing test for each defect. For example, **TODO: Test 00 (Training)** is attached to the failing test *test_toBoolean_String()* which exposes *Defect 0* in the *toBoolean()* method. ## Running Holmes To see Holmes output for each defect: 1. Double-click a TODO in the Tasks View to get to the failing test. - * *Defect 0* runs the test generation portion of Holmes (which does not include execution traces, as this was not automated in the user study version of Holmes). - * *Defects 1-6* show the pre-processed output that participants saw during the user study. +Eclipse TODOs -2. Highlight the method that takes the input being tested. Each TODO is followed by comments that specify what we asked participants to do and the method to highlight. +2. Highlight the method that takes the input being tested, as shown below. Each TODO is followed by comments that specify what we asked participants to do and the method to highlight. + +Highlighting target method 3. Right-click the highlighted method and click **"Run Holmes"** in the pop-up menu. +*Defect 0* runs the test generation portion of Holmes (which does not include execution traces, as this was not automated in the user study version of Holmes). + +*Defects 1-6* show the pre-processed output that participants saw during the user study. + If you are running Holmes on *Defect 0*, the editor will automatically go to the top of the file as Holmes generates and executes tests. This process will take a minute or two; eventually the Holmes View will open with results of the execution. If you are running Holmes on *Defects 1-6*, after a few seconds, the Holmes View will open with the pre-processed results used in our user study. For each of these defects, the output includes the original failing test, generated passing tests, and generated faiing tests. You can access the execution trace for each test by clicking the "See Execution Trace" button under the test of interest. From 723d4e40408878b2b65161e077ae59bb51485f85 Mon Sep 17 00:00:00 2001 From: Brittany Johnson Date: Thu, 23 Jan 2020 20:53:41 -0500 Subject: [PATCH 3/9] Update INSTALL.md --- INSTALL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index f2fb15d..066900c 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -30,16 +30,17 @@ At the bottom of the window, the Tasks View is open with a list of TODOs. Each T To see Holmes output for each defect: -1. Double-click a TODO in the Tasks View to get to the failing test. +1. Double-click a TODO in the Tasks View to get to the failing test. Each TODO is followed by comments that specify what we asked participants to do and the method to highlight. Eclipse TODOs -2. Highlight the method that takes the input being tested, as shown below. Each TODO is followed by comments that specify what we asked participants to do and the method to highlight. +2. Highlight the method that takes the input being tested, as shown below. Highlighting target method 3. Right-click the highlighted method and click **"Run Holmes"** in the pop-up menu. +Run Holmes command *Defect 0* runs the test generation portion of Holmes (which does not include execution traces, as this was not automated in the user study version of Holmes). From 4b2ea2b40076f0e7e8d98e90ceaf12e1c253f930 Mon Sep 17 00:00:00 2001 From: Brittany Johnson Date: Thu, 23 Jan 2020 21:46:26 -0500 Subject: [PATCH 4/9] Update INSTALL.md --- INSTALL.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 066900c..0c97112 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -42,10 +42,11 @@ To see Holmes output for each defect: Run Holmes command -*Defect 0* runs the test generation portion of Holmes (which does not include execution traces, as this was not automated in the user study version of Holmes). -*Defects 1-6* show the pre-processed output that participants saw during the user study. - -If you are running Holmes on *Defect 0*, the editor will automatically go to the top of the file as Holmes generates and executes tests. This process will take a minute or two; eventually the Holmes View will open with results of the execution. +**Test 00** runs the test generation portion of Holmes (which does not include execution traces, as this was not automated in the user study version of Holmes). If you are running Holmes on **Test 00**, the editor will automatically go to the top of the file as Holmes generates and executes tests. This process will take a minute or two; eventually the Holmes View will open with results of the execution. -If you are running Holmes on *Defects 1-6*, after a few seconds, the Holmes View will open with the pre-processed results used in our user study. For each of these defects, the output includes the original failing test, generated passing tests, and generated faiing tests. You can access the execution trace for each test by clicking the "See Execution Trace" button under the test of interest. +Test 00 Output + +**Test 01 - Test 06** show the pre-processed output that participants saw during the user study. If you are running Holmes on **Test 01 - Test 06**, after a few seconds, the Holmes View will open with the pre-processed results used in our user study. For each of these defects, the output includes the original failing test, generated passing tests, and generated faiing tests. You can access the execution trace for each test by clicking the **"See Execution Trace"** button under the test of interest. + +Run Holmes command From f69d9c9cb7a7e499b0cae60354fe1175a8f2e162 Mon Sep 17 00:00:00 2001 From: Brittany Johnson Date: Thu, 23 Jan 2020 23:43:19 -0500 Subject: [PATCH 5/9] Update INSTALL.md --- INSTALL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 0c97112..bc449eb 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -10,7 +10,7 @@ Download this file to your computer. Make sure you make note of where you put th Before you can run the virtual machine, you will need to download and install [VirtualBox 6](https://www.virtualbox.org). -Once you've installed VirtualBox, you're reading to load and run the virtual machine. +Once you've installed VirtualBox, you're ready to load and run the virtual machine. To load the virtual machine, you will need to import the **CausalTesting_Artifact.ova** file into VirtualBox. You do so as follows: 1. Open VirtualBox. @@ -30,7 +30,8 @@ At the bottom of the window, the Tasks View is open with a list of TODOs. Each T To see Holmes output for each defect: -1. Double-click a TODO in the Tasks View to get to the failing test. Each TODO is followed by comments that specify what we asked participants to do and the method to highlight. +**1. Double-click a TODO in the Tasks View to get to the failing test that exposes the defect.** +Each TODO is followed by comments that specify what we asked participants to do and the method to highlight for executing Holmes. Eclipse TODOs From be21dcee2df000b562436cc20c0add80bfe18c28 Mon Sep 17 00:00:00 2001 From: Brittany Johnson Date: Thu, 23 Jan 2020 23:51:38 -0500 Subject: [PATCH 6/9] Update README.md --- README.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index cef7def..1e25856 100644 --- a/README.md +++ b/README.md @@ -12,21 +12,20 @@ Our artifact includes the following: 1. A virtual machine file (CausalTesting_Artifact.ova) that contains the user study setup and source code for the version of Holmes used. -2. The form participants completed as they worked through -each defect in the user study (Cause-Identification-Form.pdf). +2. Instructions for obtaining and running Holmes on the virtual machine ([INSTALL.md](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/INSTALL.md)). -3. The post-questionnaire participants filled out as a form of -debriefing following the study (Evaluation-Questionnaire.pdf) +3. The form participants completed as they worked through +each defect in the user study ([Cause-Identification-Form.pdf](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/Cause-Identification-Form.pdf)). -4. The demographic questionnaire used to gather information about our - participants background and experience (Demographic-Questionnaire.pdf). +4. The post-questionnaire participants filled out as a form of +debriefing following the study ([Evaluation-Questionnaire.pdf](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/Evaluation-Questionnaire.pdf)) -5. The script used to explain to participants what they would be doing - during the user study (User-Study-Script.pdf). +5. The demographic questionnaire used to gather information about our + participants background and experience ([Demographic-Questionnaire.pdf](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/Demographic-Questionnaire.pdf)). + +6. The script used to explain to participants what they would be doing + during the user study ([User-Study-Script.pdf](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/User-Study-Script.pdf)). ## Where can I obtain the artifact components? All the above listed artifacts, with the exception of the virtual machine file, are located in this repository. - - -The **INSTALL.md** file provides details on how to locate, set up, and run the virutal machine. From 2f43b0b315751b7b946d2b15be329c0799dd98e0 Mon Sep 17 00:00:00 2001 From: Brittany Johnson Date: Thu, 23 Jan 2020 23:53:47 -0500 Subject: [PATCH 7/9] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1e25856..7f83c5b 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ This is the artifact for the paper *"Causal Testing: Understanding Defects' Root Causes"*. This artifact replicates the user study setup used to evaluate our technique, including the supplementary data -collection materials. +collection materials and prototype source code. ## What is included in the artifact? From ee98c642df46311c2a0214a41e44a0edbb25f3f0 Mon Sep 17 00:00:00 2001 From: Brittany Johnson Date: Fri, 24 Jan 2020 10:27:05 -0500 Subject: [PATCH 8/9] Update CONTACT.md --- CONTACT.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/CONTACT.md b/CONTACT.md index 9b37c9b..2297839 100644 --- a/CONTACT.md +++ b/CONTACT.md @@ -1,13 +1,21 @@ # Contact Information **Brittany Johnson** (corresponding author) + *email*: bjohnson@cs.umass.edu + *github id*: brittjay0104 + **Yuriy Brun** + *email*: brun@cs.umass.edu + *github id*: brunyuriy + **Alexandra Meliou** + *email*: ameli@cs.umass.edu -*github id*: \ No newline at end of file + +*github id*: From 1b831669a8d65f13539284614cd832955a03eb49 Mon Sep 17 00:00:00 2001 From: Brittany Johnson Date: Fri, 24 Jan 2020 10:47:30 -0500 Subject: [PATCH 9/9] Update README.md --- README.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 7f83c5b..0463cb7 100644 --- a/README.md +++ b/README.md @@ -14,18 +14,26 @@ the user study setup and source code for the version of Holmes used. 2. Instructions for obtaining and running Holmes on the virtual machine ([INSTALL.md](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/INSTALL.md)). -3. The form participants completed as they worked through +3. Contact information for the authors ([CONTACT.md](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/CONTACT.md)). + +4. The license for our materials and prototype ([LICENSE.md](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/LICENSE.md)). + +5. A description of what artifact badges we believe we support and why ([STATUS.md](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/STATUS.md)). + +6. The form participants completed as they worked through each defect in the user study ([Cause-Identification-Form.pdf](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/Cause-Identification-Form.pdf)). -4. The post-questionnaire participants filled out as a form of +7. The post-questionnaire participants filled out as a form of debriefing following the study ([Evaluation-Questionnaire.pdf](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/Evaluation-Questionnaire.pdf)) -5. The demographic questionnaire used to gather information about our +8. The demographic questionnaire used to gather information about our participants background and experience ([Demographic-Questionnaire.pdf](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/Demographic-Questionnaire.pdf)). -6. The script used to explain to participants what they would be doing +9. The script used to explain to participants what they would be doing during the user study ([User-Study-Script.pdf](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/User-Study-Script.pdf)). +10. The pdf of our technical research paper *"Causal Testing: Understanding Defects' Root Causes* ([ICSE2020-paper.pdf](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/ICSE2020-paper.pdf)). + ## Where can I obtain the artifact components? -All the above listed artifacts, with the exception of the virtual machine file, are located in this repository. +All the above listed artifacts, with the exception of the virtual machine file, are located in this repository. A link to the virtual machine file can be found in [INSTALL.md](https://github.com/LASER-UMASS/CausalTesting-Artifact_ICSE20/blob/master/INSTALL.md).