Defects4J -- version 0.1.0

Defects4J is a collection of reproducible bugs collected with the goal of advancing software testing research.

The projects

Defects4J contains bugs from the following open-source projects:

Identifier	Project name	Number of bugs
Chart	JFreechart	26
Closure	Closure compiler	133
Lang	Apache commons-lang	65
Math	Apache commons-math	106
Time	Joda-Time	27

The bugs

Each bug has the following properties:

• Issue filed in the corresponding issue tracker, and issue tracker identifier mentioned in the fixing commit message.
• Fixed in a single commit -- manually verified to not include irrelevant changes (e.g., refactorings or feature additions).
• Fixed by modifying the source code (as opposed to configuration files, documentation, or test files).
• A triggering test exists that failed before the fix and passes after the fix -- the test failure is not random or dependent on test execution order.

(B)uggy and (f)ixed program revisions are labelled with <id>b and <id>f, respectively (<id> is an integer).

Requirements

• Java 1.7
• Apache Ant >=1.8
• Perl >= 5.0.10

All bugs have been reproduced and triggering tests verified, using the latest version of Java 1.7. Note that using Java 1.8+ might result in unexpected failing tests on a fixed program version.

Getting started

1. Download the repositories for the projects by running:

   • cd project_repos
   • ./get_repos.sh These are not included in this repository for size purposes and to avoid redundancies.

2. Add Defects4J's executables to your PATH:

   • export PATH=$PATH:"path2defects4j"/framework/bin

3. Check installation and get information for a specific project (commons lang):

   • defects4j info -p Lang

4. Get information for a specific bug (commons lang, bug 1):

   • defects4j info -p Lang -v 1

5. Checkout a buggy source code version (commons lang, bug 1, buggy version):

   • defects4j checkout -p Lang -v 1b -w /tmp/lang_1_buggy

6. Change to the working directory, compile sources and tests, and run tests:

   • cd /tmp/lang_1_buggy
   • defects4j compile
   • defects4j test

7. More examples of how to use the framework are available in test.sh

Publications

• "Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs" René Just, Darioush Jalali, and Michael D. Ernst, ISSTA 2014 [download].

• "Are Mutants a Valid Substitute for Real Faults in Software Testing?" René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser, FSE 2014 [download].

Documentation

Detailed documentation for any script or module can be viewed with: perldoc <file>

Database abstraction

Use framework/bin/defects4j to:

• view configuration of a specific project
• view summary of a specific bug
• checkout and compile faulty or fixed project versions
• run test suite on faulty or fixed project versions
• perform mutation analysis on fixed project versions
• export properties such as classpaths, directories, or lists of tests

Export properties

Use defects4j export -p <property_name> [-o output_file] in the working directory to export a version-specific property:

Property name	Description
classes.modified	Classes (source files) modified by the bug fix
cp.compile	Classpath to compile and run the project
cp.test	Classpath to compile and run the developer-written tests
dir.src.classes	Source directory of classes (relative to working directory)
dir.src.tests	Source directory of tests (relative to working directory)
tests.all	List of all developer-written tests
tests.relevant	List of relevant tests (i.e., tests that touch at least one of the modified classes
tests.trigger	List of tests that trigger (expose) the bug

Test execution framework

The test execution framework provides the following scripts:

• framework/bin/run_bug_detection.pl:
  Determines real fault detection rates of generated test suites and stores the results in a csv-based database. Note that this script requires Perl DBI.

• framework/bin/run_mutation.pl:
  Determines mutation scores of generated test suites and stores the results
  in a csv-based database. Note that this script requires Perl DBI.

• framework/bin/run_coverage.pl:
  Determines code coverage ratios of generated test suites and stores the
  results in a csv-based database. Note that this script requires Perl DBI.

• framework/bin/run_evosuite.pl: Generates test suites using EvoSuite.

• framework/bin/run_randoop.pl: Generates test suites using Randoop.

Directory structure

This is the top-level directory of Defects4J. The directory structure is as follows:

defects4j
   |
   |--- project_repos:     The version control repositories of the provided projects.
   |
   |--- major:             The Major mutation framework.
   |
   |--- framework:         Libraries and executables of the database abstraction and
       |                   test execution framework.
       |
       |--- bin:           Command line interface to Defects4J.
       |
       |--- core:          The modules of the core framework.
       |
       |--- lib:           Libraries used in the core framework.
       |
       |--- util:          Util scripts used by Defects4J.
       |
       |--- projects:      Project-specific resource files

License

MIT License, see license.txt for more information.