NPEFix Fails to Run on the Defects4J Dataset

[Peanu11]

Dear NPEFix's maintainers,
Hello! I recently attempted to run NPEFix on the Defects4J dataset, but it failed on NPE examples from the Math, Chart, Mockito, and Lang etc. projects. The error messages on Chart983 and Mockito188 are shown in the images below.

Here’s how I used it:
java -jar "$NPEFIX_JAR" --source "$src_dir:$test_dir" --classpath "$classpath" --test "$test_classes"
Where:

• src_dir=$(defects4j export -p dir.src.classes)
• test_dir=$(defects4j export -p dir.src.tests)
• classpath=$(defects4j export -p cp.test)
• test_classes is a colon-separated list generated from defects4j export -p tests.relevant.
  I noticed that the usage instructions in the README and the jar example for reproducing experiments are no longer compatible with the current version. Could you tell me if there are any issues with my usage or parameter settings? Or could you provide an example that works?

[monperrus]

Dear @Peanu11

Thanks for your bug report.

Which Java version do you use for running NpeFix?

Which Java version do Chart983 and Mockito188 expect/support?

[Peanu11]

Dear @monperrus
Thanks for your reply.
I run NpeFix with openjdk version "17.0.12".
I run defects4j(Chart983 and Mockito188) with openjdk version "11.0.24".

[monperrus]

They are reasonable versions. To save time, would you copy paste the exact sequence of commands (incl. the defects4J ones) that result in the bug?

[Peanu11]

Dear @monperrus ,
1.Here's my commands to build npefix.

git clone https://github.com/SpoonLabs/npefix.git
cd npefix
mvn clean compile assembly:single

2.Here's my commands to build defects4j.

git clone https://github.com/rjust/defects4j.git
cd defects4j
cpanm --installdeps .
./init.sh

1.Here's my instruction to get Chart983 from defects4j (Chart983 is numbered 1 in defects4j).In this step you should first adjust the java version to java 11 via update-alternatives --config java, and replace the path of the first instruction with the path of your defects4j.

export PATH=$PATH:/home/defects4j/framework/bin
defects4j checkout -p "Chart" -v "1b" -w "/tmp/chart-983-buggy"
cd /tmp/chart-983-buggy/
defects4j compile

You can get Mockito188 via defects4j checkout -p "Mockito" -v "3b" -w "/tmp/Mockito-188-buggy"
2.
Here's the instruction I ran npefix.In this step you should first adjust the java version to java 17.

src_dir=$(defects4j export -p dir.src.classes)
test_dir=$(defects4j export -p dir.src.tests)
classpath=$(defects4j export -p cp.test)
export NPEFIX_JAR="/home/npefix/target/npefix-0.9-SNAPSHOT-jar-with-dependencies.jar"
java -jar "$NPEFIX_JAR" \
    --test org.jfree.chart.axis.junit.LogAxisTests:org.jfree.chart.axis.junit.NumberAxisTests:org.jfree.chart.axis.junit.SubCategoryAxisTests:org.jfree.chart.axis.junit.ValueAxisTests:org.jfree.chart.junit.AreaChartTests:org.jfree.chart.junit.BarChart3DTests:org.jfree.chart.junit.BarChartTests:org.jfree.chart.junit.ChartPanelTests:org.jfree.chart.junit.GanttChartTests:org.jfree.chart.junit.JFreeChartTests:org.jfree.chart.junit.LineChart3DTests:org.jfree.chart.junit.LineChartTests:org.jfree.chart.junit.PieChart3DTests:org.jfree.chart.junit.PieChartTests:org.jfree.chart.junit.ScatterPlotTests:org.jfree.chart.junit.StackedAreaChartTests:org.jfree.chart.junit.StackedBarChart3DTests:org.jfree.chart.junit.StackedBarChartTests:org.jfree.chart.junit.StandardChartThemeTests:org.jfree.chart.junit.TimeSeriesChartTests:org.jfree.chart.junit.WaterfallChartTests:org.jfree.chart.junit.XYAreaChartTests:org.jfree.chart.junit.XYBarChartTests:org.jfree.chart.junit.XYLineChartTests:org.jfree.chart.junit.XYStepAreaChartTests:org.jfree.chart.junit.XYStepChartTests:org.jfree.chart.plot.junit.CategoryPlotTests:org.jfree.chart.plot.junit.CombinedDomainCategoryPlotTests:org.jfree.chart.plot.junit.CombinedRangeCategoryPlotTests:org.jfree.chart.plot.junit.MultiplePiePlotTests:org.jfree.chart.plot.junit.PiePlot3DTests:org.jfree.chart.plot.junit.PiePlotTests:org.jfree.chart.plot.junit.XYPlotTests:org.jfree.chart.renderer.category.junit.AbstractCategoryItemRendererTests:org.jfree.chart.renderer.category.junit.AreaRendererTests:org.jfree.chart.renderer.category.junit.BarRenderer3DTests:org.jfree.chart.renderer.category.junit.BarRendererTests:org.jfree.chart.renderer.category.junit.BoxAndWhiskerRendererTests:org.jfree.chart.renderer.category.junit.CategoryStepRendererTests:org.jfree.chart.renderer.category.junit.DefaultCategoryItemRendererTests:org.jfree.chart.renderer.category.junit.GanttRendererTests:org.jfree.chart.renderer.category.junit.GradientBarPainterTests:org.jfree.chart.renderer.category.junit.GroupedStackedBarRendererTests:org.jfree.chart.renderer.category.junit.IntervalBarRendererTests:org.jfree.chart.renderer.category.junit.LayeredBarRendererTests:org.jfree.chart.renderer.category.junit.LevelRendererTests:org.jfree.chart.renderer.category.junit.LineAndShapeRendererTests:org.jfree.chart.renderer.category.junit.LineRenderer3DTests:org.jfree.chart.renderer.category.junit.MinMaxCategoryRendererTests:org.jfree.chart.renderer.category.junit.ScatterRendererTests:org.jfree.chart.renderer.category.junit.StackedAreaRendererTests:org.jfree.chart.renderer.category.junit.StackedBarRenderer3DTests:org.jfree.chart.renderer.category.junit.StackedBarRendererTests:org.jfree.chart.renderer.category.junit.StandardBarPainterTests:org.jfree.chart.renderer.category.junit.StatisticalBarRendererTests:org.jfree.chart.renderer.category.junit.StatisticalLineAndShapeRendererTests:org.jfree.chart.renderer.category.junit.WaterfallBarRendererTests:org.jfree.chart.renderer.junit.AbstractRendererTests:org.jfree.chart.renderer.xy.junit.StackedXYAreaRenderer2Tests:org.jfree.chart.renderer.xy.junit.StackedXYAreaRendererTests:org.jfree.chart.renderer.xy.junit.StackedXYBarRendererTests:org.jfree.chart.renderer.xy.junit.StandardXYItemRendererTests:org.jfree.chart.renderer.xy.junit.XYBarRendererTests:org.jfree.chart.renderer.xy.junit.XYLineAndShapeRendererTests \
    --iteration 100 \
    --complianceLevel 8 \
    --workingdirectory /tmp/Chart-npefix \
    --source "$src_dir:$test_dir" \
    --classpath "$classpath" \
    --repairStrategy "fr.inria.spirals.npefix.main.all.DefaultRepairStrategy"

The test parameter comes from separating each line of the output of the defects4j export -p tests.relevant with ":".

[monperrus]

Thanks a lot, super useful. I do get the same error for chart-993.

Now, it's not a failure to run, it's a failure to repair.

So two options:

• for your experiment you consider it as a NOPATCH
• we try to debug this case (best effort, time boxed) and improve NPEFix

WDYT?

[Peanu11]

Dear @monperrus ,
Thanks for your effort.
I've encountered similar errors on many examples of defects4j. I also can't successfully run npefix on your npe-dataset(https://github.com/Spirals-Team/npe-dataset). Could you please give me an example of what works successfully?

[monperrus]

Spirals-Team/npe-dataset

npe-dataset is dying from a dependency and version hell: many dependencies have disappeared from maven central, and most bugs in it cannot be compiled and run from modern Java incl. 11 and 17, which is the one Spoon/NpeFix requires.

I've encountered similar errors on many examples of defects4j.

Let's take it from a positive side: on which example of defects4j v3 (Java 11) does it run well?

[Peanu11]

Let's take it from a positive side: on which example of defects4j v3 (Java 11) does it run well?

Dear @monperrus ,

It's sad that I've tried a lot of examples these days and haven't found one that works successfully. Could you please give me an example of what works successfully? So that I can check if there is a problem with my local environment.

[monperrus]

@Peanu11

Just got an example working end to end.

1. compile npe-dataset with Java 8
2. compile latest commit of npefix with Java 17
3. adapt and run End2EndTest which fixes pdfbox_2965 from npe-dataset
4. you obtain patches.json like this one

[Peanu11]

Dear @monperrus ,

Thanks a lot. I successfully got the patches.json for pdfbox_2965.

However, NPEFIX gets stuck during multirun, and the program can not finish properly.

I've also tested on other examples, and it turns out that NPEFIX doesn't end properly either, and the generated patches.json doesn't contain patch information.

In the case of math-369, below is its patches.json file and execution information.

Here's my command for math-369.

java -jar "$NPEFIX_JAR" -s ./src/ -c ./target/classes/:./target/test-classes/:/root/.m2/repository/junit/junit/4.11/junit-4.11.jar:/root/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar -t org.apache.commons.math.analysis.solvers.BisectionSolverTest

[Peanu11]

After reviewing, we think the reason the program cannot complete in pdfbox_2965 is that it gets stuck while iterating the strategy in launchtest.java, whereas in math-369, it fails to iterate through the strategy.

[monperrus]

thanks for the additional info, would you be able to debug and fix it with a pull-request?