Regression with Pitest 1.11.6

[davidburstrom]

One of my tests (which I unfortunately cannot disclose started failing when I bumped Pitest from 1.11.5 to 1.11.6. The project is using com.groupcdg.pitest:extended-mutators:0.1.5 as well:

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index 2 out of bounds for length 2
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
        at java.base/java.util.Objects.checkIndex(Objects.java:359)
        at java.base/java.util.ArrayList.get(ArrayList.java:427)
        at com.groupcdg.pitest.mutators.swap.parameters.JunkParamSwapInterceptor.paramsHaveMismatchedGenerics(JunkParamSwapInterceptor.java:116)
        at com.groupcdg.pitest.mutators.swap.parameters.JunkParamSwapInterceptor.isJunk(JunkParamSwapInterceptor.java:58)
        at com.groupcdg.pitest.mutators.swap.parameters.JunkParamSwapInterceptor.lambda$isJunkParamSwapMutant$0(JunkParamSwapInterceptor.java:50)
        at java.base/java.util.function.Predicate.lambda$negate$1(Predicate.java:80)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at com.groupcdg.pitest.AnalysingInterceptor.intercept(AnalysingInterceptor.java:33)
        at org.pitest.mutationtest.build.CompoundMutationInterceptor.intercept(CompoundMutationInterceptor.java:51)
        at org.pitest.mutationtest.build.MutationSource.createMutations(MutationSource.java:60)
        at org.pitest.functional.FCollection.flatMapTo(FCollection.java:51)
        at org.pitest.functional.FCollection.flatMap(FCollection.java:61)
        at org.pitest.mutationtest.build.MutationTestBuilder.createMutationTestUnits(MutationTestBuilder.java:59)
        at org.pitest.mutationtest.tooling.MutationCoverage.buildMutationTests(MutationCoverage.java:387)
        at org.pitest.mutationtest.tooling.MutationCoverage.runAnalysis(MutationCoverage.java:166)
        at org.pitest.mutationtest.tooling.MutationCoverage.runReport(MutationCoverage.java:143)
        at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:129)
        at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:57)
        at org.pitest.mutationtest.commandline.MutationCoverageReport.runReport(MutationCoverageReport.java:98)
        at org.pitest.mutationtest.commandline.MutationCoverageReport.main(MutationCoverageReport.java:45)

[hcoles]

Hi David,

There error is being thrown by the arcmutate base plugin rather than pitest. We've seen this error after switching branches without cleaning, so stale bytecode is present which invalidates assumptions. We'll put a fix out so this doesn't create an error, however you may encounter further issues if stale bytecode is present.

[davidburstrom]

Hi Henry!

Where is said bytecode cached? I'm running the build through the Gradle plugin, and cleaning the project plus disabling task caches does nothing to get around the problem. The verbose logging doesn't hint anything about locations for bytecode.

[davidburstrom]

Or maybe I'm missing something about the perspective you're looking at the problem from. In my case, the build works with Pitest 1.11.5, and if the only thing I do is to change the version to 1.11.6, the mutation test crashes.

[hcoles]

By "cached" I meant still present in the build dir (pitest doesn't store any bytecode). We've seen this sort of issue with maven after branch switches, and a clean has always fixed it. Gradle's caching complicated things, but it sounds like something similar might be happening.

The only changes between pitest 1.11.5 and 1.11.6 are some update of links from http to https, and a bug fix that prevents filters being applied twice if you add some configuration to a filter that is active by default. It's hard to see how either change could result in this failure.

I think you may be using an old version of the base/extended mutators plugin. The latest version is

https://search.maven.org/artifact/com.groupcdg.arcmutate/base/1.0.2/jar

Its maven coordinates changed, so you may have missed the update. This particular error should not occur with the latest version, although you may encounter a similar issue if you have bytecode where the number of parameters in a method have changed since client code was compiled against it.

[davidburstrom]

Aha, I know for sure that there's no stale bytecode involved in the build, through various means (of which one is a clean cache-less rebuild). Between the passing and failing build, the Gradle Build Scan comparison indicates that the only task input changes are quite literally the launchClasspath:

What's worse though, is that if I run Pitest with JDK 11 both versions pass, but the later fails with JDK 17. The plot just thickened. Do you do something custom that could somehow depend on zip alignment? It's just a hunch, based on the fact that the latest Pitest version didn't have any functional changes like that.

Thanks for the pointer to the new coordinates, I've definitely missed that update. I'll see if I notice this issue with those. Just a thought, it'd be possible to release a new version of com.groupcdg.pitest:extended-mutators with a stub that directs the consumer to the new coordinates.

[davidburstrom]

With the new coordinates, the issue no longer persists, so that's good. But the issue as a whole makes me worried.

[davidburstrom]

After some arduous debugging I suspect that the issue is caused by unstable hashes which is why there is a difference between JDK 11 and JDK 17, and I believe the culprit is

pitest/pitest/src/main/java/org/pitest/plugin/FeatureSelector.java

Line 41 in 42fb6d2

Set<T> active = factories.stream()

because it produces an unordered set of interceptors.

For a failing run, the JunkParamSwapInterceptor is the first to run as part of the CompoundMutationInterceptor:

For a passing run, it's run after some other interceptors:

When some of the other interceptors have been run, the modified Collection<MutationDetails> no longer contains the problematic mutation, and thus JunkParamSwapInterceptor never sees it and doesn't crash.

The reason why the mutations are removed is because of ExtendedMutatorInterceptor, because that particular module doesn't have the correct package according to the licence.

[davidburstrom]

I can confirm that when I change the package of the problematic module, I can reproduce the issue consistently on both JDK 11 and JDK 17, which means there was a bug in com.groupcdg.pitest:extended-mutators:0.1.5. And com.groupcdg.arcmutate:base:1.0.2 works as intended.

In conclusion, if the set operation is changed to produce an ordered set, the FeatureSelector will produce more predictable results.

[hcoles]

Thanks for digging into this. I'll put a change in so features are applied in a consistent and predicatable order.