Mutants SURVIVED Incorrectly in Singleton Pattern (Example Included)

[apoorva-sriv]

For the test here with PIT 1.6.6 ALL mutators, the mutants for line 4 survive incorrectly with the singleton pattern.

Singleton

---

Without singleton

[hcoles]

Static state such as singletons are known cause of issues for mutation system that use coverage data and do not launch a jvm per mutant.

My best guess for why you are seeing the above behaviour is that the calls to Example.to getInstance exercised by the tests are being made by other static initializer code. This is just a guess however, and something else might be going on.

If you can post a complete minimal executable project (code, tests + build file in a zip) that reproduces the problem I can take a look.

[apoorva-sriv]

@hcoles It's the same repo as in the link in the question: https://github.com/apoorva-sriv/pitest-example-2.

[hcoles]

Ok, looking at this, it is a classic static state issue.

When the class is a singleton, the value of num is set when the class is initialised. The mutation is not activated until after the initialisation of the static state, so effectively does not happen.

The only bulletproof way to avoid this issue is to launch a new jvm for each mutant. Unfortuantely this makes mutation testing impractically slow.

Pitest does support this however. If you set 'mutationUnitSize' to 1, then it will launch a jvm for each mutation.

Unfortuantely that is not the end to the issues with static state. It also affects which tests are run. Pitest targets tests based on coverage, unfortunately only the test that triggers the loading of a class will appear to exercise the static initializer code. Other tests will thereofre not be selected, even if pitest is run in one jvm per mutant mode.

Pitest did use to support some tricks to deal with this, but it greatly complicated the code, while being of no practical benefit to real teams due to the performance hit. It was therefore removed.

The strategy now is to try to avoid mutating static initialisation code, but the analysis is simplistic. It could conceivably be expanded to cover this case however - not mutating code in init when the constructor is private and called only from a static initialiser.

As an aside, you have 'ALL' mutators enabled. Unless you have a good reason for doing so this is strongly discouraged. Many of the mutators are not suitable for general use.