feat(incremental): add incremental mode

[nicojs]

Add incremental mode. When incremental mode is active, Stryker will use the results from the previous run to determine which mutants need to be retested, yielding much faster results.

You can enable incremental mode with --incremental (using "incremental": true in the config file will also work but is not recommended),

Results are stored in the reports/.mutation-incremental.json file. This file adheres to the JSON result schema, so they are interchangeable.

TODO:

• Experiment with this feature in the wild to garner feedback
• Add unit tests
  • project-reader.ts
  • mutation-test-report-helper.ts
  • mutant-test-planner.ts
  • strip-ansi-appender.ts
  • test-coverage.ts
  • incremental-differ
• Add an e2e test per test runner (don't forget the command test runner)
• Decide what exactly should invalidate the cache:
  • File where the mutant originated from changed
  • File where the tests covering the mutant
  • Test coverage changed
  • Smart file diff instead of invalidating all mutants when a file changed?
  • Invalidate when the mutant coverage of a test changed
  • Normalize line endings when validating if file contents changed.
  • ❓ When Stryker options changed out of scope
  • ❓ When the Stryker version updates out of scope
• Make --incremental compatible with --mutate
  • Don't reuse old test results when they are in scope
• Do more testing with command runner. Does it work as expected?
• Make --incremental compatible with --mutate and --force
• Log incremental statistics: feat(incremental): add incremental mode #3609 (comment)
• Make the progress reporter useful when running in --incremental mode
• Reassign mutant ids in the mutation test helper don't show mutant ids in reports

Limitations:

Running in incremental mode, Stryker will do its best to produce an accurate mutation testing report. However, there are some limitations here:

• Stryker will not detect any changes you've made in files other than mutated files and test files.
• Detecting changes in test files are only supported if the test runner plugin supports reporting of test files. At the moment of writing, only the jest-runner, cucumber-runner, and mocha-runner plugins support that.
• Any other changes to your environment will not be detected, such as updates to your dependencies, devDependencies, changes to your environment variables, changes to your .snap files, updates to readme files, etc

With these limitations in mind, it is still recommended to do a full run once in a while instead of relying on incremental for all your mutation testing needs.

Fixes #2753

[nicojs]

For anyone that wants to help test this feature: please install Stryker beta and report your findings here.

npm i -D --legacy-peer-deps @stryker-mutator/core@next 

[nicojs]

I just had a talk with @rouke-broersma . This feature has a lot of overlap with Baseline of Stryker.NET.

There are some differences:

• Stryker.NET supports different providers (Disk, Dashboard and AzureFileStorage).
• Stryker.NET support multiple versions of the baseline files inc fallback files when it couldn't be found

The --incremental feature described here is a lot simpler. Just 1 file and 1 provider. I do expect to support a --incrementalFile (or --incrementalLocation) in the future. That's why I'm thinking to keep the name --incremental.

[yesmar12]

I'm excited for this feature I think it will be very valuable!

I have tried to run this and it seems to be working for me. One thing of note

Ran 0.00 tests per mutant on average. This line while technically accurate may be misleading because technically there were tests ran on the mutants but they were just run a while ago. Maybe we can add more text here to be more explicit.

And having the info statements give some details would be awesome like number of files changes, new mutants added. Also having the report say this it is an incremental run would be cool too. Having the ability to just report the new mutants would be nice. I would want to add this to a merge request process and showing all of the details is much less important than just showing the diff.

Thank you for your effort on this.

[nicojs]

Thanks for the feedback @yesmar12

Ran 0.00 tests per mutant on average. This line while technically accurate may be misleading because technically there were tests ran on the mutants but they were just run a while ago. Maybe we can add more text here to be more explicit

Yeah, good point. Suggestions are welcome. We also have the data for the total amount of tests run, so we could provide both metrics in that scenario. I've added it to the TODO list.

And having the info statements give some details would be awesome like number of files changes, new mutants added

Good idea! We currently have this: "Incremental mode: 3390/3546 are reused.", but we can indeed do better! I've added it to the TODO list.

Having the ability to just report the new mutants would be nice

Hmm this would be some more work. You are probably referring to the HTML reporter. The HTML reporter is shared across all Stryker frameworks and even other mutation testing frameworks. Changes to that central core are expensive and need to be approved by all members.

Maybe we could think of a new html-diff reporter, that only reports the changes in an --incremental scenario. That way you can have both the full report and the diff report. I would prefer it to be a separate feature, incremental support shouldn't be blocked by this.

[rouke-broersma]

Having the ability to just report the new mutants would be nice

Hmm this would be some more work. You are probably referring to the HTML reporter. The HTML reporter is shared across all Stryker frameworks and even other mutation testing frameworks. Changes to that central core are expensive and need to be approved by all members.

Maybe we could think of a new html-diff reporter, that only reports the changes in an --incremental scenario. That way you can have both the full report and the diff report. I would prefer it to be a separate feature, incremental support shouldn't be blocked by this.

In stryker-net this would be called the since feature which is different from incremental (or what we call baseline).

The point of incremental is imo that you don't have to run all mutations but you do end up with a full report.

The since feature is then intended to test only those mutants changed or new since a previous state in our case currently implemented based on git information), you then also only see a report on those mutants. We currently mark all other mutants as ignored in the report which effectively removes them from the report. I don't personally see a reason to make the report fit this specific feature, we already have the ignored state.