Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support --incremental mode. #2753

Closed
nicojs opened this issue Feb 19, 2021 · 3 comments · Fixed by #3609
Closed

Support --incremental mode. #2753

nicojs opened this issue Feb 19, 2021 · 3 comments · Fixed by #3609
Labels
🚀 Feature request New feature request

Comments

@nicojs
Copy link
Member

nicojs commented Feb 19, 2021

Is your feature request related to a problem? Please describe.
Mutation testing takes time. Right now, Stryker can only run mutation testing as a whole. This means waiting minutes or even hours between runs. Even if you just changed a small part of your code.

You can, of course, specify which files to mutate, but that only partly solves the problem, as you lose your previous report (it gets overwritten).

Describe the solution you'd like
I would like a feature where you can rerun parts of a mutation run. Stryker would then go and update the reports with this new data.

stryker run --incremental

The way this works is by reading the JSON report (emitted by the json reporter), figuring out what to mutate/run exactly, mutate and run that, and update the reports. It would use be able to compare input files to the files written in the report and do it's best to do as little work as possible to update the report. 'Incremental mode' would imply that the JSON reporter is active. If no JSON report is found, it can go and do a full run and write that to disk.

You can combine --incremental with other options to do more interesting things.

# only run foo.js and update the reports
stryker run --incremental --mutate foo.js

# only run bar.js lines 20 to 40 and update the reports
stryker run --incremental --mutate bar.js:20-40

# only run survived and no-coverage mutants again and update the reports
stryker run --incremental --only-undetected

# only run the specific mutants "1337" and "42" (identified by their id) and update the reports
stryker run --incremental --only-mutants 1337,42

# only run mutants that are changed (needs some more work)
stryker run --incremental

Describe alternatives you've considered

We've discussed this with @Garethp, @hugo-vrijswijk, @Mobrockers, @richardwerkman and @Stijn-Rutten

See #2751 for a longer discussion. We've also discussed this with @Garethp and @hugo-vrijswijk, @Mobrockers, @richardwerkman, and @Stijn-Rutten.

Additional context
The JSON report currently only has information about the "files under test" and not the test files themselves. I'm working on adding that info. It is a prerequisite for --incremental, otherwise Stryker cannot know which test files updated.

@nicojs nicojs added the 🚀 Feature request New feature request label Feb 19, 2021
@rouke-broersma
Copy link
Member

@nicojs We have decided to rename our 'incremental' to 'baseline' and split off the 'since' option from the baseline completely.

@nicojs
Copy link
Member Author

nicojs commented Jun 24, 2021

A thought: this --incremental support won't help with faster CI out of the box, without users downloading the previous incremental JSON file. I still think it's fine as a start, we can add other features later.

@nicojs
Copy link
Member Author

nicojs commented Jun 28, 2022

I've started with an implementation in #3609, but I want to get this right. That's why I've released a beta version.

You help me by installing the beta and testing it out. Report back any unexpected behavior. See the PR for details on how to install it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🚀 Feature request New feature request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants