Investigate mutation testing #7607
Comments
|
Results for url-shim: https://extra-small-cactus.surge.sh/ |
|
Results for all audits: https://irate-boy.surge.sh/ error rates are somewhat inflated because we typically don't unit test individual UI strings in each audit, we test if UI strings that are in audits make it to the UI |
|
results for Good numbers overall, and not really a surprise that files like A ton of timeouts in this run, which stryker counts as a good thing (tests failed to complete -> mutation didn't get around our tests). I set the timeout to 80s, though, so it really is likely that most of these timeouts were caught by the run never completing (e.g. switching loop conditionals in the dependency graph is a good way to end up looping forever). |
|
results for Not many timeouts this time around. The few that happened were mostly where'd you'd expect mutations to cause infinite loops. did not check against #7513, which I think should help lantern fcp and lantern fmp coverage. |
Thanks for the compliment
This is indeed painful. We're working to improve it behind the scenes (this could be done using mutation switching). However, that will take some time. Keep spreading the love ❤️ |
Once youve upgraded to Stryker v1.2 it's using a single page application:
|
investigated, it's pretty sweet. I'll keep my branch open and we can revisit in the future. |
This is an exploration bug, not a specific proposal yet. Right now the compute time is way too high to run even in a daily cron job.
Mutation testing is another aspect of code coverage that tries to evaluate the quality of code coverage by changing parts of a code base and then checking if the test suite fails in response to that change. The idea is that just because a line of code is shown as covered by a test doesn't mean that test is really exercising the line of code.
e.g. if
if (val < 10) { /* whatever*/}is tested with the number 5, both the conditional and the following block are considered covered, but the case whenvalis greater than 10 is never exercised. A mutation testing library will try a few mutations, including flipping the conditional toval > 10and replacing it with atrueconstant value, both of which would reveal the implicit path that wasn't being tested.This (along with fuzzing!) could prove invaluable as we start opening up more parts of the codebase to outside configuration (plugins, budgets, stack packs, etc).
I used https://stryker-mutator.io/ for this. It was easy to get going, though there was some tweaking necessary to get results (it spawns multiple workers as does jest, which ends up swamping the disk and CPU, and the default test timeout per mutation is too short for us). The test branch is here: master...stryker (just
yarnandyarn stryker run)The only issues are that our tests are relatively slow (just
yarn unit-corecan take 30 seconds) and there are lots of possible mutations. Runningconfig.js,gather-runner.js, andrunner.jstook about 5 hours on an 8 core machine.Type checking before running unit tests can eliminate a whole bunch of the mutants, but is an extra 7-10 seconds on every run if the types are ok.
jestcan also hinder here, as its test ordering puts some of our heaviest jsdom-based tests first when they're the least likely to fail for anything but mutating report code.Results for
config.js,gather-runner.js, andrunner.jshtml: https://quilted-jewel.glitch.me/
(their html report is pretty nice)
plain text
A good number of these are ignorable (a lot of mutating log statements that we don't (and usually don't want to) test), but there are definitely some things it caught that are useful.
I'm currently about halfway through running mutations of all the audits, which is projected to take about 24 hours in total :)
Some other background:
The text was updated successfully, but these errors were encountered: