[rfc draft wip...] start mutation testing using Stryker [...] #6681
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Using Closing for now, will open a fresh new PR once I get a better handle on this. |
brodycj
changed the title
brodycj
force-pushed
the
start-mutation-testing
branch
from
ff97a3b to
966b482
Compare
brodycj
changed the title
added 8 commits
November 13, 2019 00:55
yarn add --dev @stryker-mutator/core ref: https://stryker-mutator.io/stryker/quickstart
as described in: https://stryker-mutator.io/stryker/quickstart $ npx stryker init ? Are you using one of these frameworks? Then select a preset configuration. None/other ? Which test runner do you want to use? If your test runner isn't listed here, you can choose "command" (it uses your `npm test` command, but will come with a big performance penalty) command ? What kind of code do you want to mutate? javascript ? [optional] What kind transformations should be applied to your code? ? Which reporter(s) do you want to use? html, clear-text, progress ? Which package manager do you want to use? yarn Writing & formatting stryker.conf.js... Installing NPM dependencies... and add the following line to the beginning of stryker.conf.js (with another blank line) to make lint (eslint) happy: "use strict";
with --no-cache in order to avoid any potential surprises from Jest caching
brodycj
force-pushed
the
start-mutation-testing
branch
from
b008c3b to
a2b13f0
Compare
brodycj
changed the title
|
I tried starting Stryker mutation testing on the following sets of source files on a virtual server with 40 CPUs:
with the following command: nohup yarn stryker | tee stryker-language-js-processing-output.txt &After about 14.5 hours I found the following information at the end of Mutation testing 20% (ETC 54h, 17m, 0s) 3219/15438 tested (660 survived, 302 timed out)
Mutation testing 20% (ETC 54h, 17m, 38s) 3219/15438 tested (660 survived, 302 timed out)
Mutation testing 20% (ETC 54h, 18m, 16s) 3219/15438 tested (660 survived, 302 timed out)
Mutation testing 20% (ETC 54h, 18m, 54s) 3219/15438 tested (660 survived, 302 timed out)
Mutation testing 20% (ETC 54h, 19m, 32s) 3219/15438 tested (660 survived, 302 timed out)
Killed
error Command failed with exit code 137.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.At this point I will probably have to run Stryker on one My apologies to the community for all of the noise here. I will likely continue working in a PR on my own fork for now. |
lock
bot
added
the
locked-due-to-inactivity
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
as documented in: https://stryker-mutator.io/stryker/quickstart
with
mutateentry added tostryker.conf.jsto check JavaScript insrc/cliwith a quick report here now outdated: https://chrisbrody.com/prettier-cli-mutation-report/index.html
report generated by the following command:
npx stryker runA TODO item is to add
stryker runtoscriptsinpackage.json.I am trying to use Stryker as much as I can in the various projects that I maintain and contribute to.
I had to use a Vultr server with 16 virtual CPUs (64 GB RAM) to get reasonable performance. Others may have better luck with more powerful workstations.
If I would try Stryker with all JavaScript files in
src, or even all JavaScript insrc/language-js, it would crash on me. This seems to be discussed in stryker-mutator/stryker-js#1230 & stryker-mutator/stryker-js#1284, other results in https://www.google.com/search?q=stryker+out+of+memory.I hope this is helpful, will understand if not.
I’ve added tests to confirm my change works.(If changing the API or CLI) I’ve documented the changes I’ve made (in thedocs/directory)(If the change is user-facing) I’ve added my changes to theCHANGELOG.unreleased.mdfile following the template.✨Try the playground for this PR✨