update progress reporters to show timed out test count

[brodycj]

It can be extremely useful to see if there are any test timeouts going on when starting a long-running Stryker test run.

I have been working on using Stryker on Prettier (see prettier/prettier#6681) where test timeouts seem to be more likely if I do not adjust the timeoutMS setting (in stryker.conf.js).

The source code changes seemed to be quite straightforward; a slightly more challenging part was to update the existing tests scripts to pass with these changes.

I did try these changes in my workarea and it seems to work with both default console "progress" reporter and "progress-append-only" reporter. (It seems to downgrade if I would pipe stryker run through the tee utility.)

Here is some initial output with the console "progress" reporter (I forget if I used an adjusted timeoutMS setting or not):

$ yarn stryker
yarn run v1.19.1
$ stryker run
18:10:04 (10047) INFO ConfigReader Using stryker.conf.js in the current working directory.
18:10:05 (10047) INFO InputFileResolver Found 4 of 5884 file(s) to be mutated.
18:10:05 (10047) INFO InitialTestExecutor Starting initial test run. This may take a while.
18:10:25 (10047) INFO InitialTestExecutor Initial test run succeeded. Ran 1 tests in 21 seconds (net 20065 ms, overhead 0 ms).
18:10:26 (10047) INFO MutatorFacade 1021 Mutant(s) generated
18:10:26 (10047) INFO SandboxPool Creating 16 test runners (based on CPU count)
Mutation testing  [===                                               ] 6% (ETC 5h, 29m, 57s) 64/1021 tested (3 survived, 9 timed out)

and with the "progress-append-only" reporter (I forget if I used an adjusted timeoutMS setting or not):

$ yarn stryker | tee progress-append-only-timed-out-save1.txt
yarn run v1.19.1
$ stryker run
18:34:40 (30236) INFO ConfigReader Using stryker.conf.js in the current working directory.
18:34:41 (30236) INFO BroadcastReporter Detected that current console does not support the "progress" reporter, downgrading to "progress-append-only" reporter
18:34:41 (30236) INFO InputFileResolver Found 4 of 5886 file(s) to be mutated.
18:34:41 (30236) INFO InitialTestExecutor Starting initial test run. This may take a while.
18:35:02 (30236) INFO InitialTestExecutor Initial test run succeeded. Ran 1 tests in 21 seconds (net 20623 ms, overhead 2 ms).
18:35:02 (30236) INFO MutatorFacade 1021 Mutant(s) generated
18:35:02 (30236) INFO SandboxPool Creating 16 test runners (based on CPU count)
Mutation testing 0% (ETC n/a) 0/1021 tested (0 survived, 0 timed out)
[...]
Mutation testing 0% (ETC 70h, 50m, 0s) 1/1021 tested (0 survived, 0 timed out)
Mutation testing 0% (ETC 36h, 47m, 50s) 2/1021 tested (0 survived, 0 timed out)
[...]
Mutation testing 2% (ETC 8h, 4m, 21s) 22/1021 tested (1 survived, 0 timed out)
Mutation testing 2% (ETC 7h, 51m, 31s) 23/1021 tested (1 survived, 1 timed out)
Mutation testing 2% (ETC 7h, 38m, 20s) 24/1021 tested (1 survived, 2 timed out)
Mutation testing 2% (ETC 6h, 52m, 19s) 27/1021 tested (1 survived, 4 timed out)
Mutation testing 2% (ETC 6h, 15m, 28s) 30/1021 tested (1 survived, 6 timed out)
Mutation testing 3% (ETC 5h, 56m, 27s) 32/1021 tested (1 survived, 6 timed out)
[...]
Mutation testing 3% (ETC 6h, 54m, 45s) 38/1021 tested (2 survived, 7 timed out)
Mutation testing 3% (ETC 6h, 47m, 54s) 39/1021 tested (2 survived, 7 timed out)
Mutation testing 4% (ETC 6h, 21m, 53s) 42/1021 tested (3 survived, 8 timed out)
Mutation testing 4% (ETC 6h, 25m, 46s) 42/1021 tested (3 survived, 8 timed out)
Mutation testing 4% (ETC 5h, 54m, 19s) 46/1021 tested (3 survived, 9 timed out)
[...]

I would be extremely grateful if the owners & maintainers would consider getting these changes reviewed, merged, and released in the near future. Installation from npm would be so much easier than GitHub or local workarea, especially when integrating with a larger project such as Prettier. I wonder if would make sense to publish with special npm tag such as "rc"?

I do think it would be nice if we could unit test these updates somehow. The existing test suite does not seem to test these progress reporters so extensively at this point. Unfortunately I also cannot promise when I can work on these due to some other priorities at this time.

[simondel]

Thanks for adding this! Could you also add a test to prove that the counting work? (That way we know it will continue to work :) )

[brodycj]

Could you also add a test to prove that the counting work?

Coming up, gonna merge from master first.

(That way we know it will continue to work :) )

+1 (+100)

[brodycj]

The CI / e2e build seemed to go red after merging from master. The same thing happened to me yesterday, and it went away when I did a rebase.

I just pushed a test update, will do another rebase if the build is still red.

[brodycj]

Thanks for the quick review and merge. Any idea when we can get this published?

It would really help us (me) with mutation testing on Prettier.

[simondel]

I'm hoping to release it this evening, but I'm having some issue with the release pipeline :(

[brodycj]

That's really fast, thanks:)

[simondel]

Deployed as 2.2.1: https://travis-ci.org/stryker-mutator/stryker/jobs/608425051?utm_medium=notification&utm_source=github_status

npmjs.org may take a while to index the new version. Manually updating should already work.

The release of 2.2.0 to npm failed. It was only deployed to GitHub due to a poor solution I picked in an attempt to fix the release pipeline.

[brodycj]

I saw 2.2.1 on npmjs.com pretty quickly. I updated my branch in prettier/prettier#6681, tried it with and without "append-only" reporter, seems to work perfectly now. Thanks again @simondel!