Stryker runs are not consistent

[jeznag]

I tried replicating stryker-mutator/stryker-tape-runner#3 with Mocha and found that I had the same issues with the mocha runner. I get different mutant kill scores each time:

Steps to replicate

1. Clone this repo: https://github.com/jeznag/empath-sentiment-analysis/tree/tests-with-mocha
2. Check out tests-with-mocha branch
3. npm install
4. Modify node_modules/.bin/stryker to have babel-node instead of node
5. Modify node_modules/stryker/src/utils/parserUtils.js to have sourceType: 'module'
6. Run stryker: ./node_modules/.bin/babel-node ./node_modules/.bin/stryker run

[nicojs]

@jeznag thanks for reporting this issue.

Unfortunately i have issues reproducing this on my windows machine. Could you explain to me what it is exactly that you modify in node_modules/.bin/stryker? I'm getting this error:

C:\z\github\jeznag\empath-sentiment-analysis>node_modules\.bin\babel-node ./node_modules/.bin/stryker run
C:\z\github\jeznag\empath-sentiment-analysis\node_modules\.bin\stryker:2
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")
          ^^^^^^^
SyntaxError: missing ) after argument list
    at Object.exports.runInThisContext (vm.js:73:16)
    at Module._compile (module.js:543:28)

I will also try it on a linux environment later today or tomorrow.

[jeznag]

Ah Windows probably doesn't pay attention to the shebang. You can do:

./node_modules/.bin/babel-node ./node_modules/.bin/stryker run

[nicojs]

I can now run it on my windows machine. The command i use is:

./node_modules/.bin/babel-node ./node_modules/stryker/bin/stryker run

However, i always get exactly 60 errors. A ran it about 10 times or so. I've printed the errors to console using debug logging and the clear text reporter and it seems that they all have to do with stryker mutating a regex string. All is working as expected as far as i can see...

Mutation testing  [] 100% (ETC 0.0s)[129 killed] [214 survived] [0 no coverage] [0 timeout] [60 error]
[2017-04-27 13:55:40.880] [INFO] Stryker - Done in 13 seconds.

An example of an errored mutant:

Mutant caused an error!
C:\z\github\jeznag\empath-sentiment-analysis\src\utils\emailSignatureParser.js: line 121:2
Mutator: BinaryOperator
-     "(" +
-       "(?:" +
+     '(' - '(?:' +

Indeed, if you subtract this string, i would expect there to be an error.

[jeznag]

@nicojs I tried it on my other macbook and I still get inconsistent results. Could this relate to an issue with the operating system?

[nicojs]

I feel like it must have something to do with that, maybe in combination with your project. I can't debug this further without repoducing it.

[riezebosch]

I'm facing the same issue (on the same OS). It's not that extreme as with @jeznag and it took me a dozen of test runs to reproduce this:

I expect 10 mutants to survive. They are all the same on typescript-generated code:

Mutator: BooleanSubstitution
-   Object.defineProperty(exports, "__esModule", { value: true });
+   Object.defineProperty(exports, "__esModule", { value: false });

----------------|---------|----------|-----------|------------|----------|---------|
File            | % score | # killed | # timeout | # survived | # no cov | # error |
----------------|---------|----------|-----------|------------|----------|---------|
All files       |   92.06 |      116 |         0 |         10 |        0 |       0 |
 plugins        |   84.00 |       42 |         0 |          8 |        0 |       0 |
  CustomCss.js  |   83.33 |        5 |         0 |          1 |        0 |       0 |
  Highlight.js  |   91.67 |       11 |         0 |          1 |        0 |       0 |
  Img.js        |   50.00 |        1 |         0 |          1 |        0 |       0 |
  Reveal.js     |   50.00 |        1 |         0 |          1 |        0 |       0 |
  Slides.js     |   90.91 |       10 |         0 |          1 |        0 |       0 |
  StaticBase.js |   66.67 |        2 |         0 |          1 |        0 |       0 |
  Template.js   |   75.00 |        3 |         0 |          1 |        0 |       0 |
  Theme.js      |   90.00 |        9 |         0 |          1 |        0 |       0 |
 template       |  100.00 |       48 |         0 |          0 |        0 |       0 |
  CustomCss.js  |  100.00 |        4 |         0 |          0 |        0 |       0 |
  Highlight.js  |  100.00 |        3 |         0 |          0 |        0 |       0 |
  Index.js      |  100.00 |        8 |         0 |          0 |        0 |       0 |
  Reveal.js     |  100.00 |        7 |         0 |          0 |        0 |       0 |
  Slides.js     |  100.00 |       23 |         0 |          0 |        0 |       0 |
  Theme.js      |  100.00 |        3 |         0 |          0 |        0 |       0 |
 Index.js       |   75.00 |        3 |         0 |          1 |        0 |       0 |
 Server.js      |   90.91 |       10 |         0 |          1 |        0 |       0 |
 SlideObject.js |  100.00 |       13 |         0 |          0 |        0 |       0 |
----------------|---------|----------|-----------|------------|----------|---------|

But sometimes that mutant get killed spontaneously in CustomCss.js:

----------------|---------|----------|-----------|------------|----------|---------|
File            | % score | # killed | # timeout | # survived | # no cov | # error |
----------------|---------|----------|-----------|------------|----------|---------|
All files       |   92.86 |      117 |         0 |          9 |        0 |       0 |
 plugins        |   86.00 |       43 |         0 |          7 |        0 |       0 |
  CustomCss.js  |  100.00 |        6 |         0 |          0 |        0 |       0 |
  Highlight.js  |   91.67 |       11 |         0 |          1 |        0 |       0 |
  Img.js        |   50.00 |        1 |         0 |          1 |        0 |       0 |
  Reveal.js     |   50.00 |        1 |         0 |          1 |        0 |       0 |
  Slides.js     |   90.91 |       10 |         0 |          1 |        0 |       0 |
  StaticBase.js |   66.67 |        2 |         0 |          1 |        0 |       0 |
  Template.js   |   75.00 |        3 |         0 |          1 |        0 |       0 |
  Theme.js      |   90.00 |        9 |         0 |          1 |        0 |       0 |
 template       |  100.00 |       48 |         0 |          0 |        0 |       0 |
  CustomCss.js  |  100.00 |        4 |         0 |          0 |        0 |       0 |
  Highlight.js  |  100.00 |        3 |         0 |          0 |        0 |       0 |
  Index.js      |  100.00 |        8 |         0 |          0 |        0 |       0 |
  Reveal.js     |  100.00 |        7 |         0 |          0 |        0 |       0 |
  Slides.js     |  100.00 |       23 |         0 |          0 |        0 |       0 |
  Theme.js      |  100.00 |        3 |         0 |          0 |        0 |       0 |
 Index.js       |   75.00 |        3 |         0 |          1 |        0 |       0 |
 Server.js      |   90.91 |       10 |         0 |          1 |        0 |       0 |
 SlideObject.js |  100.00 |       13 |         0 |          0 |        0 |       0 |
----------------|---------|----------|-----------|------------|----------|---------|

Or all of a sudden in Highlight.js

----------------|---------|----------|-----------|------------|----------|---------|
File            | % score | # killed | # timeout | # survived | # no cov | # error |
----------------|---------|----------|-----------|------------|----------|---------|
All files       |   92.86 |      117 |         0 |          9 |        0 |       0 |
 plugins        |   86.00 |       43 |         0 |          7 |        0 |       0 |
  CustomCss.js  |   83.33 |        5 |         0 |          1 |        0 |       0 |
  Highlight.js  |  100.00 |       12 |         0 |          0 |        0 |       0 |
  Img.js        |   50.00 |        1 |         0 |          1 |        0 |       0 |
  Reveal.js     |   50.00 |        1 |         0 |          1 |        0 |       0 |
  Slides.js     |   90.91 |       10 |         0 |          1 |        0 |       0 |
  StaticBase.js |   66.67 |        2 |         0 |          1 |        0 |       0 |
  Template.js   |   75.00 |        3 |         0 |          1 |        0 |       0 |
  Theme.js      |   90.00 |        9 |         0 |          1 |        0 |       0 |
 template       |  100.00 |       48 |         0 |          0 |        0 |       0 |
  CustomCss.js  |  100.00 |        4 |         0 |          0 |        0 |       0 |
  Highlight.js  |  100.00 |        3 |         0 |          0 |        0 |       0 |
  Index.js      |  100.00 |        8 |         0 |          0 |        0 |       0 |
  Reveal.js     |  100.00 |        7 |         0 |          0 |        0 |       0 |
  Slides.js     |  100.00 |       23 |         0 |          0 |        0 |       0 |
  Theme.js      |  100.00 |        3 |         0 |          0 |        0 |       0 |
 Index.js       |   75.00 |        3 |         0 |          1 |        0 |       0 |
 Server.js      |   90.91 |       10 |         0 |          1 |        0 |       0 |
 SlideObject.js |  100.00 |       13 |         0 |          0 |        0 |       0 |
----------------|---------|----------|-----------|------------|----------|---------|

Sometimes mutants are not generated. Looks like concurrency related? I had to do multiple test runs to get these results.

The issue becomes prevalent when using my stryker-baseline-reporter because all of a sudden you get different baseline outputs on subsequent runs.

[khujo]

I have the same issue running Stryker on MacOS. The mutation score is different with every run of Stryker. The results are consistent when running Stryker on Windows.

[simondel]

@khujo Thanks for telling us! Would you be able to create a repository to reproduce this issue? Then we can find some MacBooks to test this on :)

[khujo]

@simondel Sure, here you go: https://github.com/khujo/stryker-on-mac

Running Stryker in this project should produce 5 surviving mutants. Running this on my Mac always produces more mutants (6-8 most of the time).

[nicojs]

@simondel did you ever find the guy with the MacBook?

[simondel]

@nicojs unfortunately not :(

I'm closing this issue for now. We've recently released Stryker v1 and made a lot of changes. If we see this issue with v1 as well. We'll look at it.

[hazems]

I see this issue on my Mac book as well.

[cwouter]

I've got the same problem on my mac. Has someone tried to run it inside a linux docker container? The results look pretty stable on my Travis CI page: https://travis-ci.org/SockTrader/SockTrader.

Edit: SockTrader github

[nicojs]

Thanks for the reactions and the demo repo.

@simondel do you think you could test it out on mac hardware using SockTrader?

[cwouter]

@nicojs , can we reopen this issue or create a new one?

[riezebosch]

Protip! With Azure pipelines you get hosted macOs agents for free and when creating an open source project you can even link to your github repo.

[cwouter]

Thanks for the protip. I tried to apply it to SockTrader, but unfortunately build fails immediately. Seems to be Jest related instead of Stryker. See Azure pipeline build. (<= open the "npm install and test" task on Azure to see the details)