Stryker with Mocha / Typescript results in different mutation score every run

[csimko]

Summary

When running stryker run with mocha and typescript it succeeds. However the issue is that on every subsequent run the score is different. No file manipulation is present, no changes to source are made during runs.

Reproduction

1. Please checkout the following branch:
   https://github.com/ChannelApe/channelape-typescript-web-service-sdk/tree/feature/channelape-actions-controller-and-sqs-functions

2. Run npm i

3. Run npm run mutate
   stryker_first_run.log
   stryker_second_run.log

4. You will note that stryker succeeds. Make note of the mutation score

5. Run npm run mutate again

6. Now you will see that the score has changed. You may repeat this as often as you like, the score is different each time.

Stryker config

module.exports = function(config) {
  config.set({
    files: [
      "test/**/*.ts",
      "src/**/*.ts"
    ],
    testRunner: "mocha",
    mochaOptions: {
      files: ["test/**/*.ts", "src/**/*.ts"],
      opts: "test/mocha.opts"
    },
    mutator: "typescript",
    reporters: ["clear-text", "progress", "html"],
    testFramework: "mocha",
    coverageAnalysis: "off",
    tsconfigFile: "tsconfig.json",
    thresholds: { high: 90, low: 70, break: 20 },
    mutate: [
      "src/**/*.ts"
    ]
  });
};

Stryker environment

+-- stryker@0.29.0
+-- stryker-api@0.21.0
+-- stryker-html-reporter@0.16.1
+-- stryker-mocha-framework@0.12.1
+-- stryker-mocha-runner@0.14.1
+-- stryker-typescript@0.13.1

+-- mocha@3.5.3
+-- mocha-typescript@1.1.17

Your Environment

software	version(s)
node	8.11.2
npm	6.4.0
Operating System	Windows

stryker_first_run.log
stryker_second_run.log

[bartekleon]

@csimko It can be caused because using windows or not enough space. Try to empty ram and CPU as much as you can, also check if there are not killed processes. Also try testing in headless browser if you are not doing it. A bit similar to #291

[simondel]

@csimko I can also reproduce this on my machine. The score seems to be more consistent if I don't test in parallel (maxConcurrentTestrunners: 1).

The transpiler errors seem to always be the same, so the issue really seems to be in your tests. What kind of application is it? What system resources (files outside of src and test, network ports on your system, etc) does it use?

[sanderkoenders]

I am seeing consistent runs on Linux with a score of 21,74%. Seems to be related to Windows indeed.

[simondel]

Thanks for testing it @archcry!

[bartekleon]

@Archcry, @simondel when I was making PR and testing on linux i also had inconsistent results (probably caused by memory limit) [travis tests it perfectly :/], but on windows it is not consistent at all. [Simondel PR is waiting for you :)]

[simondel]

I'm closing this issue for now. If you need any help from us, let us know!