Jest custom params

[FlorianBurgevin]

Hello,

I'm trying to test an create-react-app application with jest and stryker.

I have errors durings the initial tests of stryker because parameters added to my jest command line are not taken.

The test script in my package.json:
"test": "react-scripts test --env=jest-environment-jsdom-sixteen --verbose"

And the stryker script
"test:stryker": "stryker run"

The main problem is beacause my tests need jsdom-sixteen. Without I have the following error:

TypeError: MutationObserver is not a constructor

I give the param via command line because create-react-app doesn't accept overriding it directly.

Stryker config

module.exports = function (config) {
  config.set({
    mutator: 'javascript',
    mutate: ['src/**/*.js?(x)', '!src/**/*@(.test|.spec|Spec).js?(x)'],
    packageManager: 'yarn',
    reporters: [],
    testRunner: 'jest',

    coverageAnalysis: 'off',
    jest: {
      project: 'create-react-app',
    },
  });
};

Stryker environment

 "@stryker-mutator/core": "^3.1.0",
 "@stryker-mutator/javascript-mutator": "^3.1.0",
 "@stryker-mutator/jest-runner": "^3.1.0",
 "react-scripts": "3.4.1"

[simondel]

Thanks for opening this issue! The env maps to the Jest property of testEnvironment (found here) which can be set in your config.

Because you passed it to your regular test command, Stryker won't know about it. Unfortunately, even if you'd set in your config, Stryker will override it. This is quite silly because jsdom is the default env value and we explicitly set it to jsdom.

I believe that you should set your env in your Jest config because it seems like jest-environment-jsdom-sixteen is required in order to run your tests successfully so it should always be applied.

What do you think of this?

There are some things that I want to change:

1. Make sure stryker does not override your env setting
2. Make the user able to override their regular config with custom settings for Stryker (@nicojs what are your thoughts on this?)

As a temporary workaround while we fix this, you could use the command test runner by not setting the testRunner property in your stryker config. This will make Stryker execute npm test. It will be slower, but it should at least give you a working project.

note: your npm test command will have to exit after testing, just like in your CI/CD pipeline. Else Stryker will get stuck waiting for the tests to finish. You can set a custom command using commandRunner: { command: 'npm run mocha' } in your config

[FlorianBurgevin]

Hi, thanks for your answer !

If I understand, when I set testRunner to jest, my npm test command is not executed ?

jest-environment-jsdom-sixteen is not in my jest config (into package.json) because unfortunately create-react-app doesn't allow it. So I have to pass the param in the command line.

---

An other problem maybe linked with that is my env variables in .env file are not handled with jest testRunner.

I fixed it with the following command
"dotenv -e .env stryker run"

Maybe you can always use a custom command line (npm test) even with the jest testRunner ? It should fix both problems ?

---

I tried to remove jest testRunner from my config file by it's not working :(
It timeout after 5 min

10:43:37 (11851) DEBUG TimeoutDecorator Starting timeout timer (300000 ms) for a test run
10:48:37 (11851) DEBUG TimeoutDecorator Timeout expired, restarting the process and reporting timeout
10:48:37 (11851) ERROR InitialTestExecutor Initial test run timed out! Ran following tests before timeout:
10:48:37 (11851) ERROR StrykerCli an error occurred Error: Something went wrong in the initial test run
    at InitialTestExecutor.validateResult (XXX/node_modules/@stryker-mutator/core/src/process/InitialTestExecutor.js:86:15)
    at InitialTestExecutor.run (XXX/node_modules/@stryker-mutator/core/src/process/InitialTestExecutor.js:41:14)
    at process._tickCallback (internal/process/next_tick.js:68:7)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[simondel]

Indeed, when you specify a testRunner, Stryker will take control and start to interact directly with Jest. This allows us to tweak Jest in a way to better perform.

I just checked the react-test docs and they automatically try to figure out if you're running on your local machine or on a CI server. You can manually set this by setting the env variable CI=true.

Could you check what happens when you se the CI env variable for react-test?

[simondel]

You could also pass --watchAll=false to disable the watch mode

[FlorianBurgevin]

@simondel Ok it's that !

I'll launch it in my CI to test

[FlorianBurgevin]

 11:29:15 (36) INFO InputFileResolver Found 28 of 86 file(s) to be mutated.
 11:29:15 (36) INFO InitialTestExecutor Starting initial test run. This may take a while.
 11:29:24 (36) INFO InitialTestExecutor Initial test run succeeded. Ran 1 tests in 8 seconds (net 8300 ms, overhead 2 ms).

I don't know if it's OK or not but it ran only 1 test.
It should find more than 50.

And all tests timeout

11:29:24 (36) INFO MutatorFacade 784 Mutant(s) generated
 11:29:24 (36) INFO SandboxPool Creating 8 test runners (based on CPU count)
 Mutation testing 0% (elapsed: <1m, remaining: n/a) 0/784 tested (0 survived, 0 timed out)
 Mutation testing 0% (elapsed: <1m, remaining: n/a) 0/784 tested (0 survived, 0 timed out)
 Mutation testing 1% (elapsed: <1m, remaining: ~48m) 8/784 tested (0 survived, 8 timed out)
 Mutation testing 1% (elapsed: <1m, remaining: ~1h 4m) 8/784 tested (0 survived, 8 timed out)
 Mutation testing 2% (elapsed: <1m, remaining: ~40m) 16/784 tested (0 survived, 16 timed out)

[simondel]

I don't know if it's OK or not but it ran only 1 test.
It should find more than 50.

That's ok. Since the command test runner has no idea what command you're running it just reports one test. We may have to change that so it just says Ran tests in 8 seconds when using the command test runner.

Could you try to increase the timeout using the stryker config:

timeoutFactor: 5

Jest can take a while to get started on some applications when running in parallel (more info on timeouts: https://www.npmjs.com/package/@stryker-mutator/core#timeoutfactor--number-)

[FlorianBurgevin]

@simondel not better :/