Ability to run test files in the same process

[sindresorhus]

Split from #1322.

… add an option to not spawn AVA test files as child-processes. Pretty much what ava/profile.js does now. I also need this for testing Electron apps, as their tester needs to inject some globals. Having such option would also fix our React/etc performance issues.

It could be a global option that when turned on would run the test files in the same process in a new script context using the vm module instead of a child process.

[jeffijoe]

This would still run each test in parallel, right?

[novemberborn]

This would still run each test in parallel, right?

Yes.

[ORESoftware]

@novemberborn yes parallel, but not in separate processes, because require('vm') doesn't spawn a separate process (...or does it?)

[sindresorhus]

because require('vm') doesn't spawn a separate process (...or does it?)

It does not. That's the whole point of this issue, to not spawn.

[tniessen]

This would still run each test in parallel, right?

Just FYI, the vm module has nothing to do with concurrency, and it is not comparable to executing tests in separate processes. There is neither strict isolation nor will any code be executed in parallel, apart from the usual event loop.

[novemberborn]

Prompted by #1631 it might be interesting to still start workers, but reuse them for different test files. We'd use vm for sandboxing. Would be good to check Jest's sandboxing approach as well.

[lo1tuma]

I’ve experimented a little bit with implementing sandboxing in ava via the vm module. This seems to be quite tricky because per default you don’t have a node-environment in the context of the vm, which means you don’t have e.g. require.
I also tried NodeVM from vm2 which unfortunately has a bug which leads to throwing
a TypeError for every assertion (e.g. t.is()). AFAIK vm2 injects all the commonjs/node globals and the function wrapper manually to the sandbox, so I would assume that it doesn’t work nicely with ES modules.

I’ve also struggled a little bit with the way how the runner instance is loaded in each test file (i.e. import test from 'ava'). The current approach only works because ava uses a separate process for every test file.

[link89]

@lo1tuma @sindresorhus Is it possible to pack all selected files into a single bundle (webpack maybe) and then execute that file?

[lo1tuma]

TBH I’m not even sure if it is worth to have the sandboxing feature at all. It is quite hard to implement and I assume that it would also have a negative impact regarding performance.
The main reason why I want to run my unit tests in a single process is actually that I think the performance would be much better.
Apart from that I also never had any issues with test runners that don’t offer sandboxing (e.g. mocha). I would even prefer the test runner to fail when my tests or doing weird things, e.g. like manipulating the same global state.

[link89]

In may case we are using ava for acceptance testing and this feature will make it easy for us to make full use of limited resources.

[novemberborn]

TBH I’m not even sure if it is worth to have the sandboxing feature at all. It is quite hard to implement and I assume that it would also have a negative impact regarding performance.

Yea we can punt on that for now.

[novemberborn]

Closing in favor of #1428.

[lo1tuma]

Am I right, or hasn’t this feature never be implemented although #1428 is marked as done? I would love to see an option to disable threadding and sandboxing completely as it slows down the test run significantly.

[novemberborn]

Proper isolation between test files is a core feature. Worker threads have sped this up greatly compared to child processes. We're not looking to implement this issue as originally proposed.