Proof of concept: Transpile in workers

[novemberborn]

This is a proof of concept for #1052.

The code is terrible! Let's not talk about that!

What it does:

• Transpiles tests in the workers
• Integrates babel-core/register-like behavior into AVA itself
• Generates a bundle which the workers use to gain quicker access to Babel

Caveats:

• There is no cache invalidation
• The babel-core/register-like behavior just assumes any non node_modules file needs transpilation, using non-test-related Babel setup
• Bundle generation is ridiculously slow (due to Don't use Babel for rewriting require statements novemberborn/cached-module-loader#10)

Here's some unscientific comparisons comparing x690@master and x690@test-against-src (which requires the babel-core/register-like behavior) with AVA master (6165a23). Does not include bundle generation time.

Command used is time ./node_modules/.bin/ava --verbose with Node.js v6.8.1. For the master no cache tests, BABEL_DISABLE_CACHE=true is set. master cache tests are run twice with Babel's cache enabled, capturing the second result.

	x690@master	x690@test-against-src
master no cache	5.11s	10.01s
master cache	2.58s	5.04s
poc no cache	9.91s	9.78s
poc cache	2.49s	2.55s

Observations:

• This poc is slower when sources do not need to be transpiled. Presumably that's due to the remaining overhead of loading Babel in each worker, and the cost of transpilation itself
• However there is no extra cost when transpiling sources, and it seems on par with if not faster than master (except that there is no cache validation in the poc…)
• Performance improves dramatically with the native source transpilation, presumably due to better caching
• When the concurrency option is not set, we only start the next worker when we've received tests from the previous worker! We should probably spawn more workers concurrently, except that if caches are missing each workers starts fighting for CPU time and the test run ends up being slower (in the x690@test-against-src case 18.94s versus 13.50s)

Takeaways:

• Integrating source transpilation seems promising. It's faster once cached and on par when there is no cache
• Uncached performance without source transpilation seems too slow, this needs more investigation. Perhaps we could stick to transpiling test files in the main thread, but still add source transpilation in the workers
• We may need to do this together with changes to the worker pool allocation to get the best performance balance

Thoughts?

[novemberborn]

This will be fleshed out further when implementing RFC001.