Write high performance webpack loader worker

[Timer]

x-ref: #5170

It'd be nice to write a high performance webpack worker that uses shared memory instead of IPC, automatic caching with customizable cache identifier, and anything else to make it really robust.

[gaearon]

https://github.com/facebook/jest/tree/master/packages/jest-worker#bound-worker-usage

[iansu]

That's a great example. So we want to create a webpack loader, like thread-loader, using jest-worker? Is the idea to make it as flexible as thread-loader where you can pass it any loader(s) you want or would it just be a wrapper around babel-loader? I guess we'd probably want to parallelize everything, css transforms, svg, etc.

This sounds like a fun challenge. I wouldn't mind giving this a try.

[gaearon]

I'm not actually sure if it needs to be so flexible. Maybe it could just be a layer on top of Babel loader, e.g. babel-worker-loader, but with its own cache and workers?

I would try to reuse as much has we can from:

• https://github.com/facebook/metro/tree/master/packages
• https://github.com/facebook/jest/tree/master/packages
• https://github.com/facebook/metro/blob/b0572abc71a29f265e0994d7af25da4a06f81dd2/packages/metro/src/defaultTransformer.js

There's been so much work on speeding them up and any regressions are carefully tracked.

[iansu]

I like the idea of keeping it focused on babel-loader. I'll take a look at those other examples.

[Timer]

I'd like to see ESLint cached too (because that incurs a Babel parse). If we could somehow share AST between our ESLint and Babel pipeline so we're not double parsing the file, that'd be 💯.