Spawn worker from blob / inlined bundle

[pkerpedjiev]

Apologies if this is a silly question but is it possible to use threads.js with a BLOB inline worker (like this https://stackoverflow.com/questions/5408406/web-workers-without-a-separate-javascript-file) so that it can be included in the main bundle?

Having a second worker bundle in addition to the main application code can really complicate things with cross-origin requests in a browser.

[andywer]

Hi! Right now this is not possible.

It used to be possible in the old 0.x versions, but it was hardly useful in real-life use cases. The problem is that you wouldn’t be able to import / require() any other modules in your worker and neither could you reference any variables from outside the worker function.

This is because the worker function would need to be serialized, sent to the worker thread and be deserialized again. I hope my quick & dirty description of a complex matter like this is halfway understandable… 😉

[pkerpedjiev]

Thanks for the quick reply!

I think it would be really useful for packaging libraries.

A few months ago you helped me out with this issue: #156, which was a step in that direction. Since then, I've realized that if the main script is on unpkg, and the web page is my.domain.com, then cross-origin worker blocking will prevent the worker script from being loaded from unpkg.

Would it be possible to?

1. Build the worker script using webpack to resolve the imports, serialize it as a string and then include it in the main script?

Or...

2. Perhaps even just include serialized versions of the imports in the worker script itself Like here: https://github.com/flekschas/piling.js/blob/a55a359cc86a40c5a7ef4cab8a753e555544fdb6/src/umap-worker.js#L10
   And here:
   https://github.com/flekschas/piling.js/blob/a55a359cc86a40c5a7ef4cab8a753e555544fdb6/src/umap.js#L11

[andywer]

Would it be possible to?

1. Build the worker script using webpack to resolve the imports, serialize it as a string and then include it in the main script?

I guess so.

2. Perhaps even just include serialized versions of the imports in the worker script itself Like here (...)

I think you should be able to call importScripts() in your workers as you please and using importScripts() might serve as a viable work-around around those nasty same-origin restrictions.

[ericblade]

FWIW, the webpack worker-loader claims to support this. I can't find anyone who's actually used it, though, and there's a bug filed on the worker-loader that it ignores all options, and it's nearly a year old bug with no useful responses.

I'm looking for any way that it is possible to do this -- a previous incarnation of support for this inside the library that i now maintain, did exactly what @pkerpedjiev suggested -- it had a custom WebpackPlugin that would bang up all the code in the library output as a BLOB and then insert it into the end of the bundle as a string. That code no longer functions in Webpack 4 (when i got this lib, it was using webpack 1 or 2), and I haven't found an obvious way of getting there yet that actually works.

Without doing that, you have to add in the ability to serve the additional js file, and that's a total non-starter of an idea for this lib that needs to be distributed as a single bundle file.

[andywer]

I guess the best way forward here is to move one step at a time. Let's focus on being able to spawn workers from blobs first – that shouldn't be too hard.

Once this is done we can inline worker bundles and spawn them easily, so that then we can figure out an approach how to conveniently bundle the worker code and inline it in library code.

Next steps

• Define API for spawning worker from blob/text
• Implement and write tests

[andywer]

How would you like this API?

import { BlobWorker, spawn } from "threads"

// Spawn from blob
const thread = await spawn(new BlobWorker(workerCodeBlob))

// Convenience method: Spawn from string
const thread = await spawn(BlobWorker.fromText(workerCodeString))

Usage with webpack might look something like that:

import { BlobWorker, spawn } from "threads"
import WorkerCode from "raw-loader!../build/worker.js"

const thread = await spawn(BlobWorker.fromText(WorkerCode))

[ericblade]

I haven't actually personally used threads.js yet, i've just been scanning for loaders and plugins that might solve the issue, so i can't really comment as to how well that fits in with the existing API here, but it looks like it would be fine for my purposes. Might be worth mentioning that I'm also using TypeScript, so hopefully it would end up allowing me to also use TypeScript in the worker.. that seems to work pretty well with worker-loader.. but the blob doesn't work in worker-loader.

[andywer]

With that raw-loader concept you would probably have to do the webpack build in two stages which isn't ideal, but might be good enough for libraries.

Stage 1: Bundle worker(s)
Stage 2: Bundle master thread code, inlining the worker bundles using the raw-loader

That would work fine with TypeScript, but would make building the code a bit inconvenient as you would need to run webpack twice.

We could ask the webpack community if they have a better idea like a custom loader that combines building the sub-bundle and inlining it or so.

I think that proposed API or something similar will be necessary either way.

[pkerpedjiev]

We could ask the webpack community if they have a better idea like a custom loader that combines building the sub-bundle and inlining it or so.

It would be great if there was a way to do that.

If you have to run webpack twice, how would that affect running with webpack-dev-server?

[andywer]

If you have to run webpack twice, how would that affect running with webpack-dev-server?

I would hope that webpack in watch mode for the workers plus your usual off-the-shelf webpack-dev-server for the master thread code would do the trick. A solution that can do both using one webpack instance would be much better, of course.

[pkerpedjiev]

Hey, belated thanks for implementing this!

Is there a standalone example of how to use this somewhere? I'm having a little trouble putting the pieces together for what config should go where to get the snippet in the docs (https://threads.js.org/usage#blob-workers) working.

I'm also looking at the PR and trying to piece together the clues but feel like I'm missing something simple: https://github.com/andywer/threads.js/pull/249/files#diff-c8d188486f8554c76052d5184676abc7a629756d69416906479e01b73919739bR44

[pkerpedjiev]

To follow up on my previous comment. I set up two webpack files: one for the main script and one for the worker. I ran the worker webpack script first and then the main script using the example in the docs but the main script still creates a 0.worker.worker.js file (359K) which is much larger than the worker build from its own webpack script (52K)

[pkerpedjiev]

Sorry for the comment storm. Just wanted to include an update. I built the worker using its own webpack file:

...
module.exports = {
  output: {
    filename: 'worker.js',
    path: path.resolve(__dirname, 'dist'),
  },
  devServer: {
    contentBase: [path.join(__dirname, 'node_modules/higlass/dist')],
    watchContentBase: true,
  },
  target: 'webworker',
...

And the main js with its own webpack:

...
module.exports = {
  output: {
    filename: 'higlass-pileup.min.js',
    library: 'higlass-pileup',
    libraryTarget: 'umd',
    path: path.resolve(__dirname, 'dist'),
  },
...

Then the import:

import { spawn, BlobWorker } from 'threads';
import { PILEUP_COLORS, cigarTypeToText } from './bam-utils';

import MyWorkerWeb from 'raw-loader!../dist/worker.js';

And the instantiation:

      const worker = spawn(BlobWorker.fromText(MyWorkerWeb));

When running webpack serve ... I now get the following warning:

WARNING in No instantiations of threads.js workers found.
Please check that:
  1. You have configured Babel / TypeScript to not transpile ES modules
  2. You import `Worker` from `threads` where you use it

For more details see: https://github.com/andywer/threads-plugin

And when I load the browser I see the following:

Any thoughts / ideas / suggestions?

Edit

Tried changing the import to

import MyWorkerWeb from 'raw-loader!./bam-fetcher-worker.js';

but now I get the same error as in #307

[pkerpedjiev]

Oh snap! I think it's working. I had to add

  entry: path.resolve(__dirname, 'src/bam-fetcher-worker'),

To the worker webpack and now it seems to be creating one big bundle. Thanks a ton for implementing this feature!!!

[andywer]

@pkerpedjiev Awesome that you were able to figure this out already! Any ideas how to better document that feature? Maybe this is something that just requires a small example repo.

[ericblade]

As a person who really has absolutely no idea how to use webpack, and actively avoids it where possible :-D i have no idea how to make this work

[pkerpedjiev]

I just created a PR (#342) to update the docs. The solution is actually quite simple and I think it avoids having to do multiple webpack builds.