Transferring query results to worker

[sgilligan]

I’m enjoying using duckdb-wasm in the browser, and have a use case to transfer query results into a worker for post-processing. The result set is around around 9MB in size. I serialise to Arrow IPC with tableToIPC(queryResult) for transfer .. which takes around 500ms. Is this the correct approach to move the result to the worker? The 500ms serialisation noticeably locks the main thread and I can’t see that is avoidable?

Cheers,
Simon

[domoritz]

Huh, since DuckDB wasm already uses arrow ipc to send data from the db worker to the main thread, I think it should be easy to redirect the stream to another worker. I'd need to check whether have api for that already but if not, we should add that.

Another approach could be to initiate the query from the worker. Have you tried that?

[sgilligan]

Yes, perhaps I can, but my goal is to immediately display tabular query results in the main thread (via react-virtualized), and crunch some analysis in a worker by iterating across the results with a bunch of manual aggregations and intelligence logic specific to our business. I'll return that output as data insights/graphs/maps when done. I'll poke around and look at the worker movement internally to see if I'm not missing something with that serialisation hit.

[domoritz]

I haven't tried it but try https://shell.duckdb.org/docs/classes/index.DuckDBConnection.html#send. The https://arrow.apache.org/docs/js/classes/Arrow_dom.RecordBatchReader.html seems to let you redirect the stream without the step through serialization.

[sgilligan]

Thanks for pointing me in the right direction :) I'm not sure if this is the right approach, but I've monkey-patched in a new query method that copies and returns the buffer as well as the table. I can then pass the buffer into a worker as I want to, without the serialisation.

Patch: https://gist.github.com/sgilligan/996a6021601fd4f71535c8877b69cbab