Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker threads and C++ backed objects #2754

Closed
Flackus opened this issue Jun 4, 2020 · 7 comments
Closed

Worker threads and C++ backed objects #2754

Flackus opened this issue Jun 4, 2020 · 7 comments

Comments

@Flackus
Copy link

Flackus commented Jun 4, 2020

  • Node.js Version: 12.16.3
  • OS: Linux
  • Scope: runtime
  • Module (and version): worker_threads

Hi, I've been recently playing with worker threads as in version 12.16.3.

The doc explicitly says that you can't pass a C++ backed object via postMessage: https://nodejs.org/dist/latest-v12.x/docs/api/worker_threads.html#worker_threads_port_postmessage_value_transferlist
And this is true, you can't :) The standard makes it pretty obvious.

Any chance someone may shed some light on why is it so? Or even better, maybe someone has already found a workaround?

@addaleax
Copy link
Member

addaleax commented Jun 4, 2020

Any chance someone may shed some light on why is it so?

postMessage() needs to be able to serialize the value that it is being passed, but if it contains a C++-backed object, there is no generic way to know how to serialize those.

We do have a special handler for MessagePorts in place there, and are expanding that functionality in nodejs/node#33360. After that, it would not be unreasonable to also provide a public C++ API that addon authors could use to make their own objects transferable/cloneable.

@Flackus
Copy link
Author

Flackus commented Jun 4, 2020

@addaleax thanks for a quick response. I have a follow-up question if it's ok.
At the moment is it reasonable to play around custom v8 serialiser for C++ objects or it wouldn't help with transfer issues?
Even if we can make an object transferable (now or in future), it still would be a pack/unpack routine, right? So we wouldn't be able to pass an object without copying?

@addaleax
Copy link
Member

addaleax commented Jun 4, 2020

@Flackus Not sure what a custom V8 serializer would look like – of course you can write your own serializer/deserializer functions (e.g. based on https://www.npmjs.com/package/serdes) and serialize/deserialize the C++ objects yourself.

For objects that Node.js transfers internally, like the MessagePort objects, it does transfer them without extra serialization and deserialization, i.e. the C++ object identity remains the same.

@Flackus
Copy link
Author

Flackus commented Jun 4, 2020

@addaleax thanks, things are getting clearer.

Not sure what a custom V8 serializer would look like

Me, too, actually :) I'm just considering every option here.

Also, you said:

After that, it would not be unreasonable to also provide a public C++ API that addon authors could use to make their own objects transferable/cloneable.

Does that imply that there would be a way to transfer custom objects without extra serialization and deserialization? It would be a powerful tool IMO.

@addaleax
Copy link
Member

addaleax commented Jun 4, 2020

After that, it would not be unreasonable to also provide a public C++ API that addon authors could use to make their own objects transferable/cloneable.

Does that imply that there would be a way to transfer custom objects without extra serialization and deserialization? It would be a powerful tool IMO.

Yes, exactly. For example, when MessagePorts are currently being transferred, no serialization or deserialization is performed on their internals.

@Flackus
Copy link
Author

Flackus commented Jun 4, 2020

@addaleax thank you, now it's clear. Looking forward to it to happen :)

I think I'll just close the issue now as I don't have further questions. Thanks again for you help.

@Flackus Flackus closed this as completed Jun 4, 2020
@libingyu96
Copy link

libingyu96 commented Jan 2, 2024

Hi @addaleax

After that, it would not be unreasonable to also provide a public C++ API that addon authors could use to make their own objects transferable/cloneable.

Does Node.js now provide such a public C++ API ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants