Does NodeJS have any plans for shared memory?

[hitsthings]

So this API is being added to JS that allows different execution contexts to share a chunk of memory safely with interleaved access.
Meanwhile the Web Worker spec already exists for creating these separate execution contexts.

I know there are pre-existing user space modules for web-worker implementations. But they either 1) use separate processes for each thread (and thus can't really share memory well), or 2) they literally expose all memory to all threads and thus accept all the chaos that other multithreaded languages often have.

Combined, the Web Worker and Shared Memory APIs seem like a solid infrastructure to build multithreading on. There would be no bad interaction with old code since the old code would have to opt in to using the shared buffer (or a bad dev would have to pass a shared buffer to the old code and then write to it). Is there any plan for making use of shared memory and web workers in Node?

[bnoordhuis]

Is there any plan for making use of shared memory and web workers in Node?

Not at the moment. It's not completely out of the question but:

1. Web workers are straightforward to implement through a native module. If the existing modules aren't any good, that doesn't mean it's impossible, just that no one has done it yet.

2. Cross-platform shared memory and synchronization is a bit of a viper's nest. Node would have to support it out of the box on all the platforms it supports, a native module can get away with supporting just the three major ones: Linux, Windows, TOSFKAOSX (The Operating System Formerly Known As OS X.)

[hitsthings]

Thanks for the info :) Wasn't expecting such a fast answer. Thanks for the alternative of a native module, too!

[addaleax]

Also, just fyi: https://github.com/audreyt/node-webworker-threads :)

[hitsthings]

Thanks @addaleax ! I was under the impression those threads still shared memory, but sounds like I'm wrong. Also I realized that I kinda completely forgot about https://nodejs.org/api/vm.html and this whole thing is probably just a derp. Thanks for setting me straight!

[sam-github]

I don't know what a derp is, but the VM API is completely unrelated to threading or concurrency.

[hitsthings]

But it does allow executing code in a separate context. If a threading library used it to isolate the memory that different threads have access to, they could still pass sharedArrayBuffers between those threads' contexts. At least I think they could - my understanding of all of this is very high level and I'm sure there's a devil in the details.

…

On 5 Apr. 2017 00:20, "Sam Roberts" ***@***.***> wrote: I don't know what a derp is, but the VM API is completely unrelated to threading or concurrency. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#560 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmkBH-OOra8RQ0nYRlXic8KUJhWFf_Yks5rslGYgaJpZM4MyvaS> .

[bnoordhuis]

@hitsthings Contexts are tied to their isolates, and those are not thread-safe. That is, you can't have two contexts in the same isolate and access them from different threads.

(You can pass SharedArrayBuffers between contexts but there won't be any real concurrency, it'll be strictly sequential.)

[ORESoftware]

@hitsthings I would highly recommend just using different node.js processes instead of the node-webworker-threads lib; unfortunately the node-webworker-threads lib is not very stable and has major API deficiencies last time I checked.

[hitsthings]

Thanks, I get that separate processes is the current standard. I posted this originally because shared memory seemed like a great opportunity for threads. Most cases are good with share-nothing concurrency which is what you get with processes (and anything you want to pass between them is slower over IPC afaik). I had a case a few years back where I was trying to implement real-time tracking and mapping for nodecopters (for fun, so meh on stability). One big performance hurdle was that I needed to pass large chunks of image data, edge detection data, etc between processes. That's where threads would help since I wouldn't have to pass anything around to access it in multiple threads.

To be clear - I'm not asking about doing anything in production code. Was just curious about whether there were any ideas/plans for threading with sharedArrayBuffers in node that might help me with problems like that.

Ben - thanks for the VM clarification.

[sam-github]

you can share memory between processes as well as between threads, bunch of stuff on npmjs, I vouch for the quality of none of it :-) https://www.npmjs.com/search?q=shared+memory

[icodeforlove]

Would love to see some sort of equivalent for Workers, transferable objects and SharedArrayBuffer's.

node-webworker-threads is probably the closet you can get at the moment and from my experience its fairly unstable.

As it currently stands it is easier to take advantage of all cores and memory in the browser vs node. Which makes me sad :(.

[addaleax]

@icodeforlove Hey, just so you are aware: nodejs/node#13143, https://github.com/nodejs/worker. :)

[refack]

@icodeforlove Hey, just so you are aware: nodejs/node#13143, https://github.com/nodejs/worker. :)

@addaleax talk about "hitting the nail of the head" 😄
@icodeforlove You are most welcome to participate in the discussion (esp. nodejs/worker#1)

[Pacerier]

@ORESoftware, @addaleax, That's not shared memory. It's using message parsing which is slow.