Communication via TypedArrays and ArrayBuffers

[my2iu]

Modern high-performance JavaScript makes a lot of use of typed arrays and array buffers. It's simply too slow to transfer data around using JSON and strings. It would be nice if there was a way to do bulk data transfers between c++ <-> JavaScript.

For example, I currently have an Electron app that does some printing in Windows. Printing one page involves generating 8.5in x 600dpi x 11in x 600dpi x 3 colors = 100MB of data that needs to be transferred from JavaScript to C++ so that it can be fed into Windows' printing APIs. I can generate that amount of data relatively quickly in JavaScript, but without a dedicated API for transferring binary data from JavaScript to C++, the data transfer can end up being a real bottleneck (I need to encode the data as a string using base64 and then send it over, and that's slow and unresponsive, so I need to break it up into separate units of work, etc.).

AB#28510494

[my2iu]

This is probably pushing the limits of what can be expected from a WebView, but it would be totally wild if you could use SharedArrayBuffers to share memory between JavaScript and C++. Then, you could start pushing the WebView into unusual domains like VR, video processing, high-end games, etc. Though, initially, just a bulk binary data copy would be enough for me.

[liminzhu]

This is really interesting. The general need of sharing binary data between native and JS makes sense. I think there are two quite different approaches you mentioned, we can discuss different scenarios and figure out what makes sense.

1. Bulk binary data transfer between JS and native.
2. One end owns the memory and the data gets projected to the other end.

Stepping back a little to your printing example, why are you trying to pass the data to native to call Windows printing API (and which API are we talking about here)? Can you use window.print web API?

[my2iu]

My app in a vector drawing tool, so I need complete control of the printing process to get good quality printing where things are properly scaled to the page size and match the printer resolution and whatnot. The window.print web API just doesn't offer deep enough native integration to work right.

My understanding is that all the Windows printing APIs have been deprecated except for the historical GDI Print API and the newest UWP printing stuff. I'm targeting the GDI printing stuff. Regardless, I'm using a vector graphics model that it too advanced to be represented with older vector graphics technologies like pdf or SVG or WMF or whatever, so I need to rasterize it myself.

The printing is just an example though. I've also been thinking about exporting animations as video from JavaScript, or doing CUDA processing on data, and other stuff, but I've been too afraid of that data transfer overhead.

[liminzhu]

Makes sense, thanks for sharing the scenario. I think we'll look into it (but maybe not in the immediate month or so due to priorities like shipping a .NET preview).

[kevin--]

You could try intercepting the data via POST on WebResourceRequested. In my experiments this has been a relatively straightforward and efficient way to transmit binary data to and from the C++ side. You can just write the ArrayBuffer into the body of the request and then can be reinterpret_cast to their C++ equivalents when you receive the data.

https://docs.microsoft.com/en-us/microsoft-edge/webview2/reference/win32/icorewebview2?view=webview2-1.0.992.28#add_webresourcerequested

[LiangTheDev]

A new API has been developed for this scenario. Please check it out (the API is supported in current stable WebView2 Runtime, version 107).
API spec: https://github.com/MicrosoftEdge/WebView2Feedback/blob/main/specs/SharedBuffer.md.
API reference: https://learn.microsoft.com/en-us/dotnet/api/microsoft.web.webview2.core.corewebview2environment.createsharedbuffer?view=webview2-dotnet-1.0.1466-prerelease.

[my2iu]

The documentation should be more clear about whether you receive an ArrayBuffer or SharedArrayBuffer on the JavaScript side. The current terminology of “shared buffer” can cause confusion. Is it an ArrayBuffer that is shared with native but JS can only transfer ownership to web worker threads? Or is it a SharedArrayBuffer that is shared with native and can also be shared with web worker threads? Ideally, it should outright say, “this is not a SharedArrayBuffer object” or something.

[my2iu]

It might also be useful to show an example of passing data from JS to native. I assume it’s not possible to create a shared buffer from JS, so the JS side would first have to message native side to create such a buffer and pass it into JS where it can be filled with data?

[LiangTheDev]

It is ArrayBuffer, not SharedArrayBuffer. The ArrayBuffer is shared between native and JS side. On JS side, you have to transfer ownership to worker thread if we want to access it from worker thread.
The sample code in API documentation is at https://learn.microsoft.com/en-us/microsoft-edge/webview2/reference/win32/icorewebview2experimental18?view=webview2-1.0.1466-prerelease#postsharedbuffertoscript. In the WebView2 sample app, related code is in https://github.com/MicrosoftEdge/WebView2Samples/blob/main/SampleApps/WebView2APISample/ScenarioSharedBuffer.cpp and https://github.com/MicrosoftEdge/WebView2Samples/blob/main/SampleApps/WebView2APISample/assets/ScenarioSharedBuffer.html.

[phildremi]

Is there any way to trigger atomic locks (or any other mechanism of blocking concurrent access)? JS has Atomics, but I haven't seen anything in those docs that would tell me how I can coordinate with the UI thread from the host and avoid race conditions.

[LiangTheDev]

The safest way would be to access the buffer and post a message to the other side to notify that the data is produced/consumed.

If we really want to access the buffer at about same time, our limited testing suggests that Atomics can be used from JS side while std::atomic can be used on native side. Test it in your scenario to verify.

[Take-A-Byte]

A new API has been developed for this scenario. Please check it out (the API is supported in current stable WebView2 Runtime, version 107). API spec: https://github.com/MicrosoftEdge/WebView2Feedback/blob/main/specs/SharedBuffer.md. API reference: https://learn.microsoft.com/en-us/dotnet/api/microsoft.web.webview2.core.corewebview2environment.createsharedbuffer?view=webview2-dotnet-1.0.1466-prerelease.

Hey @LiangTheDev , I could not find a way to use it with UWP in the latest stable release. Could you please help me with it?
FYI - I tried to use it with 1.0.1462.37 but it complains about not being able to find CreateSharedBuffer and PostSharedBufferToScript. Am I missing something?

[LiangTheDev]

The API is still in experimental, so you would have to use a pre-release SDK. As WinUI3 seems to only support public WebView2 SDK, you might have to wait for the API to go out of experimental, or switch to WinUI2 which allows using pre-release WebView2 SDK.

[venki-thiyag]

Is it possible to implement buffer pool using this shared buffer mechanism?

[rhnatiuk]

What is the current status of this feature? Is it C++ only, or is it available for .Net as well? It would be sooo cool, if ArrayBuffer type would be automatically marshaled as byte[] both ways, like strings or numbers are (in js-to-host and host-to-js calls or in PostMessage).

[mikeduglas]

Look at the SharedBuffer class:

CoreWebView2SharedBuffer Class

Shared Buffer Between Native Application Code and Script

[rhnatiuk]

@mikeduglas , excellent, thanks! This takes care of sharing from the host to JS. Is there anything similar to go from JS to host? (of course, one can invent a convoluted example where JS would request a buffer from the host, the host would supply it, then JS would fill it with data, and so on, but that sounds very ineffective, complicated, and error-prone).

[LiangTheDev]

The "convoluted example" is currently the way to initiate data sharing from JS to host. There is no other APIs to do it easier. Note that the buffer can be reused multiple times and hopefully that would reduce number of this round trip communication.

[aaronbloom]

If we really want to access the buffer at about same time, our limited testing suggests that Atomics can be used from JS side while std::atomic can be used on native side. Test it in your scenario to verify.

Would you be able to expand on this @LiangTheDev? Given that the WebView2 shared buffer is exposed as an JS ArrayBuffer type rather than a JS SharedArrayBuffer type, would it not exclude using JS Atomics?

[LiangTheDev]

If we really want to access the buffer at about same time, our limited testing suggests that Atomics can be used from JS side while std::atomic can be used on native side. Test it in your scenario to verify.

Would you be able to expand on this @LiangTheDev? Given that the WebView2 shared buffer is exposed as an JS ArrayBuffer type rather than a JS SharedArrayBuffer type, would it not exclude using JS Atomics?

According to https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Atomics:

They are used with SharedArrayBuffer and ArrayBuffer objects.

[aaronbloom]

Unfortunately then it seems only Firefox implements Atomics on ArrayBuffer right now: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Atomics#browser_compatibility

Currently I'm trying to avoid using PostSharedBufferToScript as it has to be called on the C# main thread. With that restriction it looks like buffer synchronisation has to be via the network.

[LiangTheDev]

Try it out. It actually works for WebView2's posted ArrayBuffer in our test.

[aaronbloom]

I've created a sample project here: https://github.com/aaronbloom/WebView2/blob/master/WebView2Atomics/MainWindow.xaml.cs#L49

I encounter a TypeError when using the WebView2 shared buffer with the JS Atomics API. Hopefully I've just misused the API.

[LiangTheDev]

Got what you are saying. There are certain Atomics APIs like wait that are only work with views to SharedArrayBuffer. Those APIs would not work. The other APIs like compareExchange should work. If the scenario requires wait, then it would indeed not work.

[aaronbloom]

Thanks for the update.

I've added each Atomic method into my sample, and it matches with what you are saying.

https://github.com/aaronbloom/WebView2/blob/master/WebView2Atomics/MainWindow.xaml.cs#L72

Output:

Trying to Atomics.add(sharedArray, 0, 1)
Atomics.add succeeded 0
Trying to Atomics.and(sharedArray, 0, 1)
Atomics.and succeeded 1
Trying to Atomics.compareExchange(sharedArray, 0, 1, 2)
Atomics.compareExchange succeeded 1
Trying to Atomics.exchange(sharedArray, 0, 1)
Atomics.exchange succeeded 2
Trying to Atomics.isLockFree(sharedArray)
Atomics.isLockFree succeeded false
Trying to Atomics.load(sharedArray, 0)
Atomics.load succeeded 1
Trying to Atomics.notify(sharedArray, 0, 1)
Atomics.notify returned 0, no waiters were notified
Trying to Atomics.or(sharedArray, 0, 1)
Atomics.or succeeded 1
Trying to Atomics.store(sharedArray, 0, 1)
Atomics.store succeeded 1
Trying to Atomics.sub(sharedArray, 0, 1)
Atomics.sub succeeded 1
Trying to Atomics.wait(sharedArray, 0, 1)
Atomics.wait failed TypeError: [object Int32Array] is not a shared typed array.
    at Atomics.wait (<anonymous>)
    at EventTarget.<anonymous> (<anonymous>:112:19)
    at EmbeddedBrowserWebView.<anonymous> (<anonymous>:1:41422)
Trying to Atomics.waitAsync(sharedArray, 0, 1)
Atomics.waitAsync failed TypeError: [object Int32Array] is not a shared typed array.
    at Atomics.waitAsync (<anonymous>)
    at EventTarget.<anonymous> (<anonymous>:120:34)
    at EmbeddedBrowserWebView.<anonymous> (<anonymous>:1:41422)
Trying to Atomics.xor(sharedArray, 0, 1)
Atomics.xor succeeded 0

Given that Atomics.compareExchange works, it allows us to do cross process synchronisation, which is great!