-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
2D context zero copy from WebAssembly #5173
Comments
It seems more natural to me to offer |
Can you please elaborate on view into which bytes can be written? On Canvas side? |
Is there any progress for |
/cc @whatwg/canvas . This seems like a pretty easy win if someone wants to take the time to do spec/tests/implementation. |
IMHO, it would be best to map the canvas's backing store to a buffer at context creation time. Otherwise it's hard for the browser to know from the start that it should use VM-accessible memory. Also this does not need to be tied to WASM. We could do something that also works well in JS. Something like:
When the context is created in this way, the browser would know to stay away from asynchronous rendering to prevent synchronization issues between canvas commands and direct buffer access. I would also suggest that the size of the canvas should be immutable after context creation. WDYT? |
To hit two birds with one stone, the 'storage' context creation attribute could also point to a WebGL texture, for zero-copy render to texture. Anyways, that's a whole other feature so let's not get too deep into that here. |
I suspect that would require much more efforts to implement (at least for Chromium). And this would also mean canvas context has slightly different behavior and it might also hit performance when developer rarely needs to access a frame buffer. So the idea is nice, but it's rather something different from reusing |
These are very cool ideas! |
Could some more use cases be described? Allowing Is it strongly desired to be able to write to the canvas's backing store from WebAssembly with zero copies? CanvasRenderingContext2D implementations are preferentially GPU-accelerated in multiple browsers nowadays, so specifying a backing store region would imply fallback to CPU rasterization. There would also be a necessary copy by the browser's compositor on the way to the screen, to avoid displaying incomplete rendering results. Is |
There's definitely a desire for cpu-rasterized canvas work with zero-copy. We use something similar internally in Gecko fairly frequently. GPU-accel is mostly needed for screen-sized buffers, but smaller parts are often better suited to CPU. |
The main issue I see with the getImageData approach is that In Chromium we won't be able to implement a zero-copy version of that. By default 2d canvas rasterization happens asynchronously, usually on the GPU, and soon it will even be out-of-process. We maintain a software rendering option that can be activated with canvas.getContext('2d', {willReadFrequently: true}). The 'willReadFrequently' attribute is a hint to the browser that it should optimize for readbacks (i.e. getImageData), and since GPU readbacks are painfully slow this implies rendering on the CPU. However, even with synchronous in-thread CPU-based rendering, we cannot provide a zero-copy implementation of getImageData because the backing store is in memory that is not accessible to script or WASM, so a copy is still required. Providing a storage buffer at context creation time would allow for a true zero-copy implementation, but as Ken stated, we'd still need to make a copies for display purposes. These copies would happen once per animation frame, only if the canvas contents have changed, and only if the canvas is visible on screen. The big advantage I see with the storage buffer approach is that it simultaneously provides zero-copy read and zero-copy write access. User code could seamlessly interleave 2d canvas API calls with direct pixel reads and writes, without any need to ever call getImageData or putImageData (you just access the buffer directly), so there would be less API call overhead, and without any buffer copying until all drawing is done and it is time to display. If you only intend to call getimageData once per animation frame then the two proposals (getImageData vs. script-provided storage) are equivalent in terms of number of buffer copies. But as soon as you need to make more than one call to either getImageData or putImageData, the storage buffer approach wins in terms of reducing the number of buffer copies. Also, if the canvas is not displayed (e.g. used for background rendering), the storage buffer approach wins. Also worth noting, both of these approaches avoid creating large temporary objects that would need to be garbage collected (compared to the current API that creates ImageData objects). That's nice. |
For JS apps I think it would be ok to have one-copy getImageData API. It solves only one problem - unnecessary memory allocation on every call of it (and as a consequence higher load for GC). It's nice if at some point we'd be able to have some kind of zero-copy API, but as @junov stated - it might be very tricky to have it even designed (it'd be a shared memory between browser internals and JS/WASM - that I find to be quite dangerous). |
Could it be achieved doing from scratch ? Re-writing specification that can later be used with WASM in a proper sense ? I am thinking about Go + Skia (bindings is my issue) or C + Skia directly ? |
Currently interfacing between Canvas and WebAssembly is done this way:
get
CanvasRenderingContext2D
formCanvas
get
ImageData
from - getCanvasRenderingContext2D
formCanvas
copy array into Wasm memory
CanvasRenderingContext2D
formCanvas
ImageData
from an array slice from Wasm memoryImageData
inCanvasRenderingContext2D
There is a WebAssembly proposal to for mutiple memory access, which would allow web assembly modules accessing more than one memory object. If
CanvasRenderingContext2D
orCanvas
would export a WasmMemory
object, then a lot of copying and instantiation described above can be eliminated.WebAssembly/multi-memory#7
The text was updated successfully, but these errors were encountered: