Feature Request: Masonry: Surface/Texture widget

[lwansbrough]

I’d like to be able to directly draw to a surface or texture within Masonry using wgpu APIs. Masonry would control the visibility, location, size, etc. of the view and allow me to implement a widget which is in charge of rendering to it via wgpu APIs.

[PoignardAzur]

Something like the canvas element in web browsers?

[AdrianEddy]

yes, something like canvas, I'm also interested in this feature. For example to create a video player, we'd need a widget that we can draw anything on (in this case pixels of the video), or a widget where we can draw some 3D scene with wgpu, etc

[DJMcNab]

Conceptually, this isn't too difficult, because Vello does support using images.
There are however a few challenges:

1. Knowing when to redraw the scene/getting control into your draw function, ideally without losing all power efficiency
2. Wiring up support for wgpu Textures into Vello - currently I think its API only supports images with provided data, as far as I know. I don't think there's anything conceptually challenging for this.

Another note:

For example to create a video player

So yes, this widget could be used to support a video player. However, the "correct" way to create a video player is to connect with the compositor (see #xilem>Compositor).

Overall, I'd encourage contributions towards this direction, but I can't prioritise working on this myself.

[PoignardAzur]

Same. The concept is interesting, but it's going to be pretty low on the priority list this year. Next year may be different.

[AdrianEddy]

I assume that this also applies to drawing any custom scene with vello? Eg. if I want to draw a chart

[PoignardAzur]

I assume that this also applies to drawing any custom scene with vello?

You can do so with a custom widget, eg any type implementing the Widget trait. Basically you choose what to paint in your paint() method, and Vello works its magic with the Scene you built.

(Yes, there talks of limiting custom widgets, but people were a lot more opposed to the idea than I assumed.)

Directly using wgpu APIs is another beast, though. I haven't sketched up a design, but I think it would require us to hook into the compositor and handle composition layers; there would be many good reasons to do that (performance, power efficiency, the embedded video use-case), but it's not on our immediate roadmap.

[gfxenjoyer]

Raph mentioned two-way interoperability with bevy in his RustLab Vello talk. My goal is to make a Xilem app that can embed various small bevy applets for 3D features. I'm curious if using the compositor would still be the best practice for embedding a bevy scene?

Also, Iced has a widget that exposes a wgpu texture called shader (example).

[PoignardAzur]

Raph mentioned two-way interoperability with bevy in his RustLab Vello talk. My goal is to make a Xilem app that can embed various small bevy applets for 3D features. I'm curious if using the compositor would still be the best practice for embedding a bevy scene?

Probably.

Other options are:

• Rendering to a texture, reading that texture back to CPU memory, and using it as an image widget (horribly inefficient).
• Rendering the Bevy side to a GPU texture and doing some winit manipulations to render that texture inside the Xilem render, at which point we're pretty close to using the compositor, I think. (Don't quote me on this.)

Also, Iced has a widget that exposes a wgpu texture called shader (example).

Interesting. I'll look into it.

[DJMcNab]

Rendering the Bevy side to a GPU texture and doing some winit manipulations to render that texture inside the Xilem render, at which point we're pretty close to using the compositor, I think. (Don't quote me on this.)

I don't see why you have to go from one extreme to directly using the compositor? I really don't understand the aversion to passing the wgpu Texture produced from the Bevy render to Vello, for it to render.
Now obviously, using the compositor would be better, but that's a lot of work. But with the upcoming pass order work, I hope we can make these mostly orthogonal.

The main challenge here is scheduling when your Bevy widget gets control, as well as exposing the right APIs on the Vello side for using wgpu textures in a render.

[PoignardAzur]

Sure. I'm unfamiliar with GPU rendering, so I might just be overestimating how hard this is to do.

[AdrianEddy]

As for the "scheduling when your Bevy widget gets control" question, Qt has beforeRendering signal & friends, they are pretty nice, may give you some ideas: https://doc.qt.io/qt-6/qquickwindow.html#integration-with-accelerated-3d-graphics-apis