Displaying wgpu rendering results as a widget

[MatchaChoco010]

Hello everyone!
I would like to make the following contributions to this project.
I would like to discuss how to contribute.

---

I would like to use iced as a GUI for my own game engine.
And for this purpose, I want to place the drawing result of wgpu as an iced widget.

I think this example is to place the iced GUI on top of the wgpu drawing result, not to display the wgpu drawing result as one of the widgets.
https://github.com/iced-rs/iced/tree/master/examples/integration_wgpu

With the current functionality of iced, displaying wgpu rendering results as a widget seems to be difficult to achieve.

---

There is a PR for a proposal to add this feature to iced, but the discussion has stopped midway.
#903
I would like to take over this proposal.

The PR in the proposal above suggests a much hackier approach.
In the last comment of the PR, akhilman says it would be nice to be able to draw the result of wgpu drawing in the following way

After all rendering loop may look like:

• Render 3D objects to offscreen buffer
• Draw GUI with that buffer as Image

I also think it would be simpler and better to just show the off-screen rendered version as this comment says.

---

The image of displaying the off-screen rendering of wgpu in iced is expressed in pseudo-code as follows gist.
https://gist.github.com/MatchaChoco010/ac142cbf07ca6d4b2a6880bbf868afe0

WgpuOffscreenRenderer is the application that renders off-screen.
The texture rendered by this app is passed to iced with self.wgpu_offscreen_renderer.render_texture() to draw it.

---

But here's the challenge.

How do I prepare the device and instance that wgpu_offscreen_renderer needs?

I think there are two methods to do this.

1. Make the instance and device used by iced_wgpu available externally.

In the iced_wgpu backend, the compositor in wgpu/src/window/compositor.rs holds the instance, device, and queue of the wgpu.
These instances and devices are not exposed to the Application as APIs at present.
How about preparing an API to make them visible from outside?

If you prepare an API that makes Compositor visible to the application, and pass Compositor as an argument of new in the application, the device and instance will look usable.

However, this is not enough.
Compositor requires itself to be mutable in the "present" method.
If the Compositor has a method that returns a reference to the device, and the Application tries to get the device of the Compositor, the Compositor that holds the mutable device and the reference to the device exist at the same time, resulting in a compilation error.

Since the wgpu backend of iced uses device and instance as mutable internally, it is not possible to pass the reference of device and instance to the outside as it is.
The solution is to make the Compositor hold them as Arc<Mutex<wgpu::Device>>.
However, this solution will incur extra cost even if you are not doing off-screen rendering.
It does not seem to be a desirable solution.

Pros:

The implementation is closed only to iced.
It can be implemented with less implementation cost.

Cons:

It will be costed by Arc<Mutex>.

2. Prepare a different instance and device from the iced_wgpu device and instance.

Instead of letting the device and instance used internally by the iced_wgpu be used externally, there is a way to create a new device and instance for a different wgpu.
However, this method is a bit more difficult to implement.

If the device used for iced is different from the device used for off-screen rendering, it is difficult to exchange textures between the different devices.

For example, in the case of Vulkan, extensions such as VK_KHR_external_memory can be used to share resources such as textures used by other logical devices.
If you use this feature to exchange textures, you can do off-screen rendering on a different instance or device than the iced_wgpu.

Unfortunately, the Vulkan VK_KHR_external_memory feature has not yet been implemented in the wgpu.
This PR talks about how to use external texture in wgpu.
gfx-rs/wgpu#965

If we want to use another wgpu's off-screen rendered texture with the external texture feature, we need to implement the external memory feature in the body of the wgpu.

Pros:

No need for Arc<Mutex>.
It may be possible to display external textures drawn with OpenGL, Metal, D3D12, etc., not just wgpu.
Apps that render off-screen can be completely separated from iced.

Cons:

You have to add functionality to the wgpu itself. It looks like a lot of work.

---

Which method (or another method) do you think I should use to take over the proposal, implement it, and submit the PR?

Also, will I get any support from the community in implementing this feature?
If I use method 2, I would like some support as it seems to be very hard and difficult for me to add functions to wgpu by myself.