Run Bevy in web worker context

[haibane-tenshi]

Objective

This is a tweak to Bevy which (potentially) allows it to run as a web worker.

Related work

This PR is based on those two PRs:

• Enable creating "virtual" windows without corresponding OS window #6256
• Implemented support for creating a render surface from a canvas or offscreencanvas #6147

Although at this point it is almost an entire rewrite since there was a lot of changes to related Bevy APIs during 0.9 -> 0.11.
Regarding the state of those two:

• The first PR seems to be no longer applicable.

  The purpose of that PR was to change Window.raw_window_handle to an option to allow to create windows without an associated raw handle. However, the field since was removed. Now, WinitPlugin is in full control over creation of raw handles. Other changes are mostly various helper methods.

  I would suggest closing this one, as there is no hope for it to get merged in this state and there is no tangible way forward without a full rewrite.

• The second PR in large parts also seem to be no longer applicable due to API flux around Window struct. Current PR borrowed ideas from it but adapted them to modern version of Bevy.

Regarding the state of this PR: it isn't intended for merge in this form, but rather it is an exploration of how much effort is needed to make Bevy run on web workers. I wanted this for my pet project, but also needed a newer Bevy version compared to the other PR.

Also, relevant issue: rust-windowing/raw-window-handle#102

High-level summary

The core issue we are trying to address is correct creation of wgpu::Instance. On the web it requires types from web_sys (HtmlCanvasElement or OffscreenCanvas) to properly initialize WebGL context.

Instance relies on RawWindowHandle to provide rendering context. However, RawWindowHandle::Web contains a single u32. It makes sense in context of raw-window-handle library, but causes issues for us.

wgpu::Instance internally uses the integer to look up corresponding canvas element via css selectors - and this is where it goes wrong. There is no DOM access from inside worker thread.

We combat this by introducing an AbstractHandleWrapper which can hold either RawHandleWrapper or web canvases.

Changelog

• Introduced AbstractHandleWrapper

  enum AbstractHandleWrapper {
      /// The window corresponds to an operator system window.
      RawHandle(RawHandleWrapper),
  
      /// A handle to JS object containing rendering surface.
      #[cfg(target_arch = "wasm32")]
      WebHandle(WebHandle),
  }
  
  #[cfg(target_arch = "wasm32")]
  pub enum WebHandle {
      HtmlCanvas(HtmlCanvasElement),
      OffscreenCanvas(OffscreenCanvas),
  }

• Replace usage of RawHandleWrapper with AbstractHandleWrapper including inside ECS.
  More specifically it affected implementation of RenderPlugin::build, bevy_winit::system::create_window, bevy_render::view::windows::prepare_windows and changed type of ExtractedWindow.handle.

• Added bevy_window::Window.web_element: WebElement which allows to specify how Bevy should discover rendering context on web platform.

  pub enum WebElement {
      // Replicates existing default behavior.
      #[default]
      Generate,
  
      /// Discover `HtmlCanvasElement` via a css selector.
      ///
      /// # Panic
      ///
      /// This option will panic if the discovered element is not a canvas.
      #[cfg(target_arch = "wasm32")]
      CssSelector(String),
  
      /// Use specified `HtmlCanvasElement`.
      #[cfg(target_arch = "wasm32")]
      HtmlCanvas(web_sys::HtmlCanvasElement),
  
      /// Use specified `OffscreenCanvas`.
      #[cfg(target_arch = "wasm32")]
      OffscreenCanvas(web_sys::OffscreenCanvas),
  }

  The field is ignored on non-wasm targets.

• Removed bevy_window::Window.canvas as its functionality was subsumed by web_element.

• Changes to logic of WinitPlugin to properly take Window.web_element into account.

Fallout

It is important to keep in mind Bevy can be run in two different modes on web: on main event loop or inside worker.

When running Bevy as main situation remains largely unchanged. Internally, we end up eagerly committing canvas into ECS and using that instead of discovering it every time using selectors. Externally, users get an ability to provide canvas object directly - minor ergonomic improvements, I guess?

However, currently WinitPlugin will panic when encountering any window with WebElement::OffscreenCanvas. The reason for that is simple: we cannot create a winit window out of offscreen canvas - which causes it to create a new HtmlCanvasElement.
The result is confused state: we render to OffscreenCanvas but listen to events of some other unrelated element. To prevent this WinitPlugin disallows creation of windows out of offscreen canvases.

When running Bevy as worker things are... well, let's call it manageable. There are lots of sharp edges:

• Attempt to create window with anything but WebElement::OffscreenCanvas is doomed to panic:

  • Generate and CssSelector require access to DOM which worker doesn't have.
  • Workers cannot directly render to HtmlCanvasElement, so even if you get one (which I believe you shouldn't be able to because it is not transferrable) it won't work.

• By extension it implies that including WinitPlugin results in guaranteed panic: it is unable to create windows out of OffscreenCanvas as already discussed.

• This implies that we need to set up event loop ourselves since that is job of WinitPlugin. Also default event loop panics too, but I didn't look into that yet.

• This also implies the need to manually pipe events through to Bevy as that is also job of WinitPlugin.

I tried to teach WinitPlugin to work with "virtual" windows (e.g. Bevy windows without a backing winit window),
but that was messy. It will require some sweeping refactors to bevy_winit crate to clean up the logic.

Open questions

I don't mind tinkering with this more as I go, but there are some questions about directions.

• AbstractWindowHandle is a weird beast.

  First part (which I didn't fix) is that RawWindowHandle is completely ignored on wasm. So the type is less of a enum, but more like a struct with platform-dependent fields.

  Second part, we cannot avoid platform specific code around instantiation of wgpu::Instance because of RawWindowHandle. Or could we?

  Integer provided by RawWindowHandle::Web is ultimately for us to interpret. Its just Instance uses it for css selector query. One idea that I had is instead to use the index to look up canvas objects in global JS scope. It can be done either inside Bevy (by intercepting RawWindowHandle before it is passed to Instance like it is done here) or via introducing change to wgpu.

  Downsides of such setup are relatively obvious: this is rather obscure for anyone coming from the side and it is also prone to accidental breakage.

  The upside - platform-specific code around rendering is moved inside wgpu. Bevy still needs to put canvases where it can be discovered, but it will be mostly transparent to users.

  I guess this is an addition to discussion in linked issue.

• WinitPlugin and OffscreenCanvas support.

  As indicated by one of the linked PRs there is some interest in virtual windows (e.g. Bevy windows without backing winit/OS? window). But as I already mentioned, this will require some large refactors to bevy_winit, so it is better suited for a separate PR.

  After that is sorted out, this PR can use the new machinery to properly support offscreen canvases.

  I guess another option is to maybe try to craft self-made frankenstein winit windows, but I didn't look into that either.

• General experience for running on web worker.

  What do we do about sharp edges? I guess most sensible option is to introduce a feature which disables nonsensical options in WebElement and WinitPlugin, but that leads to more issues.

  For example, Window cannot implement Default for web workers (since we cannot provide a good default OffscreenCanvas) which falls out into more complexity for other code (e.g. WindowPlugin cannot create default primary window).

  Without WinitPlugin there is no one to make window setup for rendering. There is no event loop either. Also, how do we forward input events to Bevy?

  I think those questions loosely point toward the following:

  • web-worker feature is probably a good idea to fence off unusable functionality.

  • We might need a separate set of default plugins for web workers.

  • WinitPlugin does a bit too much:

    • Create event loop
    • Generate rendering context (+external windows) out of window descriptors
    • Associate rendering context with bevy windows
    • Convert external input events into Bevy events

    It made sense while everything could be covered by winit, but it isn't anymore. It could be good to split the plugin into smaller more modular pieces.

And, obviously, name bikeshedding is always open.

Anyway, there you have it.

Migration guide

TODO

[github-actions]

Welcome, new contributor!

Please make sure you've read our contributing guide and we look forward to reviewing your pull request shortly ✨

[haibane-tenshi]

I also put together a simple working example here.

[cormac-ainc]

Hi, this is really good work, thank you @haibane-tenshi, and I hope it gets some more attention.

We are using bevy inside a broader HTML app, and need to have the app remain responsive while the game part chugs, which it must on load. Having it off the main thread is pretty crucial. Without a web worker, the alternative is booting up an iframe, serving it from a completely distinct origin, and relying on modern browsers to spin up a separate thread because the origin is different, thus emulating a web worker. This is quite complicated to do, you can't just whack the iframe html+js+wasm on bevy.example.com because browsers consider that to be part of the same site as www.example.com and do not isolate it. There is a new header Origin-Agent-Cluster (same article) for specifying process isolation with a site, that is currently only supported by Chrome, basically.

Basically, if you want process isolation today, you need to buy another domain.

It would be great if bevy could support passing in an OffscreenCanvas and rendering to it from a web worker. It seems the main caveat is that winit can't subscribe to resize events on its own because it won't have access to the DOM tree -- am I getting that right? Is there anything else keeping this PR a draft?

[daxpedda]

It would be great if bevy could support passing in an OffscreenCanvas and rendering to it from a web worker. It seems the main caveat is that winit can't subscribe to resize events on its own because it won't have access to the DOM tree -- am I getting that right?

The current beta release of Winit has resolved this and other related issues already.

See rust-windowing/winit#2778 and rust-windowing/winit#2834.

[cormac-ainc]

I have looked into all this further. Sadly I don't think bevy in a worker context is going to work anytime soon.

The winit PR 2788 would suffice for the following scenario, aka #4078 bevy multithreaded on wasm:

• Bevy is launched on the main thread with a normal HtmlCanvasElement
• Bevy spawns its own web worker(s)
• Bevy's main thread calls transferControlToOffscreen() internally and sends the OffscreenCanvas to a render thread on a web worker. Probably postMessage is the only way to do that.
• winit beta's request_inner_size (formerly set_inner_size) should only be called from the main thread. 2788 makes this work on winit's end.
• The renderer thread(s?) take on the responsibility for calling OffscreenCanvas::set_height and set_width, ultimately through wgpu::Surface::configure which is called by a system within bevy_render to configure surfaces initially and when a bevy_window::Window's size has changed.

This PR is a different approach, aka "bevy completely inside a web worker, including winit". I have realised that it's a bit more difficult than I thought. Winit running on a worker thread won't be able to receive any interaction events, like clicks. I'm not sure how rust-windowing/raw-window-handle#134 really helps. What would winit even do with a RawWindowHandle::WebOffscreenCanvas? There appears to be absolutely no API on OffscreenCanvas that can be of any use to winit. You would need a bunch of proxying code for event forwarding, The only way I can think of to get the events into winit would be constructing a fake HtmlCanvasElement that listens to window.onmessage and acts as an EventTarget. (In that case, you would try to convince winit it's a RawWindowHandle::WebCanvas.) It seems unlikely anyone will bother given how fragile that seems.

Single-threaded bevy in an iframe has none of these issues, as it can access all the DOM it likes within the iframe, with the aforementioned caveats about getting process isolation to work. So I think we'll stick with that for now.

Other people are similarly motivated to get at least some of bevy off the main thread and one of these options will end up being the path of least resistance. It will probably be multithreading, IMO.

[daxpedda]

FWIW, I've been using multi-threaded Wasm with Winit and rendering in a separate thread with Wgpu for a while now and it works just fine.

• winit beta's request_inner_size (formerly set_inner_size) should only be called from the main thread.

rust-windowing/winit#2834 makes Window Send + Sync, so you can call any method from any thread.

[allsey87]

I've been using multi-threaded Wasm with Winit and rendering in a separate thread with Wgpu for a while now and it works just fine

@daxpedda do you have a repo demonstrating this set up?

[daxpedda]

@daxpedda do you have a repo demonstrating this set up?

I don't, but I can make one!
Will post here when I get to it.

[allsey87]

@daxpedda do you have a repo demonstrating this set up?

I don't, but I can make one! Will post here when I get to it.

Just a reminder that I would be very keen to see this @daxpedda :)

[austintheriot]

I have realised that it's a bit more difficult than I thought. Winit running on a worker thread won't be able to receive any interaction events, like clicks. I'm not sure how rust-windowing/raw-window-handle#134 really helps. What would winit even do with a RawWindowHandle::WebOffscreenCanvas?

I can't speak too much to the winit side here, but, to take inspiration from pure JS land, in Three.js, the workflow for rendering to an offscreen canvas is described here: https://threejs.org/manual/#en/offscreencanvas

All click/touch interactions would need to be received "manually" via messages to the web worker from the main thread. It might be helpful to note that messages can be sent to workers via the worker.postMessage API, but also through the MessageChannel API. Both of these APIs allow sending transferrable objects to workers cheaply.

[allsey87]

@haibane-tenshi any thoughts on using the internal JsValue integer to identify the canvas? E.g., changing wgpu-hal to something like this:

unsafe fn create_surface(
        &self,
        _display_handle: raw_window_handle::RawDisplayHandle,
        window_handle: raw_window_handle::RawWindowHandle,
    ) -> Result<Surface, crate::InstanceError> {
        match window_handle {
            raw_window_handle::RawWindowHandle::Web(handle) => {
                let window = unsafe {
                    wasm_bindgen::JsValue::from_abi(handle.id)
                };
                match window.dyn_into::<web_sys::HtmlCanvasElement>() {
                    Ok(canvas) => {
                        return self.create_surface_from_canvas(canvas);
                    }
                    Err(window) => match window.dyn_into::<web_sys::OffscreenCanvas>() {
                        Ok(offscreen) => {
                            return self.create_surface_from_offscreen_canvas(offscreen);
                        }
                        Err(_) => {
                            return Err(crate::InstanceError)
                        }
                    }
                }
            }
            _ => Err(crate::InstanceError)
        }
    }

[daxpedda]

FWIW: see gfx-rs/wgpu#4888 and rust-windowing/raw-window-handle#157.

[allsey87]

FWIW: see gfx-rs/wgpu#4888 and rust-windowing/raw-window-handle#157.

@daxpedda those changes are now released as wgpu 0.19.1 and raw-window-handle 0.6.0. Would you be interested in co-authoring a PR to bump bevy to these versions so that we have offscreen support in the next release?

[daxpedda]

Would you be interested in co-authoring a PR to bump bevy to these versions so that we have offscreen support in the next release?

Unfortunately I don't believe I would have the time for that, but I don't think it's necessary as Bevy is already hard at work: #11280.

To whom it may concern: I've finally found some time reworking my threading library and closing in on a release, you can follow the progress in https://github.com/daxpedda/wasm-worker.

[allsey87]

Unfortunately I don't believe I would have the time for that, but I don't think it's necessary as Bevy is already hard at work: #11280.

Indeed, somehow I missed that.

To whom it may concern: I've finally found some time reworking my threading library and closing in on a release, you can follow the progress in https://github.com/daxpedda/wasm-worker.

Interesting stuff! I look forward to seeing some more examples/tests for your messaging API, in particular, I am curious as to how you handle stuff that needs to be posted or transferred. I have been working on a RPC implementation to solve this problem here: https://github.com/rustifybe/worker-rpc/tree/master/worker-rpc/tests

[allsey87]

@daxpedda I am trying out OffscreenCanvas in 0.13 but I am running into a familar issue:

When I construct a WebOffscreenCanvasWindowHandle I have the following

canvas = OffscreenCanvas { obj: EventTarget { obj: Object { obj: JsValue(OffscreenCanvas) } } }
handle = WebOffscreenCanvasWindowHandle { obj: 0xff5b0 }

However, when it's retrieved in wgpu/wgpu-hal/src/gles/web.rs this is what I see just before it fails to get the webgl2 context due to the inner JsValue(undefined).

handle = WebOffscreenCanvasWindowHandle { obj: 0xff5b0 }
canvas = OffscreenCanvas { obj: EventTarget { obj: Object { obj: JsValue(undefined) } } }

This is how I am trying to set things up:

struct OffscreenPlugin {
    handle: RawHandleWrapper,
}

impl OffscreenPlugin {
    fn new(canvas: OffscreenCanvas) -> OffscreenPlugin {
        let handle = WebOffscreenCanvasWindowHandle::from_wasm_bindgen_0_2(&canvas);
        OffscreenPlugin {
            handle: RawHandleWrapper {
                window_handle: RawWindowHandle::WebOffscreenCanvas(handle),
                display_handle: RawDisplayHandle::Web(WebDisplayHandle::new()),
            }
        }
    }
}

impl Plugin for OffscreenPlugin {
    fn build(&self, simulator: &mut App) {
        simulator
            .add_event::<WindowResized>()
            .add_event::<WindowCreated>()
            .add_event::<WindowClosed>()
            .world
                .spawn(bevy::window::Window {
                    resolution: bevy::window::WindowResolution::new(500.0, 500.0),
                    ..Default::default()
                })
                .insert(bevy::window::PrimaryWindow)
                .insert(self.handle.clone());
    }
}

have you seen something similar on your end?

[allsey87]

Ok, the OffscreenCanvas was dropped at the end of the new function. For now, I have changed my code as follows:

- let handle = WebOffscreenCanvasWindowHandle::from_wasm_bindgen_0_2(&canvas);
+ let handle = WebOffscreenCanvasWindowHandle::from_wasm_bindgen_0_2(Box::leak(Box::new(canvas)))

Ideally I would just store the OffscreenCanvas in my plugin struct, but unfortunately it is not Send which Bevy requires for plugins...

[daxpedda]

Ideally I would just store the OffscreenCanvas in my plugin struct, but unfortunately it is not Send which Bevy requires for plugins...

In Winit we use a MainThreadSafe container to make things Send + Sync that aren't really. You could apply this concept to any thread really.

See also the thread-safe crate.