Add XREnvironmentBlendMode to communicate display transparency

[toji]

Addresses #145 (finally).

We've talked in the past about an isTransparent boolean or a opaqueness scalar, but after looking at the various scenarios that we want to account for on the market today it really seems what developers will actually want to know if how their content will be "blended" with the real world. As far as I can see the answers are "not at all" for VR, "additively" for HoloLens and friends, or the extremely catchy "glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)" for ARCore/ARKit style passthrough rendering. Thus this PR attempts to communicate those states with more reasonable names.

By making this an enum we can extend it in the future, but I'm not sure how many other types of display we're likely to see. I guess I've heard people theorize about one where there's a sharp cutoff point between fully opaque and fully transparent pixels, but I'm not aware of any such display being available.

[blairmacintyre]

Curious: do we want to align this with w3c/csswg-drafts#2719

And, I know @kearwood and others in the past have expressed concern about knowing what the "this is transparent" color is ... seems related.

[toji]

Aligning verbiage with CSS values that communicate the same thing is a good idea, yes.

As for concerns about what a specific transparent color is, I'm not aware of any existing displays that use a feature like that, but if we find one I think the appropriate response is likely to add a color-key blend more (or similar) with an paired attribute on the session indicating what the alpha key color is.

As with a lot of things about this API, the principle of "add only what we need now, but know how we'll extend it when needed." applies here.

[thetuvix]

This proposal looks great! XrWorldBlendMode really zeroes in on the primary thing apps are trying to get at across the wide range of devices and display technologies by asking for isTransparent: can my user see the real world, and, if so, how are my pixels treated during composition on that world, regardless of whether they're composed optically or through video.

One thing we should think through as we close on #330 is whether the same world blend mode enum should be first used by apps during requestSession to request world blending in the first place, at least on optically-opaque devices that would default to VR, but can do video AR (e.g. Vive Pro).

Based on the display technology on a given device, there are three UA support levels for a particular world blend mode:

• ✅ Natural: The inherent world blend mode of the device’s display.
• ✔️ Supported: World blend modes that a UA could achieve on that device.
• ❌ Unsupported: World blend modes that a UA cannot achieve given the nature of the device’s display.

Device	opaque	additive	alpha-blend
Oculus, Vive, WinMR headsets	✅	❌	❌
HoloLens/Magic Leap headsets	❌	✅	❌
Magic window phone/tablet	✔️	✔️	✅
Vive Pro headset (video passthrough)	✅	✔️	✔️
@toji's Ultimate XR Device™ from #330 (comment)	✔️	✅	✔️

Two things to note in the table:

• For devices like Vive Pro that are naturally opaque VR devices, but could render alpha-blend AR as well, having the app request a world blend mode of alpha-blend during session creation may be a natural way to ask the UA to turn on the cameras for a world underlay.
• If an app is primarily operating on additive displays like HoloLens, it may have additive assets authored expecting such a display. Even when running on a video AR device like Vive Pro, the app may wish to have its additive RGB assets appear as intended - this is especially important if the app assumed it could do rendering optimizations that co-opt the alpha channel and leave garbage values there. Both opaque and additive mean that the UA will ignore the alpha channel, while alpha-blend means that the UA will use it. Having apps be explicit about whether they have given us valid RGBA pixels may be another benefit of requiring explicit opt-in to alpha-blend during session creation.

[thetuvix]

This should be explicit around how an opaque world blend mode treats the alpha channel: explicit ignoring it and treating all pixels as if alpha was 1.0.

[toji]

This doesn't sound unreasonable, and I'm going to change the PR text to reflect it because specific is better than non-specific in this case. I would like to verify the behavior of other systems, though. If Daydream, for example. treated this as an alpha-blend against black we'd have to clear the alpha channel to 1.0 prior to compositing in order to comply with this. That's a cost I'd rather not incur if we have a choice.

[thetuvix]

For native apps on HoloLens, alpha values are ignored in the headset display. Can you elaborate on how a UA might use alpha values to modulate the intensity of colors?

[toji]

I actually thought this WAS how HoloLens worked, with the colors coming through as sort of pre-multiplied by the alpha channel. I'll correct the PR.

[thetuvix]

You should mention video passthrough headsets like Vive Pro as well, to keep that scenario in the reader's mind.

[speigg]

I really like this. I would also like to consider if we can define these blend modes more generally, in other words decoupling these blend modes from physical reality, and allowing them to be defined as the blend mode used by the UA to composite a layer against the UA’s chosen underlying visual representation of reality (optical see-thru, video see-thru, virtual environment, etc). Basically I’m arguing we don’t assume the underlying representation of reality (that is relevant to the user experience) is always “physical reality”.

For example, the HoloLens is perfectly capable of rendering complete virtual environments such as tours of remote locations (and obviously these are displayed “additive” against physical reality), however if we consider how a UA might allow an XRLayer to be composited on top of such a virtual environment (assuming the UA wants to use that virtual environment as the underlying represention of reality), then the blend mode for that augmentation layer should (probably?) be “alpha-blended” against that virtual environment, even though the final composite is “additive” against physical reality.

Along this same vein of thought, I suggest the name XRRealityBlendMode (since we are effectively blending against a “reality layer” for composition).

[toji]

I don't object to the name XRRealityBlendMode, though if I'm being honest that actually suggests more strongly to me that the thing I'm blending against is the real world, whereas I could see interpreting World as either a physical or virtual one. Maybe something like XREnvironmentBlendMode splits the difference?

I do like the distinction that the thing it's blending against is somewhat arbitrary. I don't think any platforms enable this kind of functionality currently, but I can definitely see a future where you can run "AR" apps in something like the WinMR Cliff House.

[toji]

Working off the assumption that the "👍2" on the previous comment was due to a general approval of my naming suggestion, I've update the PR to now use the term "environment" rather than "world" (ie: XREnvironmentBlendMode)

[toji]

Seems like this PR has general support, with the primary additions being discussed (Alex's suggestion of requesting a specific blending mode) being something that can be layered on afterwards. As such I'd like to merge this soon, especially given that I won't be available for the next VR-centric call and would prefer to not have this outstanding for another 3 weeks.

Anyone still have unvoiced strong opinions about this feature or the naming?

[thetuvix]

I would also like to consider if we can define these blend modes more generally, in other words decoupling these blend modes from physical reality, and allowing them to be defined as the blend mode used by the UA to composite a layer against the UA’s chosen underlying visual representation of reality (optical see-thru, video see-thru, virtual environment, etc).

An issue there on an additive device is what happens if a layer is partially on top of another layer's virtual pixels and partially on top of optical reality. The UA can use any blend mode it likes when composing multiple virtual layers, but is limited by the capabilities of the display technology when optically composing over reality.

In that regard, I see this per-session attribute as reading out how the final flattened layer stack will ultimately behave given the nature of the display. When we later support multiple layers, we can then additionally give each layer its own blend mode, which separately lets apps decide how that layer should be flattened. The per-layer and per-session blend modes would combine to describe how composition takes place for that session.

If an additive device's UA or platform has a virtual underlay of some sort, some app pixels will directly land on clear display pixels and some will land on the underlay's pixels - in that case, the app's content still generally expects additive blending, and so it would seem least surprising to just consistently perform additive blending for all pixels, as if the underlay's pixels were part of the optical reality.

[toji]

In that regard, I see this per-session attribute as reading out how the final flattened layer stack will ultimately behave given the nature of the display. When we later support multiple layers, we can then additionally give each layer its own blend mode, which separately lets apps decide how that layer should be flattened.

+1. That's exactly how I see it playing out as well.

Given the lack of dissent I'm going to merge this now.