[mediaqueries-5] Introduce a media feature to detect an immersive or virtual environment

[rcabanier]

add feature to section https://drafts.csswg.org/mediaqueries-5/#mf-environment

We've heard from many authors that they want to change the look of web pages when they are displayed in an immersive display and there is currently no way to do this using CSS media queries.
For an example, see the section "Pull the Web Out of the Screen" on magicleap.com

#2719 is a rough proposal

[frivoal]

I've made a couple of low level comments on the pull request itself. Here are higher level observations:

• Can we start by listing what types of VR devices exist and how they present web pages? You suggest that we group them in 2, with an open issue on whether the second category is useful. You may be right, but I am not sufficiently familiar with all what's out there to be sure without further research that this is the right way to go about it.
• What kind of adjustments do people typically want to make? The example you give is a contrast inversion, and we are already discussing another media feature for that purpose. We have been moving away from media types and attempting to categorize devices, and instead focus on media features, to reveal relevant aspects of devices. In what ways are VR environments special? What aspect of that calls for differentiated styling?

[AmeliaBR]

What kind of adjustments do people typically want to make? ... In what ways are VR environments special?

I think these are important questions to answer. It might turn out that it is more useful to break this down into multiple independent features.

For example, the ability to include stereoscopic content is obviously one feature that comes with VR/AR headsets, but regular smartphones often also have a stereoscopic mode. An author might want to be able to detect whether stereoscopic mode is available, versus whether it is currently active.

Another aspect is whether the webview is overlaid on reality in a semi-transparent display. This can be used in AR environments, but it can also be used in car displays (projecting on the inside of the windshield) and "Google glass" style headsets which don't support stereoscopic effects in the display itself.

I'm sure people who are more familiar with VR/AR work could come up with other, specific features of these displays. We need to consider that the current common grouping of features into "VR" and "AR" won't necessarily persist in the future.

[astearns]

This seems like something that the Immersive Web Community Group may have some thoughts on. @NellWaliczek, @toji, or @dontcallmedom have media queries come up there?

[rcabanier]

For example, the ability to include stereoscopic content is obviously one feature that comes with VR/AR headsets, but regular smartphones often also have a stereoscopic mode. An author might want to be able to detect whether stereoscopic mode is available, versus whether it is currently active.

CSS doesn't have any features that apply to stereoscopic viewing so it's unclear why an author would like to detect this.
Maybe there's a need to detect it for script driven content but that will rely on other features such as WebXR

Another aspect is whether the webview is overlaid on reality in a semi-transparent display. This can be used in AR environments, but it can also be used in car displays (projecting on the inside of the windshield) and "Google glass" style headsets which don't support stereoscopic effects in the display itself.

Yes, I think that would be applicable. Maybe we should go for a transparent display feature instead of "reality"

[rcabanier]

Can we start by listing what types of VR devices exist and how they present web pages? You suggest that we group them in 2, with an open issue on whether the second category is useful. You may be right, but I am not sufficiently familiar with all what's out there to be sure without further research that this is the right way to go about it.

I am unsure as well which is why I put an issue on VR. Current VR/AR browsers don't seem to be doing anything special for regular layout.

What kind of adjustments do people typically want to make? The example you give is a contrast inversion, and we are already discussing another media feature for that purpose.

It's not just contrast inversion (although that does come close to the desired color scheme).
Typically, authors want to present information in a less dense fashion with bigger and lighter colored text.

In what ways are VR environments special?

We don't have a VR environment so I don't really know :-)

What aspect of that calls for differentiated styling?

For immersive, it's a lot like looking at your TV or a poster. You could walk closer and read tiny text but it's more comfortable to sit on your couch and scroll more.
So, it's a lot like looking at your TV except you can see right it when no light is added.

I think our concept video shows it pretty well, especially around the 1:25 mark: https://www.youtube.com/watch?v=GmdXJy_IdNw

[rcabanier]

This seems like something that the Immersive Web Community Group may have some thoughts on.

I think @AmeliaBR might be onto something. Our issue is more with the transparent nature of the display and not really with being immersive.

[blairmacintyre]

I made some comments over on the PR to answer some of these questions. (@astearns I'm part of the WebXR community, so while I don't speak for them, I'm coming from that viewpoint).

I like this media query. There are two kinds of things we need in the web to deal with XR devices:

• A way to know what kind of environment the page is being presented in
• A way to present 3D content from a "web page" (web app) in XR (AR, VR, etc)

The WebXR API is concerned with the later. Regardless of how the page itself is presented (e.g., on a 2D desktop, on a computer with a Vive/WindowsMR/Rift attached; in 3D in VR in WindowsMR's "Cliff House" home space; in 3D in AR on Hololens or Magic Leap, etc), WebXR is about telling the web page what displays are available and allowing them to draw 3D graphics with WebGL on that display in a "full 3D immersive" mode.

This media query, and (I think the motivation behind it) is concerned with the former: what is the sort of display environment in which we are rendering a web page (HTML content).

In the case of a display attached to a computer (e.g., the Vive), the web page is still 2D on a traditional display, but the VR display used by WebXR is 3D VR. So this media query would return something indicating a traditional 2D display.

I mention this because things like stereoscopic seem outside the scope of this, without much more dramatic changes to HTML/CSS: there is no way to cause the page to appear differently to the 2 eyes from the viewpoint of the web developer. That is something the browser itself would deal with (e.g., you could imagine the text or images being rendered slightly differently, not just in slightly different locations, to give the best combined appearance).

What the dev cares about, as highlighted by the original post, is much more mundane: what is the sort of display environment?

• traditional: desktops, phones, tablets. Content is being rendered on a flat screen, with the pixels being what you request
• VR: opaque 3D displays. The web page is rendered in a "rectangle" in a 3D world. The rectangle is opaque, and it's very likely that notions like pixels will no longer map to pixels on a display. So, while such a page may have a virtual size in CSS pixels, the relationship to actual rendered pixels is difficult (and may change continuously as the user moves relative to the page). The "place in the 3D world" of the page is probably not available to the developer (since it's controlled by the user and browser). Knowing that it's being presented in 3D could cause the developer to change the content presented, and knowing that text entry might be hard or that scrolling might be controlled by more awkward inputs than a mouse might change how they present.
• mixed/AR: video-mixed or optically transparent displays that render a web page in 3D like VR, but where the world around the page is uncontrolled, and where the user may be moving and need to see things in the real world. for video-mixed (graphics overlaid on video in the 3D renderer by the browser), the page itself is still opaque, but may want to do things based on context (same as VR, plus perhaps reduce distracting high-motion content, or change its content when not in focus to distract less). On optically-seethrough displays (like Magic Leap, Hololens, etc), the dev may also want to increase contrast because the "page" is not fully opaque, so the world may show through a bit. I have done things in the past where I used CSS shadows on text to put a faint white shadow up+left and a faint black shadow down+right, so that the text is legible on both light and dark backgrounds.

This media query is "the most basic" one and needed as a starting point to handle the big-picture adaptations across these 4 sorts of different environments: traditional, VR, video-mixed, optically-mixed.

Beyond that, there are other things we might want to add, but they start to tell pages about the world, and have privacy implications:

• a measure of the brightness of the room (so I can decide how bright/contrasty to make the content)
• a measure of the display-angle of the page relative to the viewer. Am I looking at it head-on or from an angle. My student Gheric Spieginer (@speigg) explored this in an environment where we could set the background of the page transparent and then control the style and content of elements based on simple queries like aspect ratios, etc. See this video, where the content of DOM elements is changed based on angles, how big they are on the screen, etc https://www.dropbox.com/s/ecppit80o71u49p/video.mov?dl=0

These both feel beyond the scope of this particular query, and it's less clear how they would work.

There are also ideas around "popping 3D objects out of the page into the world" (e.g., that the Meta CEO showed demos of in TED talks, for example). These would require more substantial changes to CSS or new HTML elements. But, you could imagine wanting to change tagging automatically based on a query (e.g., if you had a DOM element that somehow could pop out into the 3D world, you might want to enable it in VR/Mixed but not on a traditional 2D display).

[kearwood]

Perhaps the media query could inform content when the page is displayed in augmented reality with a display that has an additive brightness effect. In this case, "Additive" means that black is essentially transparent/invisible and white always has the most visible contrast. Pages that normally use a dark on light background will likely want to use a light on dark background in this case.

This may not be the ideal for VR immersive displays; however, which can render dark images over light backgrounds.

In both cases, content would likely use a variation of the formatting normally used for touch-optimized phone and tablet interfaces. Unlike phones and tablets; however, scrolling is not as ergonomic in AR and VR.

[rcabanier]

Perhaps the media query could inform content when the page is displayed in augmented reality with a display that has an additive brightness effect.

We shouldn't even mention augmented reality since it applies to head up displays as well.
I like you suggestion of environment-blending. (It could even be just "environment")

What do people think of:
environment: opaque | additive

[rcabanier]

@blairmacintyre I agree that there is a lot of work on the CSS/HTML front to make web pages that can take advantage of an immersive environment. A media query for 2D/VR/AR should be part of that work.

We're very interested in collaborating with other people on exploring this but I don't think there's a group that is working on this particular problem.

[frivoal]

given that #2719 has been merged, can this be closed?

[rcabanier]

Yes.

…

On Sun, Jul 1, 2018 at 8:10 AM Florian Rivoal ***@***.***> wrote: given that #2719 <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_csswg-2Ddrafts_pull_2719&d=DwMCaQ&c=0ia8zh_eZtQM1JEjWgVLZg&r=jahSgznxrAL5kPgsRvs7bhKUEd9M5X0d-NE2WJg7VT0&m=pzHu9hSX0ZbSkDw0qlLxFG9qTjEzN9t2yGWjWOMLycc&s=4UZ4l7Nw4LvxvLKJBJBhnqlcEzRH5ouAvUmb6vQluf8&e=> has been merged, can this be closed? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_csswg-2Ddrafts_issues_2723-23issuecomment-2D401612820&d=DwMCaQ&c=0ia8zh_eZtQM1JEjWgVLZg&r=jahSgznxrAL5kPgsRvs7bhKUEd9M5X0d-NE2WJg7VT0&m=pzHu9hSX0ZbSkDw0qlLxFG9qTjEzN9t2yGWjWOMLycc&s=S1ixLZrdssNFyhE5KNMGgUt12x3942RKb37Hlit_fak&e=>, or mute the thread <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AlHhyWJidzvOujTWvE-2DuX8Bg8v-5Fy7CHOks5uCOZ0gaJpZM4UUVMe&d=DwMCaQ&c=0ia8zh_eZtQM1JEjWgVLZg&r=jahSgznxrAL5kPgsRvs7bhKUEd9M5X0d-NE2WJg7VT0&m=pzHu9hSX0ZbSkDw0qlLxFG9qTjEzN9t2yGWjWOMLycc&s=lwXSFZsvmQb9RC6_ridqXBIRJBaa_smlLRxC3pLG--g&e=> .