hdrSupported - Screen.video or simply Screen?

[chcunningham]

In #118 we proposed putting this bool on Screen.video, alongside other properties that would describe the video "plane" on devices with different video vs graphics capabilities. In #133 we're reconsidering the resolution pieces, leaving just the bool. Lets make sure we're putting this bool in the right place. Questions:

1. Up until now the discussion of the 2 plane problem has been about resolution. @lhalliwell @jpiesing, can you confirm whether we should expect HDR support to also be an issue here (e.g. only supported in video plane?)

2. Where does the 2-plane problem live? Is it the Screen or is it within the UA software? Is "video" a property of Screens? Thought experiment: say you have an HDR display connected to a laptop running a hypothetical browser which supports HDR for both images and video. Now say you disconnect the laptop and instead plug in a Chrome Cast dongle, which only supports HDR for video plane (assuming answer no. 1 is "yes"). Note: the Screen didn't change.

On the one hand, I think Screen.video.hdrSupported would function fine surface any video-vs-graphics plane limitations. On the other hand, it isn't conceptually very clean. Please discuss :).

PS - apologies for not raising in the call earlier today. my questions here are brand new since this afternoon.

[jpiesing]

1. Up until now the discussion of the 2 plane problem has been about resolution. @lhalliwell @jpiesing, can you confirm whether we should expect HDR support to also be an issue here (e.g. only supported in video plane?)

@chcunningham There are probably exceptions but, as far as I'm aware, the graphics system/plane will still operate in sRGB (i.e. SDR) even when video is HDR. When graphics and video are combined, graphics may be converted to HDR or both graphics and HDR converted to something that fits the panel.

[lhalliwell]

@chcunningham There are probably exceptions but, as far as I'm aware, the graphics system/plane will still operate in sRGB (i.e. SDR) even when video is HDR. When graphics and video are combined, graphics may be converted to HDR or both graphics and HDR converted to something that fits the panel.

Right. As long as there are no HDR image formats or CSS support, it's presumably wasteful for a browser to render at anything besides SDR, and no need for a page to query for this.

I didn't read the full proposal but for some existing devices (like CC Ultra), a boolean "hdr supported" isn't enough. Such devices can't do any significant colour space conversions, so the video's colour metadata is passed through to the TV. This in turn means the page needs to be able to detect what the TV supports. From what I've seen, newer hardware seems to have better ability to convert between colour spaces, so this problem may go away over time.

[vi-dot-cpp]

@lhalliwell
I didn't read the full proposal but for some existing devices (like CC Ultra), a boolean "hdr supported" isn't enough. Such devices can't do any significant colour space conversions, so the video's colour metadata is passed through to the TV.

@chcunningham has addressed this concern in this comment on #118 by noting that MC.decodingInfo will be used to discern the TV's capabilities.

@jpiesing
There are probably exceptions but, as far as I'm aware, the graphics system/plane will still operate in sRGB (i.e. SDR) even when video is HDR. When graphics and video are combined, graphics may be converted to HDR or both graphics and HDR converted to something that fits the panel.

To make sure I understand, are you saying there are differences with regards to hdrSupport between a TV's image plane vs. its video plane?

[jpiesing]

To make sure I understand, are you saying there are differences with regards to hdrSupport between a TV's image plane vs. its video plane?

@vi-dot-cpp Yes.

[vi-dot-cpp]

@chcunningham
Where does the 2-plane problem live? Is it the Screen or is it within the UA software? Is "video" a property of Screens? Thought experiment: say you have an HDR display connected to a laptop running a hypothetical browser which supports HDR for both images and video. Now say you disconnect the laptop and instead plug in a Chrome Cast dongle, which only supports HDR for video plane (assuming answer no. 1 is "yes"). Note: the Screen didn't change.

I think your thought experiment is pointing out that HDR support for graphics changed even though the TV remained constant -- suggesting that the graphics v. video aspect aspect of hdrSupported is not tied to Screen. Not knowing much about cast devices, may I ask whether this is, or will/can be, the case?

Alternatively, could a TV only support HDR for either video or graphics @jpiesing @lhalliwell? In which case, there would be a need for Screen to disambiguate hdrSupported.

[jpiesing]

What does it mean to support HDR for graphics when web graphics are sRGB? Isn't that always SDR by definition or am I confused? If sRGB is SDR by definition then HDR support for graphics would be a meaningless concept.

[chcunningham]

I think your thought experiment is pointing out that HDR support for graphics changed even though the TV remained constant -- suggesting that the graphics v. video aspect aspect of hdrSupported is not tied to Screen.

Agree.

Not knowing much about cast devices, may I ask whether this is, or will/can be, the case?

In retrospect, I could have simplified this a bit to make the same point. Take any non-ultra cast (no HDR for either video or graphics), any laptop w/ UA supporting HDR, and any HDR screen. Switch between laptop and cast - only laptop can render HDR while screen remains the same.

What does it mean to support HDR for graphics when web graphics are sRGB?

The HDR plan for graphics isn't settled yet. Agree its historically sRGB. Discussion is active (Color on the Web CG discussed at tpac 2019 - having trouble finding the minutes).

With decodingInfo(), we are offering an API to say what kinds of videos we can render, and we're facing this snag about doing HDR video, but not HDR graphics. Maybe its fair to expect whatever HDR graphics future shapes up will include some similar APIs to detect support for rendering HDR graphics (i.e. that it isn't a given simply because Screen.hdrSupported == true)? Then cast/smart tvs could reply false for graphics in some TBD manner.

@hober - can you help connect us with some expertise on Screen spec? This feels like a TAG type API question.

[chrisn]

The HDR plan for graphics isn't settled yet. Agree its historically sRGB. Discussion is active (Color on the Web CG discussed at tpac 2019 - having trouble finding the minutes).

The minutes from the Color on the Web CG meeting at TPAC are here. See also this email from @svgeesus. There's a draft report that surveys HDR and wide gamut colour support across various web platform APIs. (I drafted the report a while ago, so the section on Media Capabilities lags the current discussion here.)

[vi-dot-cpp]

@chcunningham
In retrospect, I could have simplified this a bit to make the same point. Take any non-ultra cast (no HDR for either video or graphics), any laptop w/ UA supporting HDR, and any HDR screen. Switch between laptop and cast - only laptop can render HDR while screen remains the same.

Why is the onus not on MC.decodingInfo() to report the HDR capabilities of the UA, which is the non-ultra cast device in this example. If the example were an ultra cast and any laptop w/o UA HDR support, MC.decodingInfo(), as opposed to hdrSupported boolean, would be used to discern HDR support of the laptop's UA.

Moreover, is the hdrSupported boolean useful for casting scenarios. Per your comment on #118, hdrSupported is redundant for cast, because MC.decodingInfo() checks the screen's properties.

[gregwfreedman]

i think of screen.video.hdrSupported as not simply a property of the panel, but a property of the panel as seen by the UA. in the case of Ultra Chromecast, screen.video.hdrSupported is obviously true. in the case of non-Ultra Chromecast, decoding.info might return true for HDR if it can decode and render as SDR. however, screen.video.hdrSupported would be false, because video would not be rendered as HDR (i.e. for video the UA sees the panel as SDR).

[chcunningham]

Why is the onus not on MC.decodingInfo() to report the HDR capabilities of the UA, which is the non-ultra cast device in this example. If the example were an ultra cast and any laptop w/o UA HDR support, MC.decodingInfo(), as opposed to hdrSupported boolean, would be used to discern HDR support of the laptop's UA.

Agree 100%. Sorry, I made a mess of the example again. The non-ultra cast (which is funcitoning as the UA) should signal lack of HDR support via decodingInfo(). What I meant to demonstrate is only that the Screen is HDR in both cases and could return hdrSupported = true in both cases. The MediaCapabilities / Screen pattern could then serve as a model for future graphics APIs (i.e. first check Screen.hdrSupported to see if its even worth asking more complicated questions, then proceed to ask about the hdr graphics rendering support via some TBD API just as you would video rendering with MediaCapabilities).

decoding.info might return true for HDR if it can decode and render as SDR. however, screen.video.hdrSupported would be false, because video would not be rendered as HDR (i.e. for video the UA sees the panel as SDR).

We bounced around on this a bit in the HDR centi-bug, but we ~settled on a wording that would actually cause decodingInfo() to return false in this case. The text part is here. In particular I'm looking at this bit...

The UA needs to be able to cause the output device to render the appropriate color, or something appropriately close enough. Valid inputs are defined by ColorGamut.

...and similar for eotf.

At least for video, I think this means we could describe the full matrix of UA + Screen support for HDR, but the hole for the graphics edge case remains (devices that support only HDR video).

I'm reading the stuff from @chrisn now (catching up on state of graphics world). Will make a follow up comment when I've digested that.

[gregwfreedman]

decodingInfo answers whether or not the UA can decode and render an HDR stream. screen.video.hdrSupported answer whether a site should use an HDR stream over an SDR stream.

if the UA can successfully decode an HDR stream and tone map it onto an SDR display then it should return true. if a site only has one stream and it's HDR, then it can use decodingInfo to know that the UA will do a reasonably good job presenting it.

if a site has both SDR and HDR streams (and decodingInfo returns true for both) it can use screen.video.hdrSupported to know which will provide the better user experience. if screen.video.hdrSupported is false, then the site would know to use the SDR stream since it was manually trimmed/graded for an SDR display. obviously, this would be better than using the HDR stream and letting the UA do the tone mapping to an SDR display.

The UA needs to be able to cause the output device to render the appropriate color, or something appropriately close enough. Valid inputs are defined by ColorGamut.

perhaps we should change this wording...

[gregwhitworth]

So to sum up, and please correct me if I'm wrong @gregwfreedman you're wanting decodingInfo can decode and render HDR content, even if that HDR content ultimately ends up being rendered in SDR; corrrect? In that case possibly adjust the wording to:

The UA needs to be able to cause the output device to render the appropriate color, or something appropriately close enough; possibly through down-sampling. Valid inputs are defined by ColorGamut.

We could also provide an example to show that a UA can convert from HDR content to SDR (eg: Rec2020 -> sRGB).

[chcunningham]

I'm reading the stuff from @chrisn now (catching up on state of graphics world). Will make a follow up comment when I've digested that.

I'm still reading... will post more soon (hoping suggests a direction on the core question of Screen.video vs simply Screen).

i think of screen.video.hdrSupported as not simply a property of the panel, but a property of the panel as seen by the UA

This resonates with me, and I think its consistent with the other things on Screen (e.g. our earlier discussion about Screen.width not being physical width, but width as seen by the UA). Do you think it motivates either: Screen.video.hdrSupported vs Screen.hdrSupported? To me it seems workable with either one.

decodingInfo answers whether or not the UA can decode and render an HDR stream. screen.video.hdrSupported answer whether a site should use an HDR stream over an SDR stream.

Similar to above, I think this interpretation is compatible with Screen.hdrSupported. Do you agree?

The UA needs to be able to cause the output device to render the appropriate color, or something appropriately close enough. Valid inputs are defined by ColorGamut.

perhaps we should change this wording...

; possibly through down-sampling.

We could also provide an example to show that a UA can convert from HDR content to SDR (eg: Rec2020 -> sRGB).

I support re-working wording. The original wording probably did not intend that decodingInfo(HDR) only returns true when an HDR display ("output device") is attached. I think the output device is only mentioned as an attempt to clarify that full support should be there if one were to be attached (i.e. the UA is doing its part prior to handing off to the Screen). As is, I think the wording runs too close to the former meaning.

Buuut, I think we should also be careful to clarify that, when and HDR display actually is attached, tone-mapping to SDR does not meet the bar for "supported=true".

Also, a thoughts on tone-mapping: you can meet the criteria above (HDR looks right on an HDR screen) while also not having tone-mapping. IIUC this is the situation with Chrome today. HDR vp9 or av1 will be accurate on HDR displays, but looks washed out on SDR displays. We may eventually implement tone-mapping but it hasn't been a priority because sites that send HDR so far have their own SDR grading they prefer to fall back on. I'm interested to hear more discussion on whether we think tone-mapping should be spec mandated in light of the above.

A little aside on this bit:

in the case of non-Ultra Chromecast, decoding.info might return true for HDR if it can decode and render as SDR. however, screen.video.hdrSupported would be false, because video would not be rendered as HDR (i.e. for video the UA sees the panel as SDR).

For this particular example the non-ultra chromecast isn't able to do any tone mapping. It doesn't understand the HDR eotfs, so any decodingInfo(HDR) call would return false. But, its also true UA doesn't view the Screen as HDR (for video or otherwise), the Screen property may as well return false too.

[gregwhitworth]

Buuut, I think we should also be careful to clarify that, when and HDR display actually is attached, tone-mapping to SDR does not meet the bar for "supported=true".

Before we try to adjust the text again, let me see if I can outline what I would expect to be returned with an HDR display and a SDR display.

	ColorGamut	HDR Display	SDR Display	Tone-map HDR to SDR
decodingInfo	rec2020	true	true	true
screen.video.hdrSupported	rec2020	true	false	false

Buuut, I think we should also be careful to clarify that, when and HDR display actually is attached, tone-mapping to SDR does not meet the bar for "supported=true".

Is this something that happens? EG: I have HDR content -> HDR supported UA -> HDR display and it gets tone-mapped to SDR?

If that is the case then somewhere I would expect this to actually return false because somewhere along the way something isn't technically supported.

[chcunningham]

Re: core question, Screen.video or simply Screen...

The Screen.video shape is entirely about solving two plane TV problem. I've been trying to consider this from all angles, reading up on the HDR graphics proposals and discussing offline with @jpiesing (HbbTV) and @lhalliwell (Cast).

My conclusions:

1. Surfacing the two planes on the web platform (Screen.video) is at odds with existing platform APIs.
2. Forcing TVs to use existing APIs (Screen, device pixel ratio, ...) may come at a cost. I think the cost is manageable and the workarounds are viable.
3. TV manufacturers are projected to continue building TVs this way, with graphics lagging video, as a cost / use-case-priorities calculation.

Let's get into it...

1. Surfacing the two planes on the web platform (Screen.video) is at odds with existing platform APIs.

Even before MC, having 2 planes breaks video's interaction with existing web apis like CSS and WebGL. Luke sums it up well here #133 (comment).

This whole setup is obviously a bit yucky and not fully featured, i.e. there are things expressible on the web that are impossible to display this way (examples: rotating video at arbitrary angles, applying transparency to video with other content visible beneath, using a video as a WebGL texture).

Adding Screen.video doesn't address these breaks, but it does give TV makers cover to continue down this path. Conversely, Screen.hdrSupported is consistent with existing web APIs, which do not distinguish between "video" and "graphics" - its all part of the same Document and CSS object model. We can already find use cases for Screen.hdrSupported emerging from that graphics proposals. For ex, see here for a Canvas HDR use case.

2. Forcing TVs to use existing APIs (Screen, device pixel ratio, ...) may come at a cost. I think the cost is manageable and the workarounds are viable.

In both this issue (hdr) and #133 (height, width), forcing TVs to use existing APIs (Screen without .video, or device pixel ratio media queries) means they must lie about one of the planes. Ex: reporting Screen.hdrSupported=true or reporting a device-pixel-ratio of the output display (rather than graphics raster) would both be lying about graphics to benefit video.

I don't think this lie will be much practical harm for HDR images, CSS, or Canvas. Supporting any of these involves work at all levels of the stack: decoding, compositing, javascript interfaces etc.. Given the graphics plane limitations, TVs are unlikely to implement even basic support for the above. Its easy for apps to detect that a new HDR Canvas API is not implemented, or that a given image format is not supported (ex for webp here).

The cost of lying about device-pixel-ratio is a little less speculative. Web authors already use this mechanism to choose higher resolution images (retina displays, phones). Choosing a 4K image that's actually downsampled to a 1080p raster wastes bandwidth and @jpeising tells me this could even OOM a low end TV (more so for 8K). The likelihood of this will depend on web content. 4K images are pretty rare today. If TV apps start using them, perhaps it's fair to assume this will be preceded by (or precipitate) a change in TV manufacturing to add support.

I don't think the costs above outweigh the downsides of standardizing 2-planes in the web platform (see point 1). In worst-case scenarios, TV manufacturers can still resort to private APIs. This has its own fragmentation cost, but I think that aligns incentives correctly. Compared to a web standard working group, the people who make TV apps truly have the ear of TV manufacturers. If they resent the fragmentation of private APIs (or the breakage of video / graphics interaction in existing APIs), they are a powerful voice for change. This could mean coalescing around a common private API (a TV standard), or a push to have TVs put graphics and video in a unified plane. Which segues nicely into my next point...

3. TV manufacturers are projected to continue building TVs this way, with graphics lagging video, as a cost / use-case-priorities calculation.

@jpiesing tells me this isn't going away in a hurry. Video, not graphics, is the primary TV use case. TV manufacturers make choices accordingly, keeping costs low.

[chcunningham]

@gregwhitworth I think your table looks correct.

Is this something that happens? EG: I have HDR content -> HDR supported UA -> HDR display and it gets tone-mapped to SDR?

I agree, this would probably be a bug (not aware of any instance). On second thought I'm open to wording that doesn't explicitly mention this is a bug.

Feel free to iterate on the wording here without a heavy PR update.

[jpiesing]

It won't surprise anyone that I don't agree with @chcunningham about Screen.hdrSupported vs Screen.video.hdrSupported. Why should a new API force the major category of media consumption devices 1, 2 to lie about their device capabilities with a media related API?

It's correct to say this is a cost issue but it's a fairly extreme one. Increasing the graphics resolution from 1280x720 to 1920x1080, 4K or indeed 8K requires both extra RAM and a more powerful GPU if the same graphics performance is to be maintained. TV sets are a very price-competitive market and under continual cost pressure.

If a device with 4K HDR video but 2K SDR graphics has to pick one to return then I agree that it will return 4K HDR so that content apps can offer the correct media for the device. It will mean that, in a few years time, apps that want to do 4K images or HDR images won't be able to rely on what the APIs return. Do we really want to start building up this problem for the future?

[mwatson2]

+1 to @jpiesing's comment. The Web Platform should accommodate fundamental hardware limitations of the target hardware plaforms (unless we are saying that the web is only for desktop / laptop / mobile hardware class devices and all other devices should level up to that).

The issue also exists, though not in exactly the same way, for any HDR laptop today and will continue to exist until CSS adds HDR and all browsers supporting HDR video implement that.

There are other ways that TV browsers don't support all the capabilities of the web platform, some of which are software limitations which we could expect people to address and some of which are fundamental hardware limitations. We should address the latter.

[chcunningham]

IMO new hardware classes are always welcome on the web, but they shouldn't be able to redefine the DOM in ways that break existing APIs (add device pixel ratio to that list).

Again, the cost of maintaining video/graphics continuity seems low. If TV graphics being 1080p and SDR is a fundamental and unavoidable limitation, then TV html apps makers will never attempt to use 4K and/or HDR graphics. On the other hand, if TVs in the future support HDR and high resolution graphics, apps will of course use them, but this indicates the platform hw limitations have changed and we'll be better off not having monkey patched 2019's TV platform limitations into web apis.

The issue also exists, though not in exactly the same way, for any HDR laptop today and will continue to exist until CSS adds HDR and all browsers supporting HDR video implement that.

This is fundamentally different. There are open questions about what it will mean to composite HDR video alongside SDR graphics, and what HDR graphics APIs will ultimately look like. But none of those have motivated a separate "video" API on Screen, and HDR video on laptops is not fundamentally at odds with other APIs on the web platform. Screen.video is proposed as a solution only for the 2-plane problem on TVs.

[mwatson2]

@chcunningham The problem with todays HDR laptops is similar, because the CSS APIs indicate only SDR support (because there is no way defined for them to do otherwise), yet HDR video is supported. And those same devices can be in a configuration where HDR decode and rendering is supported but the screen is not HDR-capable (HDMI-connected SDR screen). On such a device / browser - which will be in the field for years to come - how are we to decide whether to deliver SDR or HDR video ?

Regarding TVs and other devices used in large part for TV viewing (set-top-boxes, HDMI sticks etc.) for a long time there will be a diversity of capability in the field. Many such devices have long lifetimes (7+ years) and devices with different video / graphics capabilities will continue to be shipped for many years to come. Your characterization of the choice between TVs having disparate video / graphics and having full HDR support is a false dichotomy - both will exist for a considerable time.

There will (eventually) be high-end TVs., STBs, streaming sticks with HDR graphics support and services like Netflix will want to make use of that capability as we do today on some games consoles. But the devices with separate video / graphics will remain in the field and continue to be shipped too.

Regarding:

IMO new hardware classes are always welcome on the web, but they shouldn't be able to redefine the DOM in ways that break existing APIs (add device pixel ratio to that list).

There is no need to "break" existing APIs, but there is a need to recognize that existing capability assumptions in the web platform could have been based on particular hardware classes and if you are serious about enabling use of the platform on other hardware classes you sometimes need to provide new capability detection.

Here, though, we are not saying that a capability previously always available will newly not be available in some cases and so now needs to be detected when previously it could be assume. We are saying that a new capability (HDR), not previously available in the web platform, is sometimes available only in a specific context (video). It is not a new thing for the platform that a capability that in principle has generic application (in this case to video and graphics) might in practice only be available in a specific context - this happens all the time.

[jpiesing]

IMO new hardware classes are always welcome on the web, but they shouldn't be able to redefine the DOM in ways that break existing APIs (add device pixel ratio to that list).

IMHO requiring implementations to claim to support 4K HDR graphics when they in fact only support 2K SDR graphics is breaking existing APIs far more than what we're talking about here. What we're talking about here could be achieved with one property!

Again, the cost of maintaining video/graphics continuity seems low. If TV graphics being 1080p and SDR is a fundamental and unavoidable limitation, then TV html apps makers will never attempt to use 4K and/or HDR graphics.

While there are certainly apps that are specific for TVs, there are also plenty of app creators who want a single HTML app to work as widely as possible including TVs.

On the other hand, if TVs in the future support HDR and high resolution graphics, apps will of course use them,

I'm sure graphics will get better in future TVs but I'm equally sure that video will get better as well (e.g. 8K). I would not expect graphics to catch video - in this type of device, media consumption is more important than graphics and that's not going to change.

but this indicates the platform hw limitations have changed and we'll be better off not having monkey patched 2019's TV platform limitations into web apis.

I'm uncomfortable with this characterisation of the problem when IMHO what we're really talking about here is one property.

[chcunningham]

I'm uncomfortable with this characterisation of the problem when IMHO what we're really talking about here is one property.

This overlooks most of the issues I've raised. The one Screen.video.hrdSupported property enshrines separate video/graphics planes which is not backward compatible with a number of existing APIs

• device pixel ratio (resolution media query)
• Screen.width and height
• Various styling of video not supported by a hole punch
• Video as a texture in WebGL / Canvas

IMHO requiring implementations to claim to support 4K HDR graphics when they in fact only support 2K SDR graphics is breaking existing APIs far more than what we're talking about here. What we're talking about here could be achieved with one property!

The reason this doesn't convince me is that no one is doing 4K or HDR graphics atm and when HDR graphics become available, we can anticipate idiomatic ways to feature detect their support (examples above). This leaves only the resolution issue. Respectfully, I don't think that sits heavier on the scale than the breaks I list above.

[chcunningham]

In reply to @mwatson2

... On such a device / browser - which will be in the field for years to come - how are we to decide whether to deliver SDR or HDR video ?

Screen.hdrSupported can tell you about the output device, which serves both video and graphics use cases in a future compatible manner. For video, sites still must call MediaCapabilities decodingInfo() determine if their format is supported decoding/rendering is supported. A similar pattern will exist for graphics. Images already has this mechanism, and I expect the draft APIs to follow suite. In the interim, it is easy for sites to detect that the HDR graphics APIs are simply unimplemented.

Here, though, we are not saying that a capability previously always available will newly not be available in some cases and so now needs to be detected when previously it could be assume. We are saying that a new capability (HDR), not previously available in the web platform, is sometimes available only in a specific context (video). It is not a new thing for the platform that a capability that in principle has generic application (in this case to video and graphics) might in practice only be available in a specific context - this happens all the time.

HDR is a new thing, yes. But this argument overlooks the breaks I listed above. For example, the combination of width / height and device pixel ratio is actively assumed to describe video just like any other element of the DOM. You are now saying that it needs to be detected by some other mechanism in a non backward compatible fashion. This is a "break".

[chcunningham]

There will (eventually) be high-end TVs., STBs, streaming sticks with HDR graphics support and services like Netflix will want to make use of that capability as we do today on some games consoles. But the devices with separate video / graphics will remain in the field and continue to be shipped too.

I mean to acknowledge that these devices will linger (this is point 3 in my longer post from yesterday). Future hardware gives me some comfort, but my argument does not hinge on there being a 100% unified video/graphics TV future. Please see my last paragraph in the reply to Jon above as to why I still don't find it a strong enough reason to pursue Screen.video.

[mwatson2]

Screen.hdrSupported can tell you about the output device, which serves both video and graphics use cases in a future compatible manner.

The problem with this is that CSS will just have the same problem in reverse:

• First, Screen.hdrSupported will deployed and used to detect that it's worth delivering HDR video to a device
• Later, CSS will define support for HDR colors. This will be detectable, of course, but this is more like our decodingInfo. There can still exist scenarios where tone mapping down to SDR is supported and the graphics plane is SDR but the video is HDR. Screen.hdrSupported will be true in this case for backwards compatibility with the point above. So people will be arguing for Screen.graphics.hdrSupported ...

The one Screen.video.hrdSupported property enshrines separate video/graphics planes which is not backward compatible with a number of existing APIs

• device pixel ratio (resolution media query)
• Screen.width and height
• Various styling of video not supported by a hole punch
• Video as a texture in WebGL / Canvas

Separate video / graphics support is not something we're introducing ('enshrines') - it exists as a fundamental property of an important class of devices. Unless you are saying that the W3C doesn't consider those devices as first class citizens in the web platform ecoystem ? But then why do we had the Web & TV Interest Group, CTA WAVE etc. and all the other things over the last 8 years that suggest otherwise ?

Some restrictions are easier to deal with than others. If I do have a site that relies on advanced styling / rotation / video-as-a-texture etc. then I am not surprised when that doesn't work optimally on a TV. I would not design a site that way if I expected a significant number of users on TV platforms. HDR vs SDR, HD vs 4K, by contrast, is just a matter of picking the right stream.

I agree that there could be use-cases for capability-detecting those other things, but I don't see anyone arguing for that kind of capability detection, so I think we can assume those use-cases are theoretical only. We could reasonably ask CTA WAVE about that.

Regarding resolution (or more specifically devicePixelRatio, since width and height in CSS pixels don't change based on resolution - CSS pixels have a physical size): I think we either need video.width and video.height in device pixels as originally proposed or you could have video.devicePixelRatio. Requiring the existing devicePixelRatio to reflect the higher video pixel density will result in sites pulling down high resolution images unnecessarily.

[jernoble]

@jpiesing said:

I'm uncomfortable with this characterisation of the problem when IMHO what we're really talking about here is one property.

Well, this is what I believe is the heart of the issue: this is not about just one property. Even if this narrow property was defined in terms of the "video plane" (for devices with separate video and graphics plane), pages would still be unable to make a functional media player without out-of-band information, due to all the other inconsistencies between the two planes. So that would require another property, and another, ad nauseam, until we end up reproducing much of the CSS OM in terms of the "video plane".

So, here's one proposed resolution to this issue:

1. This spec should make a normative reference to the CSS Media Queries specification.
2. This property should be moved from a DOM property on window.screen to a "media feature".

This has three benefits:

• It automatically gives pages the ability to detect when "hdrSupported" changes in script, without having to specify an addition of events to window.screen.
• It allows some pages to add HDR support in markup, rather than requiring all pages to drop to javascript.
• "Media features" are scoped to "media types", which leads to item 3 in this proposed resolution.

3. TV manufacturers which have separate characteristics for their "graphics plane" and "video plane" should signal such through a separate "media type".

This could be an existing, but unspecified, "media type" such as "tv", or a new "media type" could be specified. This media type would scope all feature queries, including ones for display resolution, color gamut, and this new "hdrSupported" feature. Pages could then query for, e.g., @tv { (hdrSupported) and (resolution >= 2dppx) } in addition to @screen {...} to get an accurate understanding about the underlying characteristics of the system.

This third step should happen in the CSS WG. In talking with @hober offline, the best way to get traction inside the CSS WG would be to join the CSS WG. Are the relevant TV UA representatives members of the CSS WG?

[jernoble]

@mwatson2 said:

I think we either need video.width and video.height in device pixels as originally proposed or you could have video.devicePixelRatio. Requiring the existing devicePixelRatio to reflect the higher video pixel density will result in sites pulling down high resolution images unnecessarily.

No, we should not do either. Those APIs return values in CSS pixels; if devices are giving incorrect definitions of what a CSS pixel means, that should be fixed in CSS, not in the Media WG.

[tidoust]

@chcunningham said

The major con for is only that media-types are being deprecated (what luck). Still, I'm not ready to declare it DOA until they weigh in. It may at least be the least of 3 evils ;).

Regarding media types, it's certainly fine to check with the CSS WG. Let me try to clarify some of the reasons for the deprecation because this has been discussed years ago already for handheld when smartphones started to appear.

A media type encapsulates a set of properties that appears common to target devices at a given set of time, some of them being implicitly defined. For handheld, this was something like small screen, limited bandwidth, low resolution, no support for a number of CSS features such as animations, etc. Typically, authors happily created trimmed down stylesheets for handheld devices that created good user experiences for small screens, and full featured screen stylesheets for larger screens.

Inevitably, a new product arises that has some of the same properties/constraints, but also new features that go beyond the understood definition of the media type, for instance better resolution (even though the screen may be only marginally larger) or much better support for CSS technologies. If that new device continues to apply the previous media type, new features won't be used and user experience will be sub-optimal, e.g. because very low-resolution images will be used. If the new device switches to the broader screen media type, new features get used, but that breaks some of the author's (not necessarily correct) assumptions (e.g. that screen is a large screen).

In the handheld case, the new device was the iPhone (and all smartphones since then), who decided to claim that they were screen devices, and not handheld devices, because they could handle the full-featured versions of the stylesheets and higher resolution images. That both enabled better experience down the road and broke content that authors had carefully crafted for phones using handheld initially.

In the end, the problem is that media types are too coarsely defined and not stable enough over time because they embed multiple dimensions (some of them being implicit assumptions). That essentially explains the move to media features, which allow authors to be explicit about the dimensions they target, and craft more responsive layouts.

I think the same "devices evolve over time" applies to TV by the way. For instance, several years ago, the CSS 2.1 spec described the tv media type as "low resolution". And yet we are today talking about 4K/HDR support, which seems more advanced in TVs than in other types of devices.

For an example of statement that cautioned about the use of media types during the "transition period" to media queries, you can have a look at the Use client-side detection capability detection where necessary best practice in the Mobile Web Application Best Practices recommendation, published in 2010:

"CSS Media Types allow different stylesheets to be associated with different media types (e.g. print, screen, mobile) and are traditionally used to repurpose content for these destinations. Since the list of recognized media types is limited, however, and devices are notoriously idiosyncratic in their interpretation of types, it is in general not a helpful technology in this context."

[chcunningham]

Really insightful! To make sure I follow, here's my guess at what a media-feature solution would have looked like for the original pre-iphone era animation challenges.

@media screen and (animations: 0) { 
... // different styling where animations not possible
}

Sadly, this seems like a simpler class of problem than the 2-plane situation. In our case, the feature-based solutions I see simply map to the Screen{.video} options (with their same pros and cons). That is, we can either not mention video (feature: hdrSupported = Screen.hdrSupported) or we can mention it (feature: hdrVideoSupported = Screen.video.hdrSupported). Same goes for the devicePixelRatio. LMK if there's a silver bullet lurking in the MQ syntax.

[tidoust]

LMK if there's a silver bullet lurking in the MQ syntax.

I don't see any for now.

Also, if I understand @jernoble's proposal in #135 (comment) correctly, I'm not sure that reintroducing a tv media type would actually work as suggested. In particular, I don't think that "Media features are scoped to media types". At least until now, media features have been defined regardless of media types.

For instance, the definition of resolution is the same for all media types. In @media tv and (resolution >= 2dppx), the term resolution would still describe the "resolution of the output device", not the "resolution of the output device for video". Otherwise, @media all and (resolution >= 2ddpx) would have a different meaning depending on the actual device, which seems wrong.

[chcunningham]

In particular, I don't think that "Media features are scoped to media types". At least until now, media features have been defined regardless of media types.

I think @jernoble meant the same features would be defined in all scopes, but the values for those features would differ by scope. Ex: resolution in the scope of @media tv would query the dpi of the video plane, while resolution for @media screen would query the dpi of the graphics plane.

[chcunningham]

Discussed a little more at FOMs. @mwatson2 @jernoble @hober (and many others) present. We all agree the media-type solution is very appealing. @hober, also part of the CSS working group, didn't see the past deprecation as necessarily at odds with this new usage.

For the TV video plane, we discussed using @media video instead of tv, which has it has an existing (deprecated) meaning that doesn't match what we're now hoping to describe.

Going back to an earlier comment...

but the definition of media types as mutually exclusive makes it difficult to use them in a reasonable manner

Our proposal would require that video and screen types not be exclusive. TV's with two planes would need to match both 'screen' and 'video' media types for sites to query the features* of the graphics and video planes respectively. I'm hopeful that the different semantics here again suggest that we shouldn't dismiss the idea along the lines of the planned deprecation.

*I think the following media features could be expected to return different values under the video type on a 2-plane TV: color, color-gamut, color-index, device-aspect-ratio, resolution, height, width

[gregwhitworth]

The more that I've thought about this the more I'm curious if we need a new way to query the graphics plane at all. As I outlined above, every graphic scenario that I can think of already has a way in which to handle it. So the only thing we need are queries specific to video for HDR & Resolution.

I noticed that the <source> element within <video> can't harness the srcset nor the sizes attributes (although I didn't test this). This would give you the bifurcation you're wanting in a declarative manner which can be set on the media element via script. This would then be querying the video plane within TVs for

So through mime, srcset & sizes also being supported for <video> and bifurcating in the code where necessary (eg: TVs would only query the video plane for the <video> element) you should be able to accomplish any query you want. Does anyone see any holes in this? @mwatson2 @jernoble @hober @chcunningham @vi-dot-cpp

[mwatson2]

@gregwhitworth I’m not sure I follow the proposal. Could you provide example code that would decide whether to fetch the SDR or HDR version of the content - assuming media capabilities indicates support for both ?

[gregwhitworth]

@mwatson2 sure thing. I'm not saying the gaps are all filled in, but I am thinking it may make sense to extend the tech where it is rather than a single bool. Here would be images for example since
should work today:

<picture>
	<source type="image/vnd.ms-photo"
		sizes="100vw"
		srcset="imgs/hero-small.jxr 480w,
				imgs/hero-med.jxr 960w,
				imgs/hero-large.jxr 1920w,
				imgs/hero-xlarge.jxr 2500w">
         <source type="image/avif"
		sizes="100vw"
		srcset="imgs/hero-small.jxr 480w,
				imgs/hero-med.jxr 960w,
				imgs/hero-large.jxr 1920w,
				imgs/hero-xlarge.jxr 2500w">
	<img class="hero"
		src="imgs/hero-xlarge.jpg"
		sizes="100vw"
		srcset="imgs/hero-small.jpg 480w,
				imgs/hero-med.jpg 960w,
				imgs/hero-large.jpg 1920w"
	 alt="A portrait painting of a woman on a couch">
</picture>

The browser will then select the image that provides the best user experience (this is UA defined to allow network, battery, user settings etc to be taken into account). This includes both resolution, expected layout dimensions and mimetype support for optimal selection (just in case someone hasn't used these, I dig into them a bit deeper here)

I haven't tested video yet, but the spec states that srcset & sizes:

srcset — Images to use in different situations (e.g., high-resolution displays, small monitors, etc.)
sizes — Image sizes for different page layouts

So for 4K video this may need to be added to UAs and the spec to ensure 4K video is served. The only area with regards to image or video that this may fall down is if HDR can be encoded into a file format that can also contain SDR, is this common or even possible?

Let me know your thoughts.

[mwatson2]

f HDR can be encoded into a file format that can also contain SDR, is this common or even possible?

I would expect any image format that supports HDR to also support SDR, just as with video. This is certainly the case with AVIF.

In the image case, though, the model is that we give the browser a list of options and it chooses the best. In the video case, the sites needs to learn enough about the capabilities to make its own choice about what to download. How would that work with this proposal ?

[gregwhitworth]

In the video case, the sites needs to learn enough about the capabilities to make its own choice about what to download.

Why though? From what I can tell both scenarios are wanting to provide the best experience to the end user based on their device capabilities when provided with a set of possible media assets. Video, from what I can tell is the same as the image scenario unless I'm missing something.

This will solve the bifurcation scenario as well as the code path is already split within the platform as the element that these will be tied to in the engine will be that of a <source> element that is a child of either <picture> or <video> which will allow TV's, or any other media type for that matter, to adjust their answers to either the graphic or video plane accordingly.

I would expect any image format that supports HDR to also support SDR, just as with video. This is certainly the case with AVIF.

Sure, but can an SDR file format support HDR (I would assume no, but I'm pretty new to this space and don't want to assume). I ask because if that isn't the case then HDR is known by the type support of the asset on the <source> element.

So on a device that has a UA & an output device that supports HDR it would want to select an HDR video or image format available to provide the best experience for their user.

Let me know if there is a scenario that I'm missing.

[tidoust]

@chcunningham said:

I think @jernoble meant the same features would be defined in all scopes, but the values for those features would differ by scope. Ex: resolution in the scope of @media tv would query the dpi of the video plane, while resolution for @media screen would query the dpi of the graphics plane.

That matches what I had understood. Hence the question: what does @media all and (resolution > 2dppx) mean when run on a device that matches the tv (or video) media type? If that new media type is part of screen (as described in #135 (comment)), it seems it should target the graphics plane, be it only not to confuse authors who are not aware of the two-plane issue. But then, technically, that query includes @media video and (resolution > 2dppx), which targets the video plane. It may be just a matter of specifying behavior in the Media Queries spec. I still find it a bit clumsy from an author's perspective.

Another question is: how can an author create a media query that matches both a feature of the video plane and a feature of the graphics plane? For instance to define styles that only apply to certain resolution conditions on both planes? There can be only one media type in a media query, the following is not possible: @media video and (resolution > 2dppx) and (screen and (resolution > 2dppx)). It may be that it never makes sense to combine features of both planes, do we know?

In other words, it seems to me that the proposal is trying to hook into the media type mechanism to define something that should rather be at the feature level.

Media Queries define prefixes (min-, max-) for range features. Could a new video- prefix be another possible solution here? E.g. @media screen and (video-resolution > 2dppx) and (resolution > 2dppx), where video-resolution targets the video plane and resolution targets the graphics plane.

[mwatson2]

In the video case, the sites needs to learn enough about the capabilities to make its own choice about what to download.

Why though?

What I mean is that the site needs to know the decision, so that it can request the correct media to download via XHR / fetch. In our system, the device capabilities are advertised in the request for content metadata which is sent to the server and the response contains an appropriate set of streams for those capabilities. We do not know the available formats for the title before sending this request. We also would not want to get back all possible streams in the response and have the client filter, since that would bloat the response size and calculating URLs for the streams is expensive.

It's not clear to me that a declarative approach supports this.

[gregwhitworth]

which is sent to the server and the response contains an appropriate set of streams for those capabilities

This is good to know, so the current design is the inverse of how this same scenario is handled for images and video currently per mimetype

It's not clear to me that a declarative approach supports this.

Yeah it wouldn't, the scenario you outlined above is effectively the same process just flipped where the site handles it rather than the UA. Because of this, I'll abandon this option unless there comes a time where UA wants to make a decision based on information that may not be possible to provide to the site.

E.g. @media screen and (video-resolution > 2dppx) and (resolution > 2dppx), where video-resolution targets the video plane and resolution targets the graphics plane.

@tidoust I really like this approach, I agree that I'd prefer for us to avoid using the media device types as another device may have the same issue.

Does anyone else have thoughts regarding this MQ approach?

[chcunningham]

That matches what I had understood. Hence the question: what does @media all and (resolution > 2dppx) mean when run on a device that matches the tv (or video) media type? If that new media type is part of screen (as described in #135 (comment)), it seems it should target the graphics plane, be it only not to confuse authors who are not aware of the two-plane issue. But then, technically, that query includes @media video and (resolution > 2dppx), which targets the video plane. It may be just a matter of specifying behavior in the Media Queries spec. I still find it a bit clumsy from an author's perspective.

I agree, we would need to specify what this means, probably to choose the graphics plane as the path of least surprise. Also agree that its a bit clumsy. It may still be the least of 3 evils. Folks who want to do 4K HDR video would hopefully be more savvy and specific with their query.

There can be only one media type in a media query, the following is not possible: @media video and (resolution > 2dppx) and (screen and (resolution > 2dppx)). It may be that it never makes sense to combine features of both planes, do we know?

Another great point. I don't have a good use case for styling video and screen in the same way. If I'm wrong, the escape hatch is simply duplicating the style rules for the 2 queries. CSS mixins via pre-processors (e.g. LESS) may save on actual typing.

Media Queries define prefixes (min-, max-) for range features. Could a new video- prefix be another possible solution here? E.g. @media screen and (video-resolution > 2dppx) and (resolution > 2dppx), where video-resolution targets the video plane and resolution targets the graphics plane.

This is totally viable, it just means more manual duplication of things we might otherwise get for free with a media type. The running list of features to prefix includes: color, color-gamut, color-index, device-aspect-ratio, resolution, height, width

[chcunningham]

Does anyone else have thoughts regarding this MQ approach?

Sorry, didn't mean to bury this. I think we're more or less at rest on this thread now. I'll take another stab at summarizing the options for you to send over to CSS.

[chcunningham]

New summary:

For most UA implementations, video is like any other element in terms of its participation in the CSS and Document object model. But TVs (and cast dongles) have distinct code/hw paths ("planes") for rendering video vs other elements ("graphics"). Generally, video can be rendered at the devices native resolution (e.g. 4K) and can take full advantage of the devices HDR/WCG capabilities, while graphics are likely rasterized at a much lower resolution (720p or 1080p) with ~sRGB 8 bit color. Many TVs have this problem. TVs are an very important media consumer. This problem is not predicted to work itself out with time.

With this in mind, we aim to design an API that works for TV and non-TV UAs to signal when a Screen supports HDR and/or 4K resolution. We've discussed 3 approaches.

clean but half-broken: Add Screen.hdrSupported alongside existing Screen.width and height properties

• Like width and height, hdrSupported is assumed to describe hdr support for video and graphics alike. This is a natural extension of the current Screen object, but it offers TVs no way to signal their distinct video vs graphics capabilities.
• TVs can pick one plane to describe and lie about the other. Lying is obviously unpleasant, and it has various trade-offs / breaks depending on which plane is reported and specific use cases (covered earlier in this thread).

verbose but functional: Add video* prefixed media queries media features for the video plane. The running list of features to prefix includes: color, color-gamut, color-index, device-aspect-ratio, resolution, height, width and (new) hdrSupported

• TVs will use the video* prefixed versions to surface the hdr and 4k support for video, while signalling sdr and 720p-1080p for graphics via the existing (unprefixed) properties. For non-TVs, the video* prefixed values will all be the same as their unprefixed counterparts.
• This is equivalent (and could accompany) an earlier proposal to add a new "video" property bag to Screen for video plane properties.
• This completely solves the problem. It also standardizes this TV quirk into the web platform in a highly visible manner.

clean and functional, but requires new semantics media types: Add a new media-type (e.g. 'Video') where TV's could report the features of their video plane while continuing to use 'Screen' for graphics. Add a new hdrSupported media feature alongside existing width and height features, all of which would be accessible from both the Screen and Video media-types.

• This still standardizes the two-plane quirk, but hopefully with less of a footprint (if you're not writing a TV app, you might not even notice).
• New semantics required because we need Screen and Video to both be match-able by a given UA (where types historically are mutually exclusive). Media types are somewhat deprecated/discouraged - unclear whether this might breath some life back into them.

[gregwhitworth]

@chcunningham I was just talking this morning about re-pinging this thread; that's a great summation I'll file an issue with the CSSWG and get it added to the agenda for an upcoming telecon. Thanks!

[svgeesus]

@gregwhitworth

I'll file an issue with the CSSWG and get it added to the agenda for an upcoming telecon.

Thanks, Greg. I'm just reading this issue today. I think that adding this to Media Queries is the right thing to do; it also impacts CSS Color 4 (or 5, if HDR gets resolved there).

I saw a couple of assertions upthread that 'Web content is all sRGB' and 'SDR is all sRGB'. Those used to be true assumptions, but are no longer true:

• Many devices exist with P3 gamut (most current Apple screens, Samsung phones, wide gamut laptops from Dell, HP, etc). This is why there is a gamut media query, to roughly sort into 3 capability clusters: sRGB-ish, P3-ish, and (eventually) rec-2020-ish.
• CSS Color 4 extends from the sRGB gamut to include display-p3, an adobe rgb compatible space a98-rgb), prophoto-rgb, rec-2020, and indeed CIE Lab and LCH.
• CSS Color 4 (or 5) will add support for HDR as well (once there is agreement on compositing mixed SDR and HDR content)
• Canvas is busy adding wide gamut and eventually HDR support
• There is interest in displaying HDR still images, as well as video, in Web content.

Devices which have separate video and graphics planes with different capabilities have not been addressed to date in CSS specifications. I would like to see that change, and welcome this issue as a start to that conversation. I would like to see this group and CSS WG working more closely together.

[svgeesus]

Noticing this issue is still open, some updates from the CSS side:

• Mediaqueries-5 has a video-dynamic-range feature, which is separate from the dynamic-range feature on Screen
• similarly Mediaqueries-5 has a video-color-gamut feature , which is separate from the color-gamut feature on Screen
• The p3 gamut query value is defined relative to Display P3 (which now has a formal definition) rather than the related DCI P3

[chrisn]

Thanks @svgeesus. Media WG still has an open question in w3c/csswg-drafts#6891 on how to expose the video plane pixel ratio. Discussion there asked how to handle the case where there are multiple physical screens or foldable devices. Should we be looking to add this to Screen or possibly ScreenDetailed in Multi Screen Window Placement, or somewhere else?

[svgeesus]

I think multiple screens and foldable screens are a red herring here. The issue is a single screen that has multiple planes, which are composited together and thus seem to the user like a single screen, but the different planes can have different resolutions, color gamut, (pixel aspect ratio, though I thought that was always square for modern formats) and dynamic ranges.

The important thing is to avoid the situation where the screen plane capabilities (say, 1080p, sRGB, SDR) are erroneously queried or used to construct a video request, ignoring the (4K, P3, HDR10+) video plane and thus getting an SDR video.

[chcunningham]

I agree w/ @svgeesus take.

[chrisn]

I'll close this, as the solutions are outside the scope of this spec. But we do need to keep track of progress on the CSS features, and help move those forward as needed. We could do this in w3c/media-and-entertainment#96.