[cssom-1][mediaqueries-5][css-color] Dealing with bi-plane (video/graphics) when reporting values

[gregwhitworth]

Hey folks,

Over in the Media WG we're trying to help web developers serving media content provide the best end user experience and we've encountered an interesting scenario that we'd like to get the CSSWG's feedback on. @chcunningham did a great job summarizing the very long discussion down to the various options we feel are available here.

New summary:

For most UA implementations, video is like any other element in terms of its participation in the CSS and Document object model. But TVs (and cast dongles) have distinct code/hw paths ("planes") for rendering video vs other elements ("graphics"). Generally, video can be rendered at the devices native resolution (e.g. 4K) and can take full advantage of the devices HDR/WCG capabilities, while graphics are likely rasterized at a much lower resolution (720p or 1080p) with ~sRGB 8 bit color. Many TVs have this problem. TVs are an very important media consumer. This problem is not predicted to work itself out with time.

With this in mind, we aim to design an API that works for TV and non-TV UAs to signal when a Screen supports HDR and/or 4K resolution. We've discussed 3 approaches.

1. clean but half-broken: Add Screen.hdrSupported alongside existing Screen.width and height properties

Like width and height, hdrSupported is assumed to describe hdr support for video and graphics alike. This is a natural extension of the current Screen object, but it offers TVs no way to signal their distinct video vs graphics capabilities.

TVs can pick one plane to describe and lie about the other. Lying is obviously unpleasant, and it has various trade-offs / breaks depending on which plane is reported and specific use cases (covered earlier in this thread).

2. verbose but functional: Add video* prefixed media queries media features for the video plane. The running list of features to prefix includes: color, color-gamut, color-index, device-aspect-ratio, resolution, height, width and (new) hdrSupported

TVs will use the video* prefixed versions to surface the hdr and 4k support for video, while signalling sdr and 720p-1080p for graphics via the existing (unprefixed) properties. For non-TVs, the video* prefixed values will all be the same as their unprefixed counterparts.

This is equivalent (and could accompany) an earlier proposal to add a new "video" property bag to Screen for video plane properties.

This completely solves the problem. It also standardizes this TV quirk into the web platform in a highly visible manner.

3. clean and functional, but requires new semantics media types: Add a new media-type (e.g. 'Video') where TV's could report the features of their video plane while continuing to use 'Screen' for graphics. Add a new hdrSupported media feature alongside existing width and height features, all of which would be accessible from both the Screen and Video media-types.

This still standardizes the two-plane quirk, but hopefully with less of a footprint (if you're not writing a TV app, you might not even notice).

New semantics required because we need Screen and Video to both be match-able by a given UA (where types historically are mutually exclusive). Media types are somewhat deprecated/discouraged - unclear whether this might breath some life back into them.

We'd love to get your feedback on the proposals. Thanks.

[AmeliaBR]

My first instinct is that the most-CSS-y way to handle this would be through new video-* prefixed media features, although I don't know if there need to be as many new features as suggested.

It sounds like most of the use cases would be covered by video-resolution and video-color-gamut (with values and interpretation equivalent to the resolution and color-gamut queries, but specifically for the video rendering pathway). I don't think we need to duplicate the legacy color-related queries.

For rendering within a browser window, the width and height of the screen in CSS px would remain the same, only the dppx resolution would be different, so they wouldn't need separate queries.

But for full screen video, the dimensions would be different. So maybe there would need to be full-screen-video-width/height/aspect-ratio queries. If that is necessary, it might be best to report the raw pixel counts (not CSS px), since that's what people are used to when talking about video resolution. However, this is also very similar to the dropped device-width and device-height queries, and I'm not sure why those were dropped & whether this would have the same problems.

[svgeesus]

Note that the big game changer here is the hdrSupported.

I like the "verbose but functional" suggestion to add that to both Screen and Video.

I don't want TV-oriented content to be erroneously deciding that HDR is not supported because "there is no separate video plane and everyone knows the graphics plane is SDR only", for example.

I'm going to head over to the original issue to explain why we like media features rather than media types. @frivoal please feel free to help out/correct me if wrong.

[AmeliaBR]

the big game changer here is the hdrSupported.

I don't like the idea of using a Boolean for this. What happens when a new "super-high dynamic range" tech comes out? Or a "HDR lite" tech (higher than standard but not as high as other HDR devices)? The benefit of using an enumeration like color-gamut is that it can be expanded in the future.

[vi-dot-cpp]

@AmeliaBR
I don't like the idea of using a Boolean for this.

Enumerating HDR (video) display capabilities would require two (2) new properties -- color-gamut and transfer-function -- to be exposed in granular detail, which gave rise to fingerprinting concerns. Since current HDR formats use the Rec. 2020 color space and PQ transfer function, the Media WG opted for a single Boolean to assuage fingerprinting concerns whilst still satisfying content providers' needs. This #118 comment explains the risk.

What happens when a new "super-high dynamic range" tech comes out? Or a "HDR lite" tech (higher than standard but not as high as other HDR devices)?

We have kind of kicked this concern around, but I do not recall a clear resolution. For the sake of my own understanding, I welcome feedback. Please let me know if this discussion should be separated into a new issue as to not derail the bi-plane discussion.

[tabatkins]

CSS doesn't do booleans, because they're often a bad design for the reasons Amelia brought up, but it's easy to assuage that with an enum that currently just has two values, but can be extended in the future.

And Chris is right; media types other than screen/print are deprecated and we don't plan to do any further work in that space. Overall they were a failure for what they intended (implying a number of MQ features as a bundle, before we invented the rest of MediaQueries), and we instead just handle their use-cases with more MQs when necessary. They were never useful for y'all's desired purpose, of reporting different values to some MQs based on what media type is being asked about, because they're evaluated totally independently of the rest; in effect they're just a (device-type: ...) MQ.

So yeah, I'm with Amelia and Chris in preferring option 2, where we define some video-* MQs that duplicate some of the existing MQs, but specifically refer to the video plane.

Like Amelia said, I don't think we need to carry over some of the weirder legacy color/sizing MQs, but I do think height/width are useful, so I suggest video-color-gamut, video-resolution, video-width, and video-height. Is there anything else that needs to be exposed for negotiation? @vi-dot-cpp mentions transfer function; is that something y'all will need?

[chcunningham]

Agree with @vi-dot-cpp - I don't think we want to add transfer function to Screen. We instead added it to mediacapabilities.decodingInfo(), which works well for both the classic desktop/laptops as well as smart TV setups.

strawman: what if we do an enum called dynamic-range, with values { standard, high }. In 2025 we can add crazy-high or whatever the marketing folks call it ;)

@mwatson2

[chcunningham]

Seems option 2 is the favorite, so this would imply adding dynamic-range as well as video-dynamic-range (in addition to other video-* prefixed @tabatkins mentioned.

[mwatson2]

dynamic-range enum is fine for us.

Actually, the dynamic range supported by the PQ transfer function is already "crazy high" and the 2020 color gamut "crazy wide". Never say never, but it will be a long long while before consumer devices are capable of fully supporting either. Content today is mastered within a subset of the range that is expressible in this pixel encoding, but that subset still well exceeds existing consumer devices. Devices are expected to map down (tone map) from the provided range / gamut of the content to their actual capabilities. Provided a device has that tone mapping capability (which we'll discover through decodingInfo) and has a display capability that is genuinely better than SDR, then we would prefer to deliver HDR. Such devices should declare high for this new enum.

[AmeliaBR]

what if we do an enum called dynamic-range, with values { standard, high }.

The currently defined CSS color-gamut media feature uses keywords to describe existing commercial standards: srgb, p3, rec-2020. Does that cover what you need, or is that a distinct concept?

Or, could it cover what you need when combined with a measure of color bits per pixel, as suggested by the example in the CSSOM View spec describing the colorDepth value of the Screen interface? (The color media feature provides similar information to colorDepth, although not in a directly interchangeable format.)

It would definitely be preferable to harmonize the concepts used in media queries with those used in CSSOM View, even separate from the bi-plane considerations. If we can do that, then it seems like a natural extension to handle bi-plane rendering by extending the necessary media queries to have video-* versions, and extending the Screen interface to have a screen.video sub-object.

[chcunningham]

@mwatson2 - keep me honest

The currently defined CSS color-gamut media feature uses keywords to describe existing commercial standards: srgb, p3, rec-2020. Does that cover what you need, or is that a distinct concept?

They are distinct but interconnected concepts. Color gamut describes a color spectrum while SDR/HDR is about luminance range.

In practice, different gamuts do correlate with sdr / hdr, but there are edge cases (like Apple's 2017 P3 displays, which are not HDR) and I think its not clean enough to live without a separate dynamic range signal.

[mwatson2]

@chcunningham is correct that these are distinct concepts. Color gamut, color depth and dynamic range are all distinct things that can vary independently, though not all combinations are in use.

In particular, BT.2020 color gamut is used with both standard dynamic range and high dynamic range as is 10 bit color depth. Color depth is about precision, not range. (btw, Example 3 here is wrong about this).

To further clarify the problem description, given media in a specific format we must ask first whether the media can be rendered at all in some reasonable form and second whether it should in the case the site has a choice of formats. The first is a question about UA support for the codec and pixel data format and it's ability to map that meaningfully to the output device. The second is a question about the raw capability of the output device.

We separate these questions because in some cases the site might not have a choice of formats and is only interested in whether the format can be rendered or not. If the site has a choice of formats, all of which can be rendered, it needs to know which is worthwhile and will produce the best result. Unless you have no choice, there is no point sending HDR if the output device is SDR, even if the UA has the capability to map it down.

[svgeesus]

Agreeing with others that color gamut (how saturated a color can be displayed) and dynamic range (how many photographic stops of luminance can be displayed {all at once | at various times}) are distinct concepts although they are slightly correlated (HDR tends to require WCG as well, also higher bit depth per component).

Since current HDR formats use the Rec. 2020 color space and PQ transfer function, the Media WG opted for a single Boolean to assuage fingerprinting concerns whilst still satisfying content providers' needs.

That single Boolean will struggle with content that uses Rec.2020 colorspace and the HLG transfer function, though. "Everything uses PQ" is a Film/Movie-centric view; it is true there but not so much for News or Live Sports, for example.

[vi-dot-cpp]

That single Boolean will struggle with content that uses Rec.2020 colorspace and the HLG transfer function, though. "Everything uses PQ" is a Film/Movie-centric view; it is true there but not so much for News or Live Sports, for example.

Ah, that's my own mis-remembering, not negligence on the part of the Media WG. Do you all have thoughts the dynamic-range enum? It looks to offer extensibility and preserve privacy. Thanks for proffering the strawman @chcunningham.

If we opt for this enum, should we define a rubric by which to judge the dynamic range of a display?

[mwatson2]

That single Boolean will struggle with content that uses Rec.2020 colorspace and the HLG transfer function, though. "Everything uses PQ" is a Film/Movie-centric view; it is true there but not so much for News or Live Sports, for example.

PG vs HLG is important for the can this be rendered question (which is answered by decodeInfo() but not for the should I choose this one (when I have a choice).

If we opt for this enum, should we define a rubric by which to judge the dynamic range of a display?

Good luck with that ;-) The question I want to answer is whether it would produce a better result to send HDR content vs SDR when I have both available and the device can render both. At least at the margins this is a matter of opinion. So then I think we should leave this to UAs, as agents of the user, who is the one with the opinion.

[vi-dot-cpp]

Looks like we are converging on the following, feel free to chime in:

2. verbose but functional: Add video* prefixed media queries media features for the video plane.

CSS guidance is to add a video attribute to screen, denoting the video rendering pathway as opposed to that of graphics, for the purpose of addressing the bi-plane problem among TVs. The video attribute will expose the following properties:

• width
• height
• resolution
• dynamic-range

Screen HDR capabilities are represented the dynamic-range enum for the following reasons:

• Color-gamut in conjunction with existing Screen properties are not sufficient for detecting HDR capabilities.
• Bucketized in order to alleviate fingerprinting concerns.
• Enum with values { standard, high } to leave room for future augmentation to dynamic range standards.

There is concern though that { standard, high } are qualitative/un-standardized. As @mwatson2 noted:

Actually, the dynamic range supported by the PQ transfer function is already "crazy high" and the 2020 color gamut "crazy wide".

Are we able to address this concern by standardizing recommended ranges for properties that comprise dynamic range? What are your thoughts/feedback?

[chcunningham]

I think you mean for width to be in "CSS pixels" and same for height (length is typo). For clarity, resolution is "the number of device pixels per css unit". All that to say: while the values may differ between video* prefixed vs unprefixed features, but the units/definitions should be common to both.

There is concern though that { standard, high } are qualitative/un-standardized. As @mwatson2 noted:

Actually, the dynamic range supported by the PQ transfer function is already "crazy high" and the 2020 color gamut "crazy wide".

I think he's just saying that some of the underlying pieces (eotf and color gamut) have lots of headroom to describe a category of devices that don't exist yet. Said another way, 5 years from now when a new generation of TVs may have a much higher contrast ratio, and we'll call it "crazy high" dynamic range, but PQ may still be a suitable transfer function.

Mark's later comment is squarely on the side of not trying to assign tight definitions to categories of dynamic range. I think this sounds practical. There are a number of competing HDR badge requirements (e.g. UHD vs Vesa) and its a bit of a minefield to walk in.

[vi-dot-cpp]

I think you mean for width to be in "CSS pixels" and same for height (length is typo). For clarity, resolution is "the number of device pixels per css unit". All that to say: while the values may differ between video* prefixed vs unprefixed features, but the units/definitions should be common to both.

You're right. I've made edits to avoid confusion.

[astearns]

@gregwhitworth there's been a lot of discussion since you added the agenda+ tag. Are we converging on option 2, as above? Let's continue the discussion here and possibly take it up on the call next week.

[gregwhitworth]

@astearns Yep, I looked at the agenda list when I filed this and didn't expect it to make the agenda until about the 20th. Can we plan on it for that week?

[gregwhitworth]

The media working group just resolved on this:

video-* properties of width, height, resolution, dynamic-range, color-gamut within CSS for media queries

I'd like to add this to the agenda to get resolution in CSSWG as well.

[svgeesus]

I think that the media wg made a good resolution, so urge css wg to resolve the same.

[AmeliaBR]

Do we have a link to the Media WG discussion & resolution?

Specifically, did the group make any statement about a preferred CSS OM API, or only about the media queries?

[chcunningham]

The minutes from that meeting are here, but we didn't really discuss your question.

I think the focus on media queries is mostly an artifact of the discussion having started with the (now abandoned) proposal number 3 above, which would add a new media queries media-type for the video plane. We also discussed how media queries will allow you to subscribe for events when things change - important in multi-display use cases.

Having the information be also exposed on Screen seems like a nice-to-have. Is it generally a best practice to expose things in both ways? If so, I think either Screen.video.dyanmicRange or Screen.videoDynamicRange (and similar for other properties) would work - whichever is more idiomatic.

[css-meeting-bot]

The CSS Working Group just discussed [cssom-1][mediaqueries-5][css-color] Dealing with bi-plane (video/graphics) when reporting values, and agreed to the following:

• RESOLVED: Add the video prefix MQ and define how used in bi-plane and non-bi-plane environments

The full IRC log of that discussion

<dael> Topic: [cssom-1][mediaqueries-5][css-color] Dealing with bi-plane (video/graphics) when reporting values
<dael> github: https://github.com//issues/4471#issuecomment-555743057
<dael> gregwhitworth: Overview: the media WG looking into adding to capabilities API. Wanted to determine what in an output can support. WE came up with 3 options that are in the issue.
<dael> gregwhitworth: AmeliaBR chris and TabAtkins said video-[property] made sense. Why we need this is TV manufactuors have 2 graphics plans. Where movie is shown gets different high support but menus might get lower capability and they don't plan to change that.
<AmeliaBR> s/graphics plans/graphics planes/
<dael> gregwhitworth: They want to tell the truth and if we only have one plane they have to lie. chris went over it in the beginning where we could jsut tell them to lie, but ultimately what WG came to is video-[property list] that works in CSS media queries so we can use in JS or CSS and see if UA supports them
<dael> gregwhitworth: Does anyone disagree with the resolution in media WG
<dael> chris: Clarification; TVs have 4k for video and lower. I didn't want TVs to say this is how TVs work and we don't care about PCs. But would they return the same if there's one?
<dael> gregwhitworth: Yes, we'd answer same statement if there's one plane.
<dael> chris: Then yes, I think this is the best possible and I support
<dael> florian: For video playing we're talking about if it's full screen and not window mode?
<dael> gregwhitworth: Yes. We've been vague on what we mean because it's not a standarized term
<dael> florian: I mean something that does not depend on layout
<dael> gregwhitworth: My expectation would be no. I don't htink they're doing web layout. THey're different roles.
<dael> florian: As long as it's a full thing then sure
<dael> chris: I think it would work. Hardware would do a mask
<dael> florian: Sure, that works. It's a MQ and shouldn't depend on layout
<dael> AmeliaBR: I do think it's true that some environments use the separate plane for embedded videos, but it's at the level where they're rendering video data and converting to screen pixels. What MQ would match is assming full screen
<dael> gregwhitworth: Right, wouldn't be layout of plane but dimension of the plane. It would be saying TV supports 4k and returns true with width and height
<dael> myles: How does web content say I want this element to go in this plane
<dael> gregwhitworth: You don't. WebDev doesn't get to derrive which element it goes on
<dael> AmeliaBR: Part of definitions would be if rendering agent isn't using separate plane then these values would match regular MQ
<dael> gregwhitworth: Correct
<dael> myles: If every peice of content has non-standard parts what's rational for standarizing this MQ
<dael> gregwhitworth: This would be like standarizing GPU chipsets. It's their version of hardware
<dael> myles: Why standardize? Why not have own custom thing to annotate
<dael> gregwhitworth: They're rendering web content. That was an option is pretend this doesn't exist. We keep only 1 query and force TVs to lie. But we had content providers say we want to know wha video is capable of rendering. I want the truth for each. I don't want ot send 4k images on lower resolution
<dael> florian: Don't want UA sniff
<dael> gregwhitworth: Yes, that's part of why 3 options. And it's not technically TV only but that's the angle
<dael> chris: I want this to work same for non-TVs
<dael> AmeliaBR: That's important consideration. MQs are saying when you render video what are features going to be. Sensible use case is on source lements in a video element that you're using to pick correct source.
<bkardell_> what constitutes a 'tv' here?
<bkardell_> seems like a weird term of distinction?
<dael> AmeliaBR: How the browser decides what type of rendering environment it's going to use for video isn't important. Only thing that matters is UAs are using different rendering for video then for rest of content so existing MQs aren't sufficient.
<dael> gregwhitworth: Yep
<dael> AmeliaBR: If in future we have situation where this division isn't good enough and some other content wants super high level then that can be a discussion at the time. Reality of environments we have now is high resolution is for video
<dael> gregwhitworth: My expectation also is that over time things will converge so 4k is available in both. Not reality now and b/c TV isn't upgraded often content providers want to provide correct content
<dael> AmeliaBR: bkardell_ asks what's a TV here and we don't have to define that. We're defining the video resolution regardless of if it's a TV or a high powered laptop with a separate video card
<dael> chris: Important point is we don't care but the answer is clear. Many times it will be the same but there are cases where it's not.
<dael> chris: I don't see a down side. For most cases this is a bunc hof alaises
<dael> astearns: Prop: add the video prefix MQ and define how used in bi-plane and non-bi-plane environments
<dael> bkardell_: sgtm
<dael> astearns: Obj?
<dael> RESOLVED: Add the video prefix MQ and define how used in bi-plane and non-bi-plane environments
<bkardell_> s/bkardell_/gregwhitworth
<dael> chris: I think we should open a new issue on the OM thing
<dael> astearns: I was about to bring that up. I think we should have a new issue for the OM part so summerizing that separately will be easier.
<dael> chris: Anything I need to do for color 4?
<dael> gregwhitworth: No, that was me ensuring chris got on the thread
<chris> lol
<dael> fantasai: For the OM part is there stuff that's parallel with the media queries or is there some additional thinking?
<dael> AmeliaBR: Not exactly parallel.
<dael> AmeliaBR: Last comment in issue is one of the Media folks saying they want to discuss on best API
<dael> astearns: Let's open a new issue, hash it out, and bring to group
<dael> AmeliaBR: Who is doing edits?
<dael> gregwhitworth: I can take a stab at it
<dael> fantasai: MQ 4 right?
<dael> fantasai: We need a FPWD of MQ 5 at some point
<fantasai> s/MQ 4/MQ 5/
<fantasai> https://drafts.csswg.org/mediaqueries-5/#contents

[gregwhitworth]

@frivoal @tabatkins I'm finally getting around to making this edit (sorry for the delay) but I had a quick editorial question. Since ALL of them should contain the same values as the non-video counterparts, how do we want to represent that? Do we want it all duplicated or something along the lines of same values as <property link to propdef within MQ>?

[AmeliaBR]

@gregwhitworth Interesting question, maybe worth a separate issue?

CSS Values defines a way to reference the value syntax for properties (<'background-size'> refers to any valid non-list value for the background-size property), but there's no equivalent for media query values. We can't reuse the same syntax because there would be name conflicts. (E.g., the color media feature has a very different value syntax than the color property!)

Maybe <media-feature 'color'> or something like that…

[tabatkins]

I'm fine with just repeating the values for now. We can add a new non-terminal syntax if we start doing this sort of thing more often.

Remember, WET (Write Everything Twice), not DRY!

[Crissov]

So, although we all agree that the color-gamut keywords are badly designed, but need to stay as is because they are already shipping, we still copy them verbatim to video-color-gamut?

[gregwhitworth]

@Crissov Can you expound on the agreement of it being badly designed? I'm also going to tag in @svgeesus as he's the color editor and will have far more historical context than I will.

[Crissov]

Sorry, I forgot to mention the issue: #4535

[svgeesus]

"We" don't regard them as badly designed. They might have been better as more generic keywords, but that would have run the risk of needing more specific examples.

Anyway, that doesn't need to hold up this issue.

[svgeesus]

@gregwhitworth are you still able to make the required edits to close this issue, or should we look for other volunteers? Not sure if this is still in scope for you.

[chcunningham]

Hi @svgeesus - is the outstanding work tracked in #5044, or is there more? New updates on that issue just now.

[svgeesus]

is the outstanding work tracked in #5044, or is there more?

These seem at least highly related, possibly duplicates. Maybe close this one and leave #5044 ?

[chcunningham]

Maybe close this one and leave #5044 ?

Fine with me (I don't have that permission though)