VTT Type Proposal for time-coded general-flash metadata

[cookiecrook]

Earlier today, Apple released a new strobing mitigation media accessibility accommodation in tvOS 16.4, iOS 16.4, and macOS 13.3, called "Dim Flashing Lights." (Additional links below)

Prior to this, the typical behavior with flashing media was that a viewer was warned once before before a video played. Even in a feature length film, there was no indication of when the flashing may occur, and no information about the intensity or type of strobing.

On Apple devices, we can now adjust for some strobing/flashing in real time with onboard hardware and software, but as Apple's TV+ content is viewable on many other platforms, we'd also like to be able to do this on non-Apple hardware.

We think proposing a time-coded metadata track of flash level as an accessibility standard makes sense, as other platforms may wish to adopt their own form of strobing mitigation or warnings.

So we'd like the WG to consider using a WebVTT metadata track to allow streaming services and content owners to denote areas of flashing in video time codes, similar to a caption track.

Minimum proposal

WEBVTT

1
00:00:10.123 --> 00:00:15.432
{
  type: "video.strobing.general-flash",
  level: "75"
}

2
00:00:47.462 --> 00:01:04.028
{
  type: "video.strobing.general-flash",
  level: "100"
}

We're not particular to the proposed key names or value formats, but the type value above aligns with an expected update to HLS, and accounts for future expansion into other common strobing patterns such as red-flash and spatial-pattern.

Variant proposal that accounts for algorithm and version

Ideally, in order to ensure all content publishers, software developers, and users have the same understanding and representation of the level, the VTT cue format could account for different, documented algorithms and versions for each of those.

Apple published a new general flash algorithm (PDF) for use alongside the related native platform API or for use in other contexts.

In order to differentiate from other flash pattern algorithms in WebVTT (Harding, for example), the metadata cue could contain a published URI (or registered name?), as well as a version number. For example:

WEBVTT

1
00:00:10.123 --> 00:00:15.432
{
  type: "video.strobing.general-flash",
  level: "75",
  test-uri: "https://developer.apple.com/accessibilty/...",
  test-version: "1.0"
}

2
00:00:47.462 --> 00:01:04.028
{
  type: "video.strobing.general-flash",
  level: "100",
  test-uri: "https://developer.apple.com/accessibilty/...",
  test-version: "1.0"
}

We're not particular to the proposed key names or values, and look forward to discussing further with the Working Group.

Prerequisite:

I have filed another issue we believe is important to consider alongside this one.

• VTT Metadata Cue format is ambiguous; some metadata may be unintentionally presented to the user in a context outside HTML #511

Other documentation on the recently released feature

• Dim Flashing Lights Overview
• Explainer and Algorithms (PDF)
• Apple Developer API for Flashing Lights
• Sample Code: Responding to changes in the flashing lights setting
• GitHub Sample Code: Detection of flashing lights in video content

[cookiecrook]

@eric-carlson @chrisn @gkatsev

[gkatsev]

Dim Flashing Lights looks fascinating!

This is an interesting proposal. I think this would be the first time someone asked to standardize a specific metadata format in WebVTT. I think historically, this has probably been left to users of WebVTT. Furthermore, I don't think we'd want it in WebVTT itself, but could still be a related spec or something. Not exactly sure what the options available are. Happy to discuss it further.

Also, I'm wondering how this would be represented in HLS. Would it allow for segmented VTT and then the main HLS playlist would reflect that his is a Dim Flashing Lights metadata playlist?

[cookiecrook]

Thanks! I'll ping some of the HLS folks to reply to those questions.

[chrisn]

I have a couple more questions, following discussion in today's TTWG meeting:

• In your view, does this use case motivate the use of a metadata-specific cue API, i.e., DataCue, rather than using VTTCue?
• What are the synchronization requirements for handling of the time codes? For example, is frame accuracy needed? There's a long discussion in Frame accurate seeking of HTML5 MediaElement media-and-entertainment#4 on achieving frame accurate synchronization with the HTML <video> element.

[eric-carlson]

I have a couple more questions, following discussion in today's TTWG meeting:

• In your view, does this use case motivate the use of a metadata-specific cue API, i.e., DataCue, rather than using VTTCue?

Are you suggesting that allmetadata tracks should emit DataCue?

If so, what about do we about existing metadata formats like WebVMT.

If not, what makes this JSON-based metadata cue format more suitable for DataCue?

• What are the synchronization requirements for handling of the time codes? For example, is frame accuracy needed? There's a long discussion in Frame accurate seeking of HTML5 MediaElement media-and-entertainment#4 on achieving frame accurate synchronization with the HTML <video> element.

While it is important for the UA or a script to be able to process general-flash metadata in time to dim the video to block the strobing/flashing, I don't think "frame accuracy" is needed (I don't believe it is possible to guarantee frame accurate delivery of JavaScript events in any case).

[chrisn]

Are you suggesting that all metadata tracks should emit DataCue?

No, this would be a breaking change, although in principle a new kind value could be used to request DataCues.

If not, what makes this JSON-based metadata cue format more suitable for DataCue?

In this (and the WebVMT use case), the main benefit seems to be a saving a JSON.parse() call, which wouldn't be a huge improvement.

I was looking in general for motivation to progress DataCue. But given VTTCue can work well enough in these cases I'm not seeing much reason progress it.

While it is important for the UA or a script to be able to process general-flash metadata in time to dim the video to block the strobing/flashing, I don't think "frame accuracy" is needed (I don't believe it is possible to guarantee frame accurate delivery of JavaScript events in any case).

I agree, and so authors or generators of the metadata need to take into account typical client processing time.

[gkatsev]

If not, what makes this JSON-based metadata cue format more suitable for DataCue?

In this (and the WebVMT use case), the main benefit seems to be a saving a JSON.parse() call, which wouldn't be a huge improvement.

Potentially, can extend VTTCue to have a .json property that returns the cue text parsed as json.

[nigelmegitt]

Seems to me a bit the wrong way around to extend VTTCue where the real use case is removing unnecessary stuff, plus adding maybe one thing.

It would make architectural sense to support standardisation of metadata formats outside of WebVTT, and to have something like a DataCue that offers the same timing functionality as VTTCue and keeps a .text attribute for compatibility, but doesn't bother with the presumably completely unused methods and attributes for presenting text. I'm not sure if anything else would actually be needed to avoid this being a breaking change - possibly if people directly test that the cue object is VTTCue (not sure why anyone would)?

The benefits could be:
a) Allows the UA to parse the JSON in its own time, or when needed, whichever is the sooner, thus being more likely able to meet the timing requests for onenter() and onexit() handing.
b) Saves that json.parse() call.
c) Saves a small amount of per-cue-object memory associated with the pointers to the unused and not needed VTTCue attributes and methods associated with cue text presentation.

[eric-carlson]

I was looking in general for motivation to progress DataCue. But given VTTCue can work well enough in these cases I'm not seeing much reason progress it.

I believe DataCue continues to be important for exposing structured binary metadata. The alternatives, exposing the metadata as an ArrayBuffer and requiring every site to parse it, or exposing it as JSON with binary fields base64 encoded, aren't appealing.

[nigelmegitt]

If tying a DataCue to JSON is unhelpful, because it isn't always needed, perhaps there may at some point in the future be a useful subclass of DataCue optimised for JSON processing, a JSONCue for example.

[css-meeting-bot]

The Timed Text Working Group just discussed WebVTT metadata format, issue 512, and agreed to the following:

• SUMMARY: This issue isn't the right place to discuss, but it highlights that there are wider issues around text tracks that need to be understood and resolved. Probably something for MEIG initially.

The full IRC log of that discussion

<cpn> Topic: WebVTT metadata format, issue 512
<nigel> github: https://github.com//issues/512
<nigel> Nigel: There was some more discussion on this issue after last meeting
<nigel> Gary: We should invite those folk to one of our meetings
<nigel> Chris: This digressed into a discussion about DataCue which is perhaps not for this group.
<nigel> .. It feels like it's with me to do anything with it if we're going to do anythihng.
<nigel> .. Nobody is really pushing for it.
<nigel> .. So unless there's more active demand I'm not inclined to spend much more work on it.
<atai> q+
<nigel> ack at
<nigel> Andreas: There was one requestor with the VMT use case for DataCue, to enable the geographical
<nigel> .. use case or service.
<nigel> Chris: Yes, there's that. Perhaps I should propose another WICG meeting on this to think about next steps.
<nigel> .. This is a separate discussion. If there is particular interest from members here then I'd be interested
<nigel> .. to hear that. I'm happy to progress it if there is demand.
<nigel> q+
<cpn> nigel: There's a text track CG, I think, and some months ago Sylvia asked how it's going
<cpn> ... So that got me thinking. It struck me that the current state of how text tracks work on the web has a bunch of problems
<cpn> ... and it's not used in some communities. I have the impression that the core problem that's well solved is triggering things on the media timeline
<cpn> ... But VTTCue has extra baggage, but that suggests to me we should preserve that functionality. So that makes it feel like some kind of cut-down TextTrackCue type can be useful,
<cpn> ... and saving some resource in clients. Not sure whether that's useful. But it seems overloaded
<cpn> chris: sometimes browser vendors argue that having a workaround is enough and it doesn't motivate adding a new feature
<cpn> gary: sometimes it's argued that things don't meet needs with respect to regulation, hence there's no need
<cpn> ... I don't like that everything that gets pushed out to JavaScript, especially when there are features in the browser that are crucial for accessibility
<cpn> ... they don't work on new features as there's no usage, but there's no usage because they don't meet requirements to meet regulations
<atai> q+
<cpn> chris: happy to bring all that to WICG DataCue or MEIG?
<cpn> nigel: I think the whole model of how captions are supposed to work, and other timed features needs to be reopened
<cpn> ... so we should work out what the unmet needs are
<cpn> ... that doesn't feel like a TTWG thing, MEIG first
<cpn> chris: Happy to facilitate
<nigel> ack at
<nigel> ack ni
<cpn> andreas: I support Gary's point. It's hard to understand how much effort browser vendors spend on captions and text track API to enable it
<cpn> ... but it's not adequate. It's a topic for this group, there are two major formats for captions that use the TextTrack API
<cpn> gary: I think this group should be involved in the discussion, but this might not be the home. TextTrack is in HTML, DataCue is in WICG, VTTCue is in TTWG
<cpn> ... Is what we have in the specs sufficient for the needs, and if so we need to start encouraging browser vendors to fix issues and fully implement
<cpn> ... If not, figure out the gaps and get to a place where not every streaming service needs to implement full captioning support in their clients, because the support in clients is insufficient
<cpn> nigel: in terms of this group, I agree, formats live here, but I think there's a more architctural issue
<cpn> chris: what do you suggest as a next step?
<cpn> nigel: need some kind of workshop, to understand the current model and what works well
<cpn> ... a format where we can discuss openly, e.g., why particular services use the approaches they do and what doesn't meet needs
<cpn> gary: TPAC would be a good place to have something like this. But ideally we'd also get started on it before September
<cpn> chris: so we could discuss offline about how to plan something and who to involve?
<atai> q+
<gkatsev> q+
<cyril> q+
<nigel> ack atai
<nigel> group: [discusses freely, off the record]
<nigel> ack gk
<atai> q+
<nigel> ack cy
<nigel> ack at
<atai> q+
<cpn> q+ for the final agenda item
<nigel> SUMMARY: This issue isn't the right place to discuss, but it highlights that there are wider issues around text tracks that need to be understood and resolved. Probably something for MEIG initially.
<cpn> ack acpn
<cpn> ack atai
<cpn> ack cpn
<Zakim> cpn, you wanted to discuss the final agenda item

[gkatsev]

SUMMARY: This issue isn't the right place to discuss, but it highlights that there are wider issues around text tracks that need to be understood and resolved. Probably something for MEIG initially.

This is more specifically about the larger conversation around Text Tracks on the web, rather than this metadata "profile" for WebVTT.

---

To that end, @cookiecrook would you (and other relevant folks) be available for joining one of the TTWG calls? I think it would be a good forum to figure out next steps. We meet on a fortnightly schedule, with the next meeting tentatively scheduled for April 27 and 15:00 UTC.

[cookiecrook]

I plan to attend that day, and will try to bring others to the meeting as well. Thanks.

[nigelmegitt]

This is on the agenda for the call scheduled for April 27 15:00 UTC.

[css-meeting-bot]

The Timed Text Working Group just discussed WebVTT issue 512, and agreed to the following:

• SUMMARY: Apple to think about next steps for incubation

The full IRC log of that discussion

<cpn_> Topic: WebVTT issue 512
<nigel> github: https://github.com//issues/512
<nigel> jcraig: There seems to be a negative response to #511, so should we close that?
<nigel> .. This is on the ambiguity of metadata, and there being no path to know
<nigel> .. if it is metadata or a caption.
<nigel> .. I suggested either a JSON block or a URI, but others are using other types of metadata
<nigel> .. so it is not always clear
<nigel> q?
<nigel> cpn_: I'm wondering what other alternative options for signalling the format we could think of.
<nigel> Gary: I suggested a Metadata block akin to the Region block, should be backwards compatible.
<nigel> .. I haven't tested that, but generally WebVTT says to ignore stuff you don't know about.
<nigel> .. My main concern is backwards compat
<nigel> .. General idea of having an in-file signal for the type of file is a valid request
<nigel> jcraig: OK, I'll leave that one open and Eric and I will look into it more.
<nigel> Gary: Do you want to give an overview of this proposal?
<nigel> jcraig: Sure. [shares screen showing issue]
<nigel> .. A few weeks ago Apple released the ability for Apple hardware devices like a Mac, AppleTV or iOS device
<nigel> .. to automatically dim the screen by looking a few frames ahead at the flash patterns.
<nigel> .. We released an open source library for this on GitHub as well.
<nigel> .. The idea is to help anyone who has a negative reaction to the flashing,
<nigel> .. as an alternative to only having a content warning.
<nigel> .. We'd like to timecode where the flashing is.
<nigel> .. We have an HLS proposal that Eryk V worked on. More information about that soon.
<nigel> .. We would ideally like a way to specify the algorithm optionally.
<nigel> .. In my VTT proposal it just says "level" without defining what that means.
<nigel> .. If we were to include a test-uri and a test-version we could specify unambiguously what that level meant.
<nigel> .. We could do a number of things with this, most obviously adjusting the timeline in some degree.
<nigel> .. Like skipping over the flashes.
<nigel> .. Thank you all who contributed to the discussion.
<nigel> .. It seems like the general consensus is to not use WebVTT for this.
<cpn_> q+
<nigel> .. I'm not as familiar with WebVMT - Eric might want to add more context.
<nigel> .. Happy to try to answer any more questions.
<nigel> .. I know there were some HLS questions in the channel that haven't been answered yet.
<nigel> ack cp
<nigel> cpn_: WebVTT in terms of metadata is completely generic, and does not define any semantics
<nigel> .. about metadata. WebVMT is an example for synchronising location and orientation data with video media.
<nigel> .. It's a standalone specification that builds on top of WebVTT,
<nigel> .. and defines the JSON object carried in the VTT file and its semantic.
<nigel> .. The approach is to define it in its own specification as an application
<nigel> .. rather than being in the WebVTT spec itself.
<nigel> Gary: Yes, WebVMT also is a bit of a fork because it adds extra features to WebVTT.
<nigel> .. If we go that route this metadata format would be simpler because it would not need
<nigel> .. to redefine what WebVTT already defines, it would point to them directly.
<nigel> q+
<nigel> Gary: I was asking about how it would be represented in HLS.
<nigel> .. How do you put it in the manifest, and how do you represent it in the TextTracks object.
<nigel> Eryk: We use the DATERANGE tag in HLS. For conveying this data we have a specific class
<nigel> .. and an added attribute that denotes the risk.
<nigel> .. That's it, in the multi-variant playlist.
<nigel> jcraig: And the class mentioned is more or less the same as the type value.
<nigel> Eryk: Yes, it's a little more verbose.
<nigel> Gary: So the HLS wouldn't be a segmented version of this WebVTT?
<nigel> jcraig: It's different.
<nigel> Gary: That sounds fine.
<nigel> jcraig: The similarity is the timecode and the level - the type would be equivalent but not the same.
<nigel> .. We chose that because general flash and red flash definitions are common in WCAG so we could expand
<nigel> .. it to that pattern.
<nigel> Gary: It's exposed how HLS and Safari expose DATERANGE on the video element?
<nigel> Eric: We expose DATERANGE as a data cue
<nigel> Gary: That answers my questions.
<jcraig> q?
<cpn_> Nigel: Is this an Apple only feature, are other implementers interested?
<cpn_> James: We hope others will be interested, users seem to be excited about it, so we'd like to have other platforms benefit from the idea
<cpn_> ... And we'd like to enable it for Apple's services on other platforms, Android apps and TV+ content for Samsung TVs etc
<cpn_> Eryk: We'll be publishing it at the developers conference
<atai> q+
<cpn_> Nigel: I'm supportive of the idea it should be a separate spec. A good approach could be to draft a spec for this, and try to get support
<cpn_> ... Consider whether it's a Rec track document or a Note
<cpn_> James: I'd defer to Eric or any of you on that
<cpn_> Eric: Not sure there's a benefit to it being on the Rec track, it's more work, and may not be a benefit to anyone else
<cpn_> Nigel: There's a lower bar for publishing a Note
<cpn_> ... A Note isn't normative, it doesn't carry any imprimatur of W3C as a whole, it's a document that people may find useful
<cpn_> ... We've used it for things that look normative, but things useful for industry more than things that are a recommendation
<cpn_> James: Would a Note be a link to an external document?
<nigel> cpn_: A Note is its own document rather than a note within a different document.
<nigel> jcraig: Yes. It's a W3C Document type
<jcraig> q+ EricC
<nigel> Gary: You can't reference a Note normatively from a Rec
<cpn_> q+
<nigel> Nigel: The status of the document on a Note will say nobody should reference it normatively.
<nigel> ack nigel
<nigel> ack atai
<pal> q+
<nigel> Andreas: That's definitely a good activity that I would support.
<nigel> .. More generic question. We're discussing the syntax and the transport container.
<nigel> .. Is there any more thought about a generic semantic model
<nigel> .. that could be used for other formats.
<nigel> .. Is the model already there? Or would it be useful to define it in a way that
<nigel> .. could be used in other syntaxes or containers?
<nigel> Gary: Like in IMSC?
<nigel> Andreas: Yes
<nigel> Eric: Do you mean specifically this kind of metadata or more generically to define a way to be able
<nigel> .. to have other types of metadata in a VTT file?
<nigel> Andreas: This specific flashing metadata
<nigel> q?
<nigel> jcraig: I'm not sure I fully understood that
<nigel> .. We can dig into that
<nigel> Andreas: It's independent of format, it's just a thought that this kind of information
<nigel> .. would be structured specifically somewhere so it could be used regardless of format.
<nigel> .. The semantics and expected behaviour could be similar.
<nigel> jcraig: The data itself is not so complex that it would be difficult to do that transformation into other formats.
<nigel> .. We're using it a different way in HLS.
<nigel> .. I'm not opposed to a reusable structured format, that e.g. MPEG DASH could choose to implement,
<nigel> .. using a format that this group defines, say, but I would not want a requirement to have a pass-through
<nigel> .. format that would need to be supported. If we did want that we would need a Rec track document.
<nigel> .. But a JSON block that can be used elsewhere, I'm all for that.
<nigel> Eric: Or is the suggestion more to have a document that defines a value range, the elements of the metadata
<nigel> .. so that in this case we could use it in JSON form, but it could also be used in XML form, pointing
<nigel> .. back to a document that defines how to interpret them.
<nigel> Andreas: Yes, thank you that's exactly what I had in mind.
<nigel> jcraig: Seems reasonable to me. I don't think value should be defined in the transfer format
<nigel> .. because it will be different depending on the algorithm that's used.
<nigel> .. Apple's general flash algorithm outperforms Harding in some specific ways.
<nigel> .. I could see there being multiple tracks, one using Harding, one the Apple open source algorithm,
<nigel> .. another some other algorithm.
<nigel> .. It's unlikely that anyone would ship all three.
<nigel> .. The algorithm would define it, not the transport format.
<nigel> q?
<nigel> EricC: Do we have a document that defines these things for our new algorithm?
<Eric> q-
<nigel> ack Eric
<nigel> Eric: If we do write up a Note or a Rec track document I don't think it needs to mention WebVMT.
<nigel> .. It is just another type of metadata track.
<jcraig> OSS project https://github.com/apple/VideoFlashingReduction/
<nigel> jcraig: Posting some links here. The above is an algorithm.
<nigel> .. There are also ePubs and PDFs that explain more, the first two links on the Whats New page:
<jcraig> https://developer.apple.com/accessibility/#whats-new
<nigel> -> https://developer.apple.com/accessibility/#whats-new What's New Page
<nigel> jcraig: The ePub embeds a video.
<nigel> q?
<nigel> ack cpn_
<nigel> cpn_: Coming back to the Note/Rec track question.
<nigel> .. If this does become something more widely adopted across implementations,
<jcraig> Those are also linked in Issue 512
<nigel> .. is there anything that prevents a Note from being promoted to the Rec track?
<nigel> .. Once there are multiple implementations you would want the benefits of a Rec track.
<nigel> jcraig: I can't think of anything that would prevent us from promoting a Note to a Rec
<nigel> Nigel: Are there any IPR considerations here?
<nigel> jcraig: Good question, I will have to get back to you on that one.
<nigel> .. We did an IPR review before I posted this issue and the algorithm and open source project.
<nigel> .. If it were within the TTWG it would be covered by the W3C patent policy.
<nigel> Pierre: Whatever is submitted to this group (by members) is subject to the IPR policy.
<nigel> .. There's an exclusion that starts with the FPWD.
<gkatsev> q?
<jcraig> ack p
<nigel> .. I've never heard that publishing a Note relaxes the constraints for a Rec track.
<nigel> .. At a high level, a WG Note is a WG decision, really. The consensus body and review will be a lot
<nigel> .. narrower than for a Rec track document.
<nigel> .. You'll get much less attention, and there will be a lot less overhead.
<jcraig> s/TTWG it would be covered by the W3C patent policy./TTWG I think it would be covered by the W3C patent policy, but I will take a note to follow up internally./
<nigel> .. We talked about IPR. Typically, what I've mostly seen in Notes is application of existing standards
<nigel> .. Something with its own applications, process and technology is typically better as a Rec track document.
<nigel> .. I don't have a strong opinion, just trying to answer the question.
<nigel> jcraig: Maybe the alternative is to take this as an incubator and not decide on Note or Rec track until
<nigel> .. later in the process.
<nigel> .. We can publish in WICG and bring into TTWG if that's the appropriate place for it to land.
<nigel> Pierre: That's true too.
<cpn_> Nigel: WICG isn't the only place to incubate, can do in WG or another CG
<cpn_> ... We have a broad charter in terms of applications of timed text
<cpn_> James: Easier for us to participate where other organisations have joined already
<cpn_> Nigel: Publishing a Note in a WG is a bit like an incubation
<cpn_> James: My typical process in WICG is write in a wiki page, hence a reason to prefer that. The other benefit is you get people who are just interested in the one topic, which helps with organising meetings
<cpn_> Nigel: Any final thoughts on this topic?
<cpn_> James: Thank you all
<nigel> SUMMARY: Apple to think about next steps for incubation
<cpn_> Nigel: In summary, Apple to think about next steps for incubation

[cookiecrook]

Minor editorial corrections in the official meeting minutes.

[cookiecrook]

I realized after the call we had a vocabulary mismatch due to my unfamiliarity with WebVMT. The term "mapping data" came up a few times, which I and at least one more person took as the general meaning: "associating data." It's now clear in the context of WebVMT that others meant literal cartography data. 😉

After the fact, it's clear WebVMT was referred to as an example only, not a proposed format. Hopefully this acknowledgement clears up some of the communication breakdown regarding whether this should be a Rec, Note, or Incubation referring to some other normative Rec.

[cookiecrook]

Follow-up on the HLS Docs:
https://developer.apple.com/documentation/http-live-streaming/hls-authoring-specification-for-apple-devices-appendixes#Dim-Flashing-Lights