Proposal: <transcript> element inside <video> and <audio> for deafblind users

[accessabilly]

Transcripts of multimedia are essential for deafblind users

Deafblind multimedia users need everything in a machine-readable text format to be displayed via a screen reader in a refreshable braille device. Closed captions in videos are not usable text for them, because, though some modern braille readers can display the captions, the captions change too quickly to read them in real time as the video plays. Also, there is no easy way to access the captions separately from the video, even if the captions are in a text format.

A separate transcript to audio or video is the only way that multimedia content can be made accessible to deafblind users.
There are ways to create transcripts already, like placing transcript content into a separate markup containers after the multimedia content.

Problems

• Transcripts are not connected directly nor semantically to the <video> or <audio>.
• A connection via aria-describedby to the multimedia content is not usable for deafblind users, because it does not allow pausing or navigating the text in a screen reader. It makes the screen reader read the whole thing at once.
• A large transcript would need a skip link placed before it to make it possible for other users to skip the content.
• There should be an accessible way to show or hide the transcript for all users like there is for close captions (like through buttons in the UI of the multimedia element).

Why not allow a <transcript> element inside the <video> or <audio>?

To make this usable for all users we would need a button in the multimedia player to show/hide the transcript, something like there already is for close captions. This would ensure that there is always a semantic connection between the transcript and the multimedia content. A native HTML element would highlight the importance of a transcript for deafblind users who can't access multimedia in another way than text. Also search engines and their users could benefit from a semantically correct transcript.

References

• https://www.w3.org/WAI/WCAG21/Techniques/html/H96.html
• http://www.uiaccess.com/transcripts/transcripts_on_the_web.html

[pshaughn]

I am not an accessibility developer in particular, so take me with a grain of salt, but this reads to me like it might be easier to define and implement as a new kind for the track element, not a new element entirely. A transcript needs some of what's in kind=description and some of what's in kind=subtitles, so it's not already covered by existing track element concepts, but it otherwise seems to me track-like.

[accessabilly]

This might be true under the following conditions (I'm technically not too familiar with the <track> element):

• I assume, you can't access a <track> without searching for it in the dev tools or as a visual part of the video. So my point is: <transcript> should be open to everyone (even search engines) and not be hidden behind markup. If we can make <track kind="transcript"> open, this would be ok as well, I think.
• It should be possible that the referenced <track> file is a HTML document itself, possibly like an <iframe>. This referenced transcript document should be accessible (and secure) for all. If it would be part of the initial document (like a <section>), this would be much easier and more accessible. But maybe it’s possible that way.

[pshaughn]

I hadn't considered how the contents of the transcript would benefit from being able to use HTML markup in them. That would be a bigger change to the <track> element than I was thinking, particularly since <track> is a void element so the transcript couldn't go inline into the main HTML document that way. A new element does make more sense with that in mind.

[Yay295]

Instead of a new <transcript> element, maybe it could be a transcript attribute on a <track> element, which would be the ID of some other element on the page. So you could have a transcript in a regular paragraph element, give the paragraph an ID, and reference it with <track transcript="paragraph_id">.

Though I suppose that would require you to have a track element...

[accessabilly]

That sounds like a nice solution. But my extended point on the <transcript> take was to have a control in the UI of <audio> and <video> that toggles the visibility of the transcript content. And I don't want to place skip links before that content. I can't figure a possibility to show/hide an element via it's id though. I'm thinking of a native HTML solution strongly incorporated into multimedia players, so authors/developers don't have to build it by themselves and that forces them by design to think about the accessibility of their multimedia element.

[scottaohara]

if the end goal is a control in the UI, does this actually need to be an element? Not that I'm personally opposed to a specific element (though I have no concrete ideas on how this would be different than other existing elements from a visual or a11y standpoint). However, I do not think it should/need to be something that is a child of either media element. That seems like it could be very limiting in how authors would be able to render / position the transcript on their web pages.

If the primary point here is to standardize a control in native media players that can allow users to navigate to a transcript (same page / new page), and thus help some users not have to search around a page to see if there is a transcript or link to one, then that seems like a good end goal to me. Whether or not we need an element, I'm not so sure.

[accessabilly]

You're definitely right: the <transcript> as a child element of <audio> or <video> would result in a strong binding to its media element with styling limitations.

But the not so obvious benefit of a new element would be: we have an accessible role for this kind of UI component and not different namings for "transcript", depending on authors. In parallel to a <transcript>, there also should be something like a <div role="transcript" id="my-transcript"> , referenced like <transcript src="#my-transcript">, which would enable an in-page variant that could be designed freely. And stuff like <transcript src="/transcript.html"> for larger content as a dedicated document. But a child <transcript> would be a standardized solution that any web author/developer could use. And I think, if there is a dedicated element for that, this would raise awareness for devs.
And as you stated: the biggest benefit would be for users who depend on it: they could find it immediately, if it's referenced correctly.

[scottaohara]

There is no transcript role though, which makes this not only an ask for a new element, but an ask for platforms to create an implement a new role. ARIA would need to take that into consideration.

Seems to me this is all worth fleshing out more... But perhaps that should be done by creating an issue with open ui or open a discussion on the wicg?

[Malvoz]

As noted in WICG/proposals#45 (comment), there's already an existing proposal/spec for transcripts.

[accessabilly]

As there is an official proposal for a <transcript> element, I decided to close this issue. Sorry for the noise...