Removing VTT Files From HLS Manifests

[joncameron]

Description

One point of investigation in fixing the double caption issue in Apple's media player is removing the VTT files from the HLS manifests we generate. If we're only going to be using file references outside of the context (like the player putting everything together from the IIIF manifest), we could instead be supplying the text tracks to the native player outside of the HLS manifest. But, Video.JS may not give us the tools to interact with the native player in a more granular way.

Done Looks Like

• Investigation into removing VTT file references from HLS manifest

Questions

• Will removing this fix the double caption display issue? ([BUG] Captions are showing up twice on screen on iPad, macOS Safari, iPhone #5650)
• Will multiple captions show up in native player? ([BUG] When multiple captions exist, only first caption is showing up in the iOS native player samvera-labs/ramp#405)
• Will this cause captions to not be available in the native player?
• Is changing CSS for the native player to hide overlay captions there an option?
• Does this solution work for embedded player?

[joncameron]

Is changing CSS for the native player to hide overlay captions there an option?

[Dananji]

1. Sample IIIF Manifest with an HLS stream without captions from avalon-dev with a supplementing annotation with the following Video.js options, in Chrome, iPad iOS 17:

...
html5: {
  nativeTextTracks: IS_MOBILE && !IS_ANDROID // evaluates to `true` for iPads
}

Captions are only available in the IIIF Manifest as supplementing annotations with an HLS stream media file

	Ramp player	Native player
First Caption Selected
Second Caption Selected

Builds an empty CC option into captions even when there is no captions attached to the HLS manifest.

---

2. Cookbook captions manifest with the same Video.js options;

Captions are only available in the IIIF Manifest as supplementing annotations with an .mp4 media file

Ramp player	Native player

---

3. Avalon IIIF Manifest from avalon-dev with the same Video.js options;

Identical captions are available in both HLS manfiest and IIIF Manifest

	Ramp player	Native player
First Caption Selected
Second Caption Selected

Builds the captions option from HLS manifest, but doesn't populate text overlay for the captions when this option is selected.

4. Same Avalon manifest with the following Video.js options;

...
html5: {
  nativeTextTracks: IS_MOBILE && !isStream && !IS_ANDROID // evaluates to `false` for Avalon manifests in iPads
},

Identical captions are available in both HLS manfiest and IIIF Manifest

Ramp player	Native player

Summary of the observations:

• Native player in iOS is able to display captions build by the Video.js instance in Ramp using supplementing annotations from the IIIF Manifest.
• No duplication of the captions were observed when using the Video.js option nativeTextTracks: IS_MOBILE && !IS_ANDROID. In the both IIIF manifests in 1 & 3, there was a captions option built into the captions menu which did not populate text track overlay on top of the video in both Ramp and native player instances. And from a first glance these empty options seem to be populated from the HLS manifest captions.
• It seems that the duplication of the text overlay for captions can be avoided by setting nativeTextTracks: true for HLS resources, which we only enabled for non-stream media (e.g. cookbook captions recipe)

Conclusion:

• Ramp can rely solely on captions provided only in the IIIF Manifest, since the tracks build by Video.js is picked up by the native player in iOS context.

[Dananji]

For the embedded player, it seems reversing the changes in #5688 would make it possible to feed captions into the Video.js instance instead of using captions generated by the HLS manifest.

[Dananji]

After a conversation with Emily; came to the conclusion that, presenting captions in the IIIF Manifest is the best approach at the moment. The reasoning behind this decision are;

• Embedding captions only in the HLS manifest would be useful if Avalon intended to share media items as raw files, which is not a use-case. Avalon expects to share resources via IIIF manifests and within embedded player.
• In embedded player context, Avalon can build the player instance with captions provided via track HTML elements to the Video.js instance instead of relying on HLS captions.
• At the moment there is no way to indicate the presence of embedded captions in the media resource in the IIIF Presentation 3.0 API. Therefore, IIIF manifests are not able to convey the availability of embedded captions. The only way to access the captions presented in this manners (i.e. in the HLS manifest) would be to paint the Canvas in a IIIF viewer, which supports HLS. To evaluate HLS support in other IIIF A/V viewers listed in the IIIF Viewer Matrix, I tested one of the public items in avalon-dev in these viewers;
  • UV: loads the manifest into the viewer, but does not support streaming
  • Mirador: loads the manifest into the viewer, but does not support streaming
  • Clover: loads all Annotations with supplementing motivation from the IIIF Manifest + HLS captions into the player's captions menu. In this particular item, the same file is used as a caption and a transcript, thus populating the captions menu with the same file 3 times.

[elynema]

If we utilize IIIF manifest captions only, we do lose the caption settings options in iPads and iPhones.

If we put the captions only in HLS manifest, not in IIIF manifest, there's no indication in IIIF manifest that there are captions unless the canvas is painted in a viewer that supports HLS. So for this reason, we prefer IIIF manifest captions now.

[elynema]

@joncameron @Dananji Should we proceed with a new issue to actually remove the VTT files from HLS manifests? Do we need further discussion?

[Dananji]

There are still some issues with captions in iOS context. Related new issue with details: samvera-labs/ramp#451

Testing note: not ready to be tested on iOS devices yet.

[Dananji]

One more issue related to caption selection: samvera-labs/ramp#452