Multiple attachable/detachable SourceBuffers

[dsilhavy]

This issue is a follow-up on the discussion we had in the W3C Media and Entertainment IG calls. See also Additional Information below.

Overview

MSE based media players initialize the required SourceBuffer objects at the beginning of a media playback session. Currently, only one SourceBuffer per media type (e.g. audio,video) can be attached and used.

Description / Background / Use Cases

Server-guided ad-insertion

In the context of server-guided ad-insertion a media player is temporarily switching playback between different content/manifest files (switch from main content to an ad and back). In this case, segments of different content might not be aligned perfectly:

In the example above, segment 2 is not aligned with segment 1, both would overlap in the buffer. As a consequence, segment 1 needs to be played until the end before segment 2 can be appended to the SourceBuffer. In such cases, a virtual buffer can be maintained on a client, but appended segments there will not be parsed. In addition, the timing when to push from the virtual buffer to the main buffer can be tricky.

Suggested Solution

Having the possibility to maintain multiple SourceBuffers that can be detached and switched would make it easier for a media player to cover scenarios as the one described above. See also #359 for details on dynamically adding and removing SourceBuffers.

Additional Information

For related information, please refer to the meeting minutes of the W3C Media and Entertainment IG:

• Joint meeting W3C/SVTA
• MEIG Monthly Meeting - 2025/01/07

[jyavenard]

You may want to have a read of #357 which I believe cover the use case you describing in a different fashion (detach the MediaSource instead of the SourceBuffer).

Detaching a SourceBuffer (and not the MediaSource) would likely be a more complex enterprise from an implementation perspective.

[dsilhavy]

You may want to have a read of #357 which I believe cover the use case you describing in a different fashion (detach the MediaSource instead of the SourceBuffer).

Detaching a SourceBuffer (and not the MediaSource) would likely be a more complex enterprise from an implementation perspective.

Thanks I think you are right and this goes in the same direction. As you pointed out we would need to detach/attach the MediaSource instead of individual SourceBuffers.

[ntrrgc]

MSE based media players initialize the required SourceBuffer objects at the beginning of a media playback session. Currently, only one SourceBuffer per media type (e.g. audio,video) can be attached and used.

As far as I know this is not a spec limitation, this is a quality of implementation issue. Quoting the MSE spec:

https://www.w3.org/TR/media-source-2/#activesourcebuffers-attribute

Implementations MUST support at least 1 MediaSource object with the following configurations:

• A single SourceBuffer with 1 audio track and/or 1 video track.
• Two SourceBuffers with one handling a single audio track and the other handling a single video track.

MediaSource objects MUST support each of the configurations above, but they are only required to support one configuration at a time. Supporting multiple configurations at once or additional configurations is a quality of implementation issue.

If you have multiple tracks of the same type, that is meant to be handled with the {Audio,Video,Text}TrackList APIs: https://html.spec.whatwg.org/multipage/media.html#audiotracklist-and-videotracklist-objects

The track APIs allow you to have at any point any number of audio tracks and text tracks enabled, and up to one video track enabled. The MSE spec also addresses how this should be handled in MSE. Quoting the MSE spec again:

3.3 activeSourceBuffers attribute

Contains the subset of sourceBuffers that are providing the selected video track, the enabled audio track(s), and the "showing" or "hidden" text track(s).

[...]

Note: Section 3.15.5 Changes to selected/enabled track state describes how this attribute gets updated.

Unfortunately, if you look in Browser compatibility here https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement#browser_compatibility you'll see only WebKit ships AudioTrackList and VideoTrackList enabled without a flag.