Media Chrome state management with no src

[cjpillsbury]

The Problem

Currently, some Media Chrome controls will have unexpected behavior, including throwing errors, if either:

• The src has not been set
• The media has not been sufficiently loaded (with the corresponding expected readyState)

Some concrete examples:

1. When the src has not been set and the media's metadata hasn't been loaded, if you attempt to enter picture in picture mode (e.g. via the <media-pip-button/>), an error will be thrown.
2. When the src has not been set, if you attempt to play (e.g. via the <media-play-button>), the slotted media element will dispatch a waiting event, suggesting the media is loading, and a play event, resulting in the <media-play-button/> showing the pause icon.
   1. If preload="metadata" or preload="auto", playback will begin once the src is set
   2. If preload="none", the play() promise will be rejected and it will still look like the media is loading (and the <media-play-button/> will still show the pause icon)
      1. If you attempt to play again, (by "pausing" and replaying via the <media-play-button/>), playback will begin.
      2. If you eventually load() the media, the slotted media element will go back to a paused state.
3. When the src has not been set and the media's metadata hasn't been loaded, if you attempt to seek via the <media-time-range/>, slotted media element's currentTime will remain unchanged, however the thumb position will look as if it has until a timeupdate or loadedmetadata event is dispatched.

Possible Broad Strokes Solutions (not mutually exclusive)

1. "Gate" events in <media-controller/>'s state management based on the slotted media element's src state and readyState
2. Reprovide state from <media-controller/> (potentially asynchronously) even if there is no corresponding event under conditions where we know the state will not update (for e.g. the <media-time-range/> case, described above).
3. Add src and readyState or something derivable from them as attributes/properties of <media-controller/>'s state management and provide them to the relevant controls.
   1. This can also be used to "bake in" disabled state via e.g. CSS, or allow implementors/users of Media Chrome to apply these styles themselves (like our solution for "availability" states)

Recommendation

I believe we should do all three, but not "bake in" any disabling CSS (at least for now, since we can decide to add that later).

For (1), we should minimally guarantee that state request events that throw errors should be "gated". Also, (1) will already need to be keeping track of the state described in (3) in <media-controller/>.

For (2), I think we should minimally guarantee that our controllers accurately reflect the media's state. Our general control/display element ("Media State Receivers") architecture is based on having their state provided to them from the "outside" (via the <media-controller/>'s media state management). The only reason this is an issue for things like the <media-time-range/> is because we're relying on an inherently stateful element "under the hood" (an <input type="range"/>), so we need to effectively "renotify" the element that the state is exactly what it was before interaction. Also, as with (1), (2) will already need to be keeping track of some of the state described in (3) in <media-controller/>.

For (3), aside from its partial relationships to (1) and (2), providing these new state attributes will allow us or implementors to disable controls when their actions will not succeed or will result in different behaviors based on order of operations.

There are at least two outstanding details that will need to be decided if we implement (3).

1. Should the control/display elements be responsible for "auto-disabling"? (I lean towards no for greater versatility and consistency with our availability states usage)
2. Should the state/attributes's key/value pairs correspond to src and readyState directly, or do we want to model them as something more specific (e.g. media-has-src, media-has-metadata)? (I lean towards the former, but only slightly for versatility/future usage/less presumption on was the state would/could be used. Given the fact that implementors will likely use these attributes for CSS selectors, however, I can see the merit of clearer, derived values like the example I provided in the latter.)

[heff]

we should minimally guarantee that state request events that throw errors should be "gated"

I worry some about gating and putting media controller in a place where it's hiding some operations from the media element. It feels like that's taking media controller to a new level of ownership of the system, that we should consider the consequences. Like, building a custom media element alongside media chrome could get annoying if you have to dig into media controller to understand why an operation isn't getting to your element. Yes, we could console log some helpful things, but then media controller is essentially taking over responsibility of error reporting from the media element, and something about that feels not great.

Put another way, if you want to build a button that throws an error on the media element, is it media-controllers job to stop you? Or is it just media controller's job to give you the tools to avoid the error? My S1 is the latter, but interested to hear thoughts on that.

This can also be used to "bake in" disabled state via e.g. CSS, or allow implementors/users of Media Chrome to apply these styles themselves (like our solution for "availability" states)

This is where my mind mind went first. Somewhere between media states and availability states. Before a certain ready state, Chrome's buttons will be disabled.

Should the control/display elements be responsible for "auto-disabling"? (I lean towards no for greater versatility and consistency with our availability states usage

I'm not sure what you mean by this. If a play button disabled itself when it was told playback was not available, is that against what you're saying here?

...src and readyState ... media-has-src, media-has-metadata ... I lean towards the former

Yeah, this is an interesting one. There's a nice thing about readyState (and CSS) that we can target greater than 0 or less than X on a media-ready-state attribute. So I like that answer, despite the less readable naming.

Can't we basically solve all these issues withe a media-ready-state attr and have all controls disable themselves if the value is zero?

[cjpillsbury]

Taking a half step back, what are the real world use cases of "The src has not been set"?

The most likely case I'm hearing about are different variations of folks wanting to be able to load a page for some live media, but loading it before the source is known/available for playback. This keeps coming up in various permutations for Mux customers and partners, at least. Making the devex smoother for them sounds like a value add (and likely would be true outside of Mux as well)

[heff]

[Talked with Christian a little more IRL today]

I appreciate the whole scope that's been laid out here, and now I think it's worth breaking down and prioritizing the specific user problems/scenarios we want to solve right now. It's tempting to just solve all these problems, but each can come with added complexity that I'm not clear on if it's really worth it yet.

1. User creates player before a live stream is available/ready -> Viewer does what?
   We know this one happens with Mux users a lot today, right?
   This one feels complex, like there there's possibly a lot of different angles we can come at it. Including from the mux-video media element.

2. Viewer clicks PIP button before the media is ready -> nothing happens for the viewer
   Is there a clearer use case around this? I think the answer is no, it's just whenever preload=none, right?
   This one I wonder if media-controller could just check the readyState before calling the API, rather than needing to propagate state. That plus some kind of manual, global disabling of controls like described with the attribute on media controller.

3. Viewer seeks before media is ready -> nothing happens for viewer, time-range reflects new time incorrectly
   Same as the previous
   This one I'm skeptical about solving for right now, if it were the only one at least. I could see pushing in the direction of a custom component rather than built-in to solve this. Or maybe just media-controller listens/catches the error that thrown and resends state.

4. What am I missing?

[cjpillsbury]

I believe that sounds right in terms of user/viewer problems/scenarios. I do generally worry about approaching issues in a "one off" manner per scenario (rather than thinking about "under the hood" architecture in less "ad hoc" ways), but I get wanting to be able to prioritize based on use case/need. At least with respect to Mux Player goals/needs, (1) & (2) are high priorities but (3) probably is not.

I'd also treat "no way to easily disable all or particular controls" as a separate issue, rather than as a solution to (2), as that will also be a priority Mux Player.

[heff]

I made a quick PR exploring the PiP error case specifically. Interested in your feedback on that.

[heff]

I do generally worry about approaching issues in a "one off" manner per scenario (rather than thinking about "under the hood" architecture in less "ad hoc" ways)

I'm not explaining myself well but it's less about solving these sequentially and more about getting clear on the value of solving each of these, and avoiding added complexity if it's not bringing real value with it.

If "tell users don't use preload=none when not setting the source" is ultimately an acceptable alternative to one where media controller is now "keeping track of some of the state described", that's a win.

[gkatsev]

2. If preload="none", the play() promise will be rejected and it will still look like the media is loading (and the <media-play-button/> will still show the pause icon)

This seems like something we should be able to solve regardless of the rest of this discussion. We should be able to handle the play promise being rejected and then resetting the state appropriately.

[cjpillsbury]

agree.

[heff]

Doing more digging into the empty src details

• Context: No source, preload=none, call play(), then add src =>

  • On Safari it will error the play promise but not update the paused state
  • On Chrome it will do nothing
    • If you then load(), it will abort the original promise but not update the paused state
    • If you instead play() again, it will resolve both the old and new play promises.
  • On Firefox it will successfully load and play the video (woo!)

• Context: No source, preload=metadata, call play(), then add src =>

  • On Safari it will error the play promise but successfully play (woo?)
  • On Chrome it will successfully play (woo!)
  • On Firefox it will successfully play (woo!)

There's multiple chrome bugs and a safari bug in there that should be reported. Good job Firefox!

I don't think the preload=none issues are media-chrome's job to solve. I think we:

1. Report the bugs
2. Tell media chrome users "don't use preload="none" with delayed source setting, browsers have bugs".
3. If this specifically is a priority, let's solve it at the media element level. e.g. mux-video. And make it so the media element behaves like Firefox in all browsers (I think we can assume that's the desired behavior). At the same time we might want to just solve for the weird Safari play promise failure when preload=metadata.

Assuming that detail is solved in one or all of those ways, we can then focus on the specific workflow of "Player is created before the live stream is ready, and viewer is sitting there waiting and probably clicking everything they can".

[gkatsev]

What's the muted state? Because that probably makes a difference in how the browsers handle what happens when the source gets set after play is called.

For iPhones and iPads, it's not possible to set preload to a value other than none (i.e., Safari ignores the set value and assumes none)

[heff]

Good callout. I didn't think about it at the time but in all the testing it was muted. Might be worth testing unmuted, but also media chrome is kind of in a unique place because all play() calls should be preceded by a user click, right? Not sure how autoplay falls into all of this.

For iPhones and iPads, it's not possible to set preload to a value other than none

Also good callout. Interestingly though, I just tested on my iPhone and preload=metadata works differently than none, just like on desktop -- the promise errors but the video does play. The readyState does not go above 1 though, so they're half ignoring preload.

---

Update: Realizing that readyState 1 is HAVE_METADATA. Is my setup unique somehow or is our understanding of iOS preload support dated?

[gkatsev]

Interesting. Maybe they did change things? Not sure.
I did just remember that it'll preload if autoplay is enabled or play() is called.

[cjpillsbury]

I don't think the preload=none issues are media-chrome's job to solve. I think we:

1. Report the bugs

Do we know if these are "bugs" or how they are (potentially under) specified in the relevant specs? I'm going to do a bit of digging to see what I find. If they are non-compliant with the specs, they're definitely bugs and should be treated as such. If they aren't, sounds like a good discussion for FOMS?

[heff]

Do we know if these are "bugs" or how they are (potentially under) specified in the relevant specs?

Yeah maybe it's a gap in the spec, but there's unexpected behavior that's inconsistent with other browsers, so that's worth reporting. If it hasn't already been reported it's a good sign this isn't a big issue in practice. The preload=none case specifically.