Codec switching when remoting.

[markafoltz]

From #172 (comment) and the associated review thread, we identified a need for a streaming session to switch codecs; for example when a media element being remoted changes the type of its MSE source. Other use cases are if the source material changes resolution (which might require a more efficient codec) or if network conditions allow upgrading from lossy to lossless codecs.

@pthatcherg suggested we could have the receiver enumerate codecs at the start of the session and allow the sender to switch mid-session.

[markafoltz]

https://www.w3.org/2019/09/15-webscreens-minutes.html#x18

RESOLVED: We'll try 1. Receiver capabilities that allow the streaming sender to switch codecs, plus 2. An API on RemotePlayback that allow JS to query post-prompt for capabilities like the MediaCapabilities API does (issue #223)

Item #1 can be accommodated strictly within the OSP specification, but #2 will require spinning off an issue for the Remote Playback API and figuring out how that fits into the scope of work for it.

[anssiko]

ACTION: @mfoltzgoogle to add "needs research" on issue #223 for second point

via https://www.w3.org/2020/10/20-webscreens-minutes.html#a03

[markafoltz]

Based on the discussion linked above, it sounds like there is interest in exposing the capabilities of the receiving agent to script while remote playback is active, so that the site can customize the media that is sent (for example, choosing a more appropriate codec, color space, audio track, etc.) . That API would need to be defined first in the Remote Playback API, to determine if any changes are needed in the protocol.

Separately from the issue of exposing capabilities, if the site changes the media source during remote playback, the protocol supports this by sending a remote-playback-modify-request with the remote-playback-controls.source-url field set to a new value. However, we are not passing the extended-mime-type like we do with the remote-playback-start-request, so that is something that can be fixed in the spec.

If the remote playback session is remoting (i.e. sending encoded media, not a URL), then there isn't currently a way for the sender to offer new encodings during an existing session; only the receiver can make a request to modify the encodings sent during a session. As a workaround, the sender can terminate the current session, and start a new session by offering new encodings. I think this is good enough for now, but once we have a working implementation we can determine if this causes quality issues (i.e., a gap in playback before the new encoding starts to play out).