In mobile browsers, seamlessly switch video (with audio) to audio-only on screen lock/when backgrounded

[nickrobillard]

Is your feature request related to a problem? Please describe.
(Apologies if this isn't flushed out enough to belong under Feature Request.)

I have a use case that I think is becoming very common on mobile devices. Consider video + audio media where audio is the priority - like a podcast, a DJ playing music, or playback of a music video. When the user puts the video out of view, rather than pausing video and audio (as is standard for all mobile browsers), it would be really great to seamlessly transition to audio only and continue playing. When the user puts the video element back into view, the video picks up at the same time as the audio. On the surface, this is what the native Spotify Android and iOS mobile apps have done for video podcasts (although they may just keep playing the video underneath). I haven't found a good way to achieve this in mobile browsers aside from some hacks, like playing audio in an <audio> tag and audio-less video in a <video> tag at the same time. This works quite well at first - it's not difficult to wire up custom controls and keep them both in sync (event handlers do a good job here). It's also nice that when the video is out of view and paused by the system, we can easily stop it from downloading and bandwidth is saved. But this hack quickly starts to fall apart the moment external controls or external viewing is used - like the notification panel Play/Pause actions (especially on iOS), picture in picture controls, or chromecast. The 2 separate media elements being played are unexpected and not handled well.

Describe the solution you'd like
I would like to consider using Hls.js to remove video on-the-fly and in a way that works with current mobile browser behavior and allows a video's audio to continue playing when the video element is put out of view, achieving a similar experience in a mobile browser that one has when playing a video podcast in a native Spotify mobile app. A basic approach could be to disable video (at the demuxing or the SourceBuffer stage for example). The video track is simply being enabled and disabled at the right time. Any approach taken would seem to require some kind of "onBeforeVideoPausedBySystem" event in order to remove the video "in time" for the audio to continue seamlessly. (Although this may not be reliable and may be the wrong approach.)

Describe alternatives you've considered
I realize this may be better solved by the browser (in Chromium, Webkit, etc) and may even require it (considering the browser behavior that pauses video when it goes out of view is not controllable and may not be directly observable from JS land). It could be a relatively simple <video> tag attribute/property that achieves "continue playing audio when system pauses video because it is out of view" and abstracted away from the implementation layer. That being said, there are details about behavior that one could want control over.

UPDATE: I see on-the-fly video to audio-only switching is covered a bit here - #163. Curious if things have progressed far enough to achieve this kind of thing.

[phillipseamore]

If your playlist contains an audio-only level this should be the default behaviour on iOS (since it's handled natively by Safari and not by hls.js).

However I think this might be of benefit in other situations. Being able to switch to an audio-only level with the Page Visibility API would save bandwidth for both the viewer and publisher. If that isn't possible, just being able to turn off video decoding when a page is hidden could save client resources like battery.

I seem to remember that hls.js would switch to audio-only levels in the past.

Quick mockup (this will not work with hls.js!!!):

document.addEventListener(visibilityChange, function(e) {
	if(document.hidden) {
		hls.audioOnly();
		// Change overlay on player with a message like "Now playing only audio"
	} else {
		hls.audioOnly(false);
	}
}, false);

hls.audioOnly() would switch to the audio-only level if availble, otherwise it would disable video decoding and switch to the lowest bitrate video.

[nickrobillard]

Thank you for the feedback!

If your playlist contains an audio-only level this should be the default behaviour on iOS (since it's handled natively by Safari and not by hls.js).

From my observations, iOS's audio-only switching behavior only automatically occurs in error or low bandwidth situations. I've tested this by using my own sources with the "Basic stream" examples here - https://developer.apple.com/streaming/examples/. I can see that when the video + audio stream is cut at the source, Safari switches to the audio-only stream (with a small audible gap while it is switching, as expected). This is my understanding of audio-only switching in general (be it native iOS HLS or Hls.js). It seems to exist to solve a different problem.

Page Visibility API

This does seem like a good road to go down. But there are some issues. In Android Chrome and iOS Safari (likely all browsers), the video "paused" event that occurs as a result of the video going out of view fires before document "visibilitychange" event.

log: videoPlayer paused, document.hidden = false
log: document "visibilitychange" event, document.hidden = true

So any switch action done at this time is late and results in a small gap of no audio. Depending on your use case, this may be considered ok - but I would like to achieve truly seamless switching. But much worse, using the "visibilitychange" event to do anything comes with the side effect of the browser pausing the video before you have a chance to do anything. Now you have to decide whether the audio needs to be resumed or not. This is a big problem because it is likely impossible to know if the "paused" event/state was triggered by an explicit user action or if it was an automatic pause due to video going out of view. So even if you are ok with a small gap in audio playback, you don't know if the videoPlayer.paused = true state resulted from an explicit user "Pause" action and you don't know if audio should be resumed or not. I'm fighting with the browser at this point, and that doesn't seem like a good idea. But I may have missed something here - any input is very welcomed.

Some code to illustrate:

videoPlayer.addEventListener('paused', (event) => {
  // Nothing in event arg allows us to differentiate an explicit user pause with an automatic one 
  // due to video being hidden.
});
document.addEventListener('visibilitychange', () => {
  if (document.hidden) {
    // Page is invisible.
    if (videoPlayer.paused) {
      videoPlayer.play();
      // This won't work because videoPlayer.paused will always be true. The video may have been
      // playing or paused before the page became invisible. We don't know. And because of "external"
      // controls like Picture in Picture, notification panel controls, airplay/chromecast, etc it's
      // a bad idea to try to keep track of our own internal playing state based on explicit user clicks.
    }
  } else {
    // Page is visible.
  }
});

I did find something interesting. iOS 14.4 Safari allows a video to be resumed after/while it is in the background. On the "visibilitychange" event, after the video is automatically paused due to being hidden, you can immediately call play() again. The video keeps playing its audio (Play/Pause buttons in iOS control center affect audio as expected) and when you bring the video back into view, you can see the video fast forward to catch up to the audio. This seems intentional by Apple. But this suffers from the same problem as just described - it is very difficult to know if the videoPlayer.paused = true state was from an explicit user interaction or due to the video element being hidden.

I like your idea of a new method like hls.audioOnly() because it doesn't seem to concern itself with the playing state. However, calling it from "visibilitychange" handler means that the video has already been automatically paused and introduces the problem described.

[phillipseamore]

The mobile behaviour has changed. In Safari it will play audio if I go to another tab but going out of Safari it now pauses but you can resume playing from the notification center (same on Android except it pauses when going to another tab). This might also be related to the recent addition of PiP on iOS. The old behaviour was that you could go out of Safari and it kept playing audio. Note that this requires the playsinline attribute on the video tag (which Apple's examples don't have).

Regarding the video being paused when out of view - I'm not able to reproduce that when moving between tabs but it does pause when going out of Safari. But I can reproduce playing a manually paused video (e.g. it has had user interaction) on visibilitychange.

The window blur event is sent before the video's pause when going out of Safari.

[phillipseamore]

We are getting way outside the discussion of hls.js - and I apologize for that.

But I'd like to add that using <audio> results in the correct behavior of continuing to play in background without interuption on iOS. Also if the video element is set with height and width = 0 (changing it from JS will not work, e.g. having a 640x360 video element and setting it to 0x0 at any time after page load will pause the video on moving out of Safari). I also tested MediaController with a muted video element and an audio element using the same playlist. A/V sync seems fine but it will pause both on moving out of Safari.

[robwalch]

Regarding the video being paused when out of view

hls.js will not pause media when the tab or browser is backgrounded - the browser may, and you developers can call pause too on visibilitychange. If it's the browser's policy to do that, that is a limitation you need to live with. Users can resume playback in the background from the lock screen media controls.

As for audio-only variants, hls.js removes them when using variants with audio and video. We don't support switching to audio-only variants, because MSE requires both buffers to be appended to, in order to grow a playable buffer. If we started to only append audio, the buffered range would not grow and playback would stall.

There could be a couple of ways to address that, but we haven't yet. That's the first order of business if we're going to even support switching to audio-only variants at all. If we did then you could use the cap-level controller to limit playback to the lowest level on visibilitychange.

For now, you might want to do just that to save on bandwidth. This is not something we would add to the cap-level controller as it's fairly simple to do yourself, and we don't want existing users to be surprised with a drop in quality the moment that the tab is foregrounded again.

[nickrobillard]

Thank you all. I appreciate the feedback.

The mobile behaviour has changed. In Safari it will play audio if I go to another tab but going out of Safari it now pauses but you can resume playing from the notification center (same on Android except it pauses when going to another tab).

You are right - I overlooked this iOS Safari behavior change.

Note that this requires the playsinline attribute on the video tag (which Apple's examples don't have).

Yes, playsinline attribute is required in iOS Safari to avoid the forced fullscreen behavior (iOS 10+). I've been using this in my testing.

The window blur event is sent before the video's pause when going out of Safari.

In Android Chrome, I am observing the order you mention - the video pause event comes after window blur. However, in iOS 14.4 Safari (using Simulator.app - I don't have a real device), I am consistently seeing the pause event before both blur and visibilitychange. So in iOS Safari, the "can't tell what triggered the pause" issue is a problem here. @phillipseamore Are you using a real iOS device? What's your iOS version?

iOS Safari Log:

20:59:15: videoPlayer pause
20:59:16: window blur
20:59:16: hidden

But I'd like to add that using <audio> results in the correct behavior of continuing to play in background without interuption on iOS.

User experience wise, this is the closest I have got to my "seamless" audio ideal. Playing audio and video tags separately appears to work well at first - but problems arise in iOS when lock screen & notification panel controls are used (I'll refer to these as "external controls"). While Android Chrome external controls affect/reflect the audio playback state as one would expect, iOS does not. iOS external controls appear "dead" initially and intermittently because the controls are actually cycling between the video and audio elements.

In iOS Safari:

• press Play button on my test player (separate audio and video elements begin playing) [✓]
• show home screen (audio continues playing) [✓]
• pull down notification panel - observe "Pause" button action [✓]
• press pause - audio continues playing, press play (repeats 2x) [x]
• press pause - audio stops [✓]
• press play - audio starts [✓]
• press pause - audio continues playing, press play, press pause - audio stops (repeat ∞) [x]

I tried to work with this behavior but to no avail. The remote controls are switching between the video and audio media elements and it is very difficult/impossible to handle both elements play/pause events and keep them in sync. Completely destroying the Hls player and removing the <video> element on visibilitychange hidden helps a little bit, but not initially (it's possible something related to the video player is not completely destroyed). Something I observed - the last element to have been played or paused is the initial element that the external controls Play/Pause action reflects/controls. In my case, I am calling play on my audio element, listening for audio play event, and then playing video to keep it in sync. This is why video is the initial element being controlled by the remote controls (and then it begins switching back and forth). Fyi, I'm using attributes "playsinline" and "muted" on the video tag and my video source stream has no audio channel.

I also tested MediaController with a muted video element and an audio element using the same playlist. A/V sync seems fine but it will pause both on moving out of Safari.

Yes, I had tested MediaController (and mediagroup attribute) as well and observed the exact same behavior you describe - both audio and video pause when moving out of iOS Safari.

As for audio-only variants, hls.js removes them when using variants with audio and video. We don't support switching to audio-only variants, because MSE requires both buffers to be appended to, in order to grow a playable buffer. If we started to only append audio, the buffered range would not grow and playback would stall.

There could be a couple of ways to address that, but we haven't yet. That's the first order of business if we're going to even support switching to audio-only variants at all. If we did then you could use the cap-level controller to limit playback to the lowest level on visibilitychange.

This is good to know and answers my biggest question. My goal is to solve this at the lowest level possible and source buffers seemed to fit that (but not currently possible, as you mentioned).

The 2 approaches

1. Separate audio and video elements
   Audio element has an audio-only source. Video element has video-only source. One must manually keep them in sync (I use audio currentTime as my base line, and keep video in sync with it). The major advantage is that we don't fight browser behavior, for the most part. The video is automatically paused when going out of the browser (a good thing) - and we can even stop downloading video segments to save bandwidth. (Issue with external iOS controls is a problem here.)
2. One video element that switches to audio-only
   a. Removing video frames at the source buffer stage in theory could achieve a seamless switchover, but bandwidth saving would not be possible since the audio + video source would continue to be used.
   b. Alternatively, switching to a secondary audio-only source defined in the master m3u8 would require a slight seam in the audio, but it would achieve bandwidth saving.
   For both 2a and 2b, we seem to be stuck using visibilitychange (or blur) events to do the switch. The problem is that you are already in a paused state and deciding if you should resume playback after switching is very difficult to do (I don't see a way) and is really a hack in my mind.

Assuming there are no other options for triggering the switch (a big assumption), "the right" solution could be to propose an addition to the HTMLMediaElement (or HLS?) spec and introduce policies that are set when initializing the media element (or in HLS master playlist). Obviously this is completely outside the realm of the Hls.js library (and even if accepted could take years to be implemented and fully supported in modern browsers).

I appreciate all of the insight shared and I am still hoping to make something work within the realm of Hls.js, even if it means the audio-only switch isn't perfectly seamless.

[kontrollanten]

Thanks for doing this deep investigation, @nickrobillard

For both 2a and 2b, we seem to be stuck using visibilitychange (or blur) events to do the switch. The problem is that you are already in a paused state and deciding if you should resume playback after switching is very difficult to do (I don't see a way) and is really a hack in my mind.

Shouldn't hls.js be able to determine whether it's a user or the browser that pauses the screen? Like adding a hasUserPaused attribute that gets updated each time the user is pausing the stream (or the stream ends).

[robwalch]

Not a Contribution

Shouldn't hls.js be able to determine whether it's a user or the browser that pauses the screen?

It should not. play/pause functionality is implemented via HTMLMediaElement, not HLS.js. This responsibility falls on the application using HLS.js.

[zhengcling]

2. One video element that switches to audio-only
a. Removing video frames at the source buffer stage in theory could achieve a seamless switchover, but bandwidth saving would not be possible since the audio + video source would continue to be used.
b. Alternatively, switching to a secondary audio-only source defined in the master m3u8 would require a slight seam in the audio, but it would achieve bandwidth saving.
For both 2a and 2b, we seem to be stuck using visibilitychange (or blur) events to do the switch. The problem is that you are already in a paused state and deciding if you should resume playback after switching is very difficult to do (I don't see a way) and is really a hack in my mind.

@nickrobillard How to switch to audio and play automatically when IOS locks the screen？

[Chocobozzz]

Hi,

We're interested in providing a custom "Audio only" "option in our player, that can only be selected manually by users (so the auto quality system doesn't choose it).

I think we can tweak our player to destroy the old hls.js object, and re create a new one with just the audio level in the manifest so hls.js only plays the audio. The user experience would not be very good since there would be a player freeze on "Audio only" selection.

We'd prefer to have a built-in system in hls.js, even if it seems to be complicated with MSE: #4881 (comment)

@robwalch (and other hls.js contributors): Framasoft, the non-profit I work for can try to fund this feature. Is this something that could help to implement this use case? We can discuss this by email if you prefer (available in my github profile).