PTS gaps in audio input

[hariszukanovic]

• What version of Shaka Player are you using?
  Shaka Player v2.0.5-debug

I would like your opinion and comments on the subject of having PTS gaps in input audio track which then propagate through the dash packager to the player.
Please take a look at this issue at shaka packager where we tried to address the problem in the packager itself.
shaka-project/shaka-packager#196

A couple of basic questions I believe might help clear up my own understanding of the situation. I would appreciate comments...

Should this kind the gaps be handled by player?
Should this be handled/treated by transcoder or packager in any way?
Is this kind of input (with audio gaps) at all valid?
Observed behaviour

When no attempt to handle the gaps is done at packager, player stops after about 30-60 seconds.
Packager is correctly adding gaps in the SegmentTimeline by inserting S entries

When in the packager, I ignore the gaps and collect audio samples as they come, player works but audio/video slide out of sync after around 5-10 minutes. Packager is not adding any gaps in the SegmentTimeline. They repeat "perfectly" with the same duration

When in the packager, I "prolong" the duration of the audio samples that have gaps after them, the packager adds correctly entries in SegmentTimeline the player with different durations. The player works but audio/video slide out of sync after around 5-10 minutes

[TheModMaker]

Gaps should usually be handled on the packager (or transcoder) side. They are the ones producing the content and they should be the ones to produce valid, playable, content. However, we are currently working on adding gap jumping to the player because we see a lot of content with gaps in it (especially with live streams). See #180 and #555.

Having gaps is probably valid in terms of the spec, but it makes playing it in the browser difficult. Because we use MSE, we just give data to the browser and it then it creates "buffered ranges" based on what data it has. If there are gaps, the browser doesn't know whether that content will be added later. When the browser plays to the gap, it will stop playback, waiting for the content.

But, since we have the manifest, we know that the content will never be added. So we could see the gap and jump it automatically. And that is what we are working on.

[TheModMaker]

I should also note that small gaps, the browser will jump automatically. What is small depends on the browser. Firefox will jump a gap of 200ms while Safari will only jump 22ms. Currently, we won't buffer past a gap of 20ms because that is the minimum amount that will be jumped. This provides a consistent cross-browser experience and makes bad content more obvious.

[hariszukanovic]

I will try other browsers... a bit. I used so far only Chrome.

I have tried to create a fix in the shaka packager, specifying in the dash manifest that the segment duration is longer when there are gaps in any of it's contained audio samples. But, the pts gaps still do exist in the audio elementary stream itself.

In Chrome, would that explain the fact that on player reload (stream restart) the player successfully starts perfectly synced audio/video but they slide apart after a short while as those gaps present in elementary audio stream accumulate?

[hariszukanovic]

@TheModMaker, does Chrome skip gaps at all?
In my case, using Chrome I can see desync very soon.
Using Firefox there seems to be no desync at all.

[TheModMaker]

This may be a bug in Chrome. What I think may be happening is that Chrome is not using the times in the container (MP4) to determine the frame times.

What happens is we append the data to the media source. The times for each frame is encoded both in the container and in the elementary stream. However, in the stream, it only encodes the duration of the frame, not the absolute time. So as Chrome is decoding frames, it is using the duration to determine when the next frame is and this leads to the audio lagging behind since the duration doesn't account for the gaps.

Can you generate an on-demand version of the content? If you generate an on-demand stream, the packager will generate a single MP4 file that also is playable by itself. This will allow you to play the file by itself in Chrome to see if it get desync there too.

[hariszukanovic]

I do have a .ts recording of this live stream that I can convert to an on-demand version and could try that.
The .ts recording and the live transport stream as well are perfectly playable by itself in VLC for example

[TheModMaker]

@hariszukanovic Any update on this? Were you able to generate the on-demand content?

[joeyparrish]

@hariszukanovic, since we haven't heard back from you in a few weeks, I'm closing this issue due to inactivity. We can reopen it at any time if you wish.