Found a gap of size with otherwise working input stream

[hariszukanovic]

I am looking for an advice as I am not sure what might be wrong here...

Transcoder is ffmpeg and it's input a satellite transport stream. It produces transcoded transport stream output.
In the audio stream in this TS, I have verified that the PTS of audio packets are not aligned to what the sample rate is, 48000. Each packet is 1920 samples in length but every now and then, and always at the boundary of interleaving the video and audio streams in the TS, I see the not expected PTS
For example the 1290174 should never appear as packets have length of 1920:
...
pkt_pts=1281840
pkt_pts=1283760
pkt_pts=1285680
pkt_pts=1290174
pkt_pts=1292094
...
I attached a longer sample of PTSes for this audio stream. All PTSes are only 1 audio elementary stream.

On each such not expected PTS... and corresponding "Found a gap of size" warning, shaka packager inserts a new S entry in SegmentTimeline and my player dash.js actually freezes on each such warning. That should be expected behavior... I think.

Now,
This kind of input stream seems to work in VLC, work with ffmpeg segmenting into HLS chunks.
I have tried to run it through a couple of mpegts validators and I see no apparent errors.

I modified the shaka to ignore the "Found a gap of size" in packager/mpd/base/mpd_builder.cc and return true from bool Representation::IsContiguous()

When ignoring "Found a gap of size", it seems to actually work fine, but I would like to understand exactly why it works.

pts.txt

What do you think?
Any advice?

[hariszukanovic]

After couple of hours, the stream does not work all that well... and the player shows standard signs of instability.
I have seen the player desync audio/video and intermittently begin buffering and stop playing.
After I reload the player it starts to play well again.

[hariszukanovic]

I have verified the pkt_pts anomaly in the input transport stream as well. It is a digital satellite stream.
I guess, that stream is not perfect but it does work very well in VLC and ffmpeg does not complaint while transcoding it.
If you have any thoughts please share :)

[kqyang]

I read through your attached pts.txt. All the gaps seem like very small. It does not look like reset, but timestamp adjustment. Have you tried shaka-player? Is it able to handle the stream smoothly?

It seems to be happening every ~90 seconds. Can you save the VLC output - 5 minutes should be sufficient? It will be great if you can save the packager output for the same time period so we can compare packager and vlc outputs. Don't know if VLC is doing anything special to compensate for the gap.

[hariszukanovic]

That gaps indeed are always rather small. Mostly between 2ms and 6ms.
What shaka packager is doing in fact seems very reasonable sinces these gap do exist...
I will try to capture and send the material.

[hariszukanovic]

Some more info:
At first, after the packaging is just started and I suppose while the for audio track is still relatively short the stream seems to work. Already after several minutes I can see that the player starts the stream and very often but not always after 10-20 seconds it stalls showing a "loading" animation.
Shaka player shows the same behaviour as well, in addition if it does not stall after a while the audio and video go out of sync.

[hariszukanovic]

Captured material
https://drive.google.com/file/d/0BwkkojiD24s0UWtHNGtBYmxmb2c/view?usp=sharing
https://drive.google.com/file/d/0BwkkojiD24s0RF9wWVV4a0xVbU0/view?usp=sharing

[kqyang]

Thanks for the data. It does not look like VLC is doing anything special. There are still gaps in the stream. So I think VLC/FFmpeg is able to tolerate such small gaps when playing.

For DASH, I prefer the gaps to be handled in the player side. I heard that Shaka Player plan to support the gaps; I don't know their detail timeline though.

That said, for such small gaps, it would be fine with me to compensate for it in shaka packager, e.g. by increasing the duration for the frames before the gap. We don't have resource to work on it at this moment though.

[hariszukanovic]

I already tried totally ignoring the gaps by making the Representation::IsContiguous() always return true on such gaps, but in the end it did not work. I trust I drove audio/video out of sync by doing so, and from what it seems the audio/video became out of sync in the player as soon as the ignored gaps added up to a significant duration.

In the transcoder before shaka, I also tried to resample the audio and in fact it worked perfectly with regards to shaka package process (no more gaps detected), but the resampling itself was very much audible in the player exactly at the gap locations.
I am also reluctant to interfering with the contents of PES while packaging. I would prefer to understand why these gaps exist in the first place, while VLC seems to be playing the stream without any problems.

Regarding the consequent pileup of entries in the SegmentTimeline in manifest due to the gaps, I have had a similar problem using ffmpeg dash packager and dash.js player which was unable to play when the list became too long. With ffmpeg, it seemed that this problem could be mitigated by limiting the SegmentTimeline entry list by using window_size which I set to 6.

window_size: "number of segments kept in the manifest"

If we assume that shaka packager is in fact 100% correct when adding those SegmentTimeline entries due to gaps, but the player could be having problems with the pileup of those...
This limit seems to be 180 by default in shaka packager.
Would an option to limit that number or change the default make any sense for shaka packager?

[kqyang]

I already tried totally ignoring the gaps by making the Representation::IsContiguous() always return true on such gaps, but in the end it did not work.

It wouldn't work because the actual media timeline is not changed. One possible method, other than player change, that could work is to increase the frame duration here if there is a gap: https://github.com/google/shaka-packager/blob/master/packager/media/formats/mp2t/es_parser_adts.cc#L166

With ffmpeg, it seemed that this problem could be mitigated by limiting the SegmentTimeline entry list by using window_size which I set to 6.

There is a similar parameter in shaka-packager: time_shift_buffer_depth, which is set to 1800 seconds currently. Feel free to experiment with it, though it is hard to believe it would make a difference.

[hariszukanovic]

Setting time_shift_buffer_depth to 60 does not help. The players , both shaka and dash.js behave the same

[hariszukanovic]

I tried increasing duration at es_parser_adts.cc#L166..
and, now in the manifest I get different varying durations of segments when gaps are present in them...

SegmentTimeline
            S t="1080240" d="901495"
            S t="1981735" d="900480" r="2"
            S t="4683175" d="900292"
            S t="5583467" d="900480" r="1"
            S t="7384427" d="900296"
...

the player seems to work flawlessly, but the audio and video still slide apart into desync after a while, I would say exactly like when ignoring the gaps all together...

I wonder if you meant I need to adjust duration of the dash segment or adjust the duration of the elementary audio stream packet as well?

I attached the patch I have for now.
Can you please comment?

sample_duration_from_pts_diff.txt

[kqyang]

Yes, that is exactly what I meant, though I realized later it might not work as audio also encodes the duration in the frame.

the player seems to work flawlessly, but the audio and video still slide apart into desync after a while.

Are audio and video back to sync if you reload the player to the live edge?

Have you filed the bug to DASH-IF player or Shaka Player? It would be better to solve the "gap" problem in the player.

[hariszukanovic]

No, audio and video remain out of sync when I reload the player. Also, after quite some time the player cannot start any more. I believe the audio and video have gone out of sync so much at that point that the player gets confused.
I will file a report to both players today and ask for opinion

[kqyang]

No, audio and video remain out of sync when I reload the player.

I had expected it to be in sync again if reloading the player. In comment 2, you mentioned that the "After I reload the player it starts to play well again." So reloading the player can recover the sync problem with the original shaka packager?

I will file a report to both players today and ask for opinion

Thanks for filing the issues. It will be great if you can reference the issues here (I saw the issue you filed to dash-if player, but I don't see the shaka player issue.

[hariszukanovic]

After reverify-ing...
I have tested giving some "run-time" in the packager (more than 1h on each test)

..and you are right!
I must have accidentally exchanged different test results.

So, in both cases, the audio video desync is manifested in a way that audio precedes video. The video is late.

Testing player reload:

1. patched to prolong the duration of the sample just before the gap:
   On player reload, both Dash.js and Shaka, it does get back into sync and continues to work but always slides out of sync again after a short while. A couple of those gaps seem to quickly add up to a visible desync in the player.

2. no changes of mine, shaka master branch:
   On player reload, both Dash.js and Shaka, it does get back into sync, but then it works well for 10-30 seconds before it just stops. No errors visible in player console.

I attached the manifests of both test cases.
manifest_sample_duration_patched.mpd.txt
manifest_shaka_master.mpd.txt

Please let me have you comments. Thanx
Also, I asked for opinion in a shaka player issue
shaka-project/shaka-player#707

[hariszukanovic]

When I record the transport stream... analyze with ffprobe the frames...
This particular recording is 7 hours long.
It is very strange how the pts for audio and video of last packets recorded are so close... even though we have those gaps and no overlaps at all.

In this recording case, I counted the gaps to 889 and avg. gap size of 2195 which should equal to 1951355 in difference. The difference between pts of last packets is 1937.

Am I missing something?

last packets pts
==> video.frames.csv <==
2287269877

==> audio.frames.csv <==
2287267940

[kqyang]

It is very strange how the pts for audio and video of last packets recorded are so close... even though we have those gaps and no overlaps at all.

Why is it strange? Isn't that expected? Isn't the existence of the gaps to align the audio and video streams? It is weird that your transcoder uses gaps to align A/V streams though.

[hariszukanovic]

In this case, the transcoder is not mine, and it is unavailable to me. I only get to see it's transport stream output :( ..so I have no information about how it is actually setup. In this deployment we have around 15 channels with gaps and other 50 channels have no gaps at all.
I will try to analyze other deployments we have to see how those inputs behave...

Some thoughts:
"kRoundingErrorGrace" setting (in code) that "allows/ignores" a gap limited to only 5..., by it's name as well, also suggests that gaps might exist due to some kind of "small error" and that they are not really expected.

In fact, I admit I am quite unsure about why those gaps might exist there in the first place...
My logic would assume that there must be missing audio packets if there is a "significant" gap in pts. I do not know how it might be possible to verify this though... and gap durations do not correspond to audio packet size of 1920.

For a long time we have been using HLS with .ts chunk with muxed 1 audio and 1 video in those chunks. And we have never noticed these problems. In fact, somehow the same input stream works fine when I package it in this way with ffmpeg.
I assume the HLS players can somehow handle these gaps in the same way as the VLC handles (plays) this input transport stream without problems, but that might be a bigger problem for dash player which handles separately audio and video streams.

[hariszukanovic]

Just a note:
The audio/video desync does not happen in Firefox. Only Chrome.
Seems to be unrelated to the player but browser as it behaves exactly the same on both Shaka and Dash.js

In shaka-project/shaka-player#707 @TheModMaker provided some insight about different browsers behaving differently when it comes to skipping the gaps...