Skips 4s when playing fMP4 but MPEG-TS is fine (non-native VHS 1.13.4, opposite in VHS 2.2)

[RafalLukawiecki]

Description

Playing back HLS H264 video stream with demuxed AAC audio, both contained in a fragmented MP4 using video.js 7.8.4 and VHS 1.13.4 causes the first 4 seconds to be skipped. Also, resetting the playhead to the beginning, either by hand or in JS, causes the player to hang indefinitely. Playing back the same H264 segmented into a single-file MPEG-TS works. The issue does not affect Safari or the iOS native browser, only non-native playback, it seems. I have tested this in Chrome and Firefox on macOS. Interestingly, the situation is reversed when using VHS 2.2.

Steps to reproduce

I have uploaded two version of a test stream that causes the issues:

1. Problematic under VHS 1.13.4, fine in VHS 2.2: use https://videojs.github.io/http-streaming/ to play https://d2gw5qqm1tusud.cloudfront.net/pml-test-fmp4/m.m3u8
2. Fine in VHS 1.13.4, problematic in VHS 2.2: use https://videojs.github.io/http-streaming/ to play https://d2gw5qqm1tusud.cloudfront.net/pml-test-mpeg-ts/m.m3u8

Results

Expected

Both videos to play fully from the start.

Actual

The fMP4 one skips a few seconds at the start while MPEG-TS plays fully. Also, moving the playhead to the start causes video.js to hang indefinitely when playing back fMP4. Also, the fMP4 playback shows a black screen at the beginning, rather than showing the first frame as MPEG-TS does.

I have also tried those streams in videojs-http-streaming 2.2 as hosted at https://videojs-http-streaming.netlify.app hoping it would solve the issue. The situation is almost reversed. The fMP4 stream plays correctly, but the MPEG-TS stream hangs throwing the following errors in the console:

Uncaught TypeError: Cannot read property 'endTimeline' of undefined
    at Transmuxer.endTimeline (02f57f23-c422-4b68-b94e-9e420557d204:5453)
    at MessageHandlers.endTimeline (02f57f23-c422-4b68-b94e-9e420557d204:7144)
    at TransmuxerWorker.self.onmessage (02f57f23-c422-4b68-b94e-9e420557d204:7181)
Transmuxer.endTimeline @ 02f57f23-c422-4b68-b94e-9e420557d204:5453
endTimeline @ 02f57f23-c422-4b68-b94e-9e420557d204:7144
TransmuxerWorker.self.onmessage @ 02f57f23-c422-4b68-b94e-9e420557d204:7181

Error output

No errors in browser console under VHS 1.13.4, only under VHS 2.2.

Additional Information

versions

videojs

7.8.4

browsers

Chrome 86.0.4240.80 and Firefox 81.0.2. No problems in desktop or iOS Safari where both streams play perfectly.

OSes

macOS 10.15.7

plugins

videojs-http-streaming version 1.13.4, although I have also tested it with 1.13.2. See note above about 2.2.

[gkatsev]

Looks like the ts stream is failing because the audio isn't containerized and we're trying to treat it as a ts segment for whatever reason and failing. We should be able to handle it but it doesn't seem like we have a specific example for it, and we don't handle it in VHS 2 for some reason. I guess the workaround for now would be to make sure that the audio is in a ts container as well. The fmp4 stream uses mp4 for both the video and the audio.
VHS 2 is what we'll be switching to next, and so we'll take a look to see if we can fix it there but it may take a while until we get to it.

[RafalLukawiecki]

Thank you @gkatsev for your analysis. For what it is worth, there should be more examples of this appearing in the wild as the the audio and video, in both examples, were segmented using the official Apple HLS mediafilesegmenter 1.6, the most recent one.

If I understood you correctly, you have addressed the VHS 2.2 side of things only. Do you have any feedback on 1.13.4 as to why the fMP4 is failing? Is it simply unsupported? Or is it a waste of time to even consider 1.13 because 2.2 is coming very soon?

I need to render a good amount of material in the next week so I am also thinking of workarounds available to me. If I cannot use VHS 2.2 with the fMP4s then I will need to ship with TS and 1.13.4, and re-render it all again into fMP4 as soon as 2.2 is out. Any hint on expected timescales (days/weeks/months) would greatly help me make some decisions. Many thanks.

PS. I tried TS audio with fMP4 video but video.js threw errors in that combination. I can try again and get more detail on them.

[gkatsev]

We would be very hard-pressed to do any fixes on the 1.x release line.

VHS 2.2 will be in the next minor Video.js release, it is currently only blocked on videojs/http-streaming#967 which we plan on getting out this week.

Good to know that the apple tools generated that stream as that will make it easier for us to make such an example.

[RafalLukawiecki]

This is excellent news about VHS 2.2 possibly being a matter of a week. Fingers crossed for your efforts.

I am setting up an AWS device farm to focus on validating our streams and the site’s video.js player later this week, hoping to run all the combinations early next week. I would like to give VHS 2.2 a chance as early as that, but if you don’t manage to get it into the video.js release, would it be feasible for me to manually configure 7.8.4 to use 2.2? I tried but I have failed. I was getting player errors about undefined not having endtimeline properties no matter what stream I played.

I cannot express how much I appreciate your help at this time, deadlines and all. Thank you.

[gkatsev]

Yup, you can definitely include VHS 2 on your page as well. It will then use that. If you're concerned about both versions of VHS being included, you can use the core dist of Video.js (https://unpkg.com/video.js@7.8.4/dist/alt/video.core.js).

[RafalLukawiecki]

Great! I will try again and I will make a note of the errors if any. I have tried both approaches, ie. with just the core + vhs, and also by loading vhs on top of the full version of vjs (got warning about redefining)—but both ended up with player errors. However, because I have tried so many combinations I cannot be sure what was the cause. I will have more data, negative or positive, tomorrow. Thank you again.

[RafalLukawiecki]

@gkatsev, once again, thank you for your support. I can confirm that using the combination of video.core.js 7.8.4 with videojs-http-streaming.js 2.2.2 indeed does work on our site using purely fMP4 iso-segmented test stream produced by the Apple tools. Unfortunately, I am unable to playback MPEG-TS streams on my site, even though they do play using your hosted https://videojs-http-streaming.netlify.app/ player. I have tried both VHS 2.2.0 and 2.2.2. I have tried having both audio and video in TS (https://d2gw5qqm1tusud.cloudfront.net/pml-test-mpeg-ts/m.m3u8) and having audio TS with video fMP4 (https://d2gw5qqm1tusud.cloudfront.net/pml-test-audio-mpeg-ts-video-fmp4/m.m3u8). Any suggestions as to what I should check next would be much appreciated. Thank you.

The error I get is:

Uncaught TypeError: Cannot read property 'endTimeline' of undefined
    at Transmuxer.endTimeline (4fdb4683-890b-48df-bf0e-7f8389f1964d:5453)
    at MessageHandlers.endTimeline (4fdb4683-890b-48df-bf0e-7f8389f1964d:7144)
    at TransmuxerWorker.self.onmessage (4fdb4683-890b-48df-bf0e-7f8389f1964d:7181)
Transmuxer.endTimeline @ 4fdb4683-890b-48df-bf0e-7f8389f1964d:5453
endTimeline @ 4fdb4683-890b-48df-bf0e-7f8389f1964d:7144
TransmuxerWorker.self.onmessage @ 4fdb4683-890b-48df-bf0e-7f8389f1964d:7181

VIDEOJS: ERROR: (CODE:4 MEDIA_ERR_SRC_NOT_SUPPORTED) The media could not be loaded, either because the server or network failed or because the format is not supported. 
MediaError {code: 4, message: "The media could not be loaded, either because the network failed or because the format is not supported."}

Under Firefox the error is a little different:

TypeError: this.transmuxPipeline_.headOfPipeline is undefined

[RafalLukawiecki]

I can confirm that video.js 7.10.1 with its VHS 2.2.3 is able to playback fMP4 without skipping the first 4 seconds. Unfortunately, I am still unable to play MPEG-TS as per the comment above. Switching back to VHS 1.13.4 reverses the situation: the above TS plays well but fMP4 does not. I wonder if there is an issue with the versioning of .m3u8.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gkatsev]

Good news, I just tried both URLs on https://videojs-http-streaming.netlify.app/ and they both seem to be working!

[RafalLukawiecki]

Thank you, @gkatsev. I can also confirm the streams play well in Firefox and Chrome on macOS, using videojs 7.11.0 and vhs 2.4.1. Thank you for your work on this.

I am still experiencing issues with Edge (Chromium) on some Surface devices. I will recheck the status and submit a separate report this week.