Intermittent Issue with some VODs, Download Works, Problems at Combine

[cmpteki]

I have been using concat for years (great util!); however, just recently, our family started hitting internet data caps, so today I tried to switch from 720p60 as our default, to something lower, this is when I discovered an intermittent issue at combine. Most stream downloads work, others are quite stubborn and do not.

I tried to find one that always fails for easier reproduction of the error; usually everything is fine, or at most, a retry at a different setting will work, though this Minecraft vod I tried to grab for the kids fails consistently at all settings (493274494).

I used the -qualityinfo flag to get valid / available quality settings.

The download seems to work fine; ie, at 720p60 this 4hr stream will grab ~5GB of chunks, at 480p30 this 4hr stream will grab ~2GB of chunks, at 360p30 this 4hr stream will grab ~1GB of chunks, and at 160p30 this 4hr stream will grab ~512MB of chunks. If you look at the 'thumbnails' for each chunk, they are different and show progression through the entire 4hrs, and each piece can indeed be played directly in a media player.

However; when concat_win stitches, the output is approx 10% of that total download size, and while an mp4 file is produced, there is only a few seconds of video, and four hours of audio.

I tried downgrading to v26, then v25, same issue. I have not gone back further.

I tried with ffmpeg v421 and v401, same issue. I have not gone back further.

I tried manually combining the resulting ts files with ffmpeg (not sure if I am using correct syntax) and see errors relating to "non-monotonous DTS in output stream" and "non-existing PPS 0 referenced".

For the most part I have no issues and am happy to abandon this particular vod; already used ridiculous amounts of data in all this testing, but thought it worth reporting in case others encounter the problem and the issue is able to be accounted for / corrected.

I am also happy to test alternate manual combines if you have any suggestions at the ffmpeg syntax.

[ArneVogel]

I am able to reproduce the behavior you described. Researching the problem it seems like its a problem with the key-frames in a chunck that get downloaded from twitch. Unfortunately I have no idea how to fix this. The problem seems to be present in all of the available quality options for the vod, making me assume its a problem with the vod.

For this vod the problem seems to be only in the 2nd chunck, so to bypass the error you can -start="0 1 0" to start downloading after the erroneous chunck. I only tested up to 10 minutes but that seems to combine into a valid video.

[cmpteki]

Thank you very much for the investigation, workaround, and quick response. Unless you need this left open as a note to look for a method to work in error checking (in a future version) and skip over a bad chunk (or copy last good chunk until valid chunks resume) vs stitching with error at a fraction of the size and dumping the source, this can be closed.

(side note; is there a flag to have concat keep the source until the combine is verified by user? essentially, an ability to bypass the housekeeping feature on a per call basis from the command line?).

Love the util!

[mhepaixao]

Just to tip in.

I'd also love to have a flag to bypass the deletion of temp chunk files after the combine. Twice I lost 20gb+ downloads due to invalid chunks.

Still, love this util!

[Wiethoofd]

After aborting the combining I got to keep the chunks and tried to figure out why it kept failing (output was correct length, only the first chunk of video and then audio for the rest).

Turns out that the first and last chunk twitch downloads have a '1000.00 frames/second' when you look at the file property media details.

Reusing the generated createConcatFile in the folder and commenting out the first chunk results in working video for all chunks with the correct framerate (60fps in this example), but then fails with playback of the video for the final chunk and only plays audio in that section, similar to the original concat operation where only the first chunk plays correctly and then results in audio only.

ffprobe on the starting (and end) chunk results in:

Input #0, mpegts, from '626209815_0.ts':
  Duration: 00:00:08.98, start: 547.016000, bitrate: 1726 kb/s
  Program 1
    Stream #0:0[0x100]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 118 kb/s
    Stream #0:1[0x102]: Data: timed_id3 (ID3  / 0x20334449)
    Stream #0:2[0x101]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, unknown/bt470bg/unknown, progressive), 1280x720, 60 fps, 60 tbr, 90k tbn, 2k tbc
Unsupported codec with id 100358 for input stream 1

and the 2nd (and 3rd, 4th, 5th) chunk report:

Input #0, mpegts, from '626209815_1.ts':
  Duration: 00:00:10.01, start: 496.015000, bitrate: 6380 kb/s
  Program 1
    Stream #0:0[0x100]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 158 kb/s
    Stream #0:1[0x101]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg/unknown/unknown, progressive), 1280x720 [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 90k tbn, 120 tbc
    Stream #0:2[0x102]: Data: timed_id3 (ID3  / 0x20334449)
Unsupported codec with id 100358 for input stream 2

Note the stream number which is reported as incorrect in each ffprobe output!

---

Manually fixing the first and last chunk with ffmpeg -i chunk_0.ts -map 0:a:0 -map 0:v:0 -map 0:d:0 -c copy chunk_0_fix.ts then results in the following ffprobe output:

Input #0, mpegts, from '626209815_0_fix.ts':
  Duration: 00:00:08.97, start: 1.433000, bitrate: 1796 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 121 kb/s
    Stream #0:1[0x101]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, unknown/bt470bg/unknown, progressive), 1280x720, 60 fps, 60 tbr, 90k tbn, 2k tbc
    Stream #0:2[0x102]: Data: timed_id3 (ID3  / 0x20334449)
Unsupported codec with id 100358 for input stream 2

Which reordered the streams in the file; using these fixed reordered streams in start and end chunk with ffmpeg to concat results in a successful VOD output.

I have tried using the -map 0 during the -f concat from the original .exe, but that results in incorrect mapping when combining; the first and last chunk need to be fixed before the concat result is successful.