How to seek to and cut from a frame in ffmpeg?

[mifi]

This is the age-old question that I have not yet found a definite answer to:

How does seeking (-ss) actually work in ffmpeg?

More specifically:

• How to seek to frame number N when cutting, so that the output begins at input's frame N? (lossy encoding)
• How to seek to a keyframe K when cutting, so that the output begins at (and includes) that keyframe K? (lossless -c copy)
• How to do this consistently across most of files supported by ffmpeg?

This has been discussed in #13 #1087 #126 #330 #1513 #1585 #1855. This also affects smart cut.

To complicate further:

• The HTML5 video player in LosslessCut sometimes renders the wrong frame at a particular timestamp. This also affect snapshot capture. Avidemux cuts more accurately than losslesscut #1786 Duplicate frame preview at keyframes #1616 Saving .jpeg with FFmpeg sometimes saves frame +1 #1487
  • This can be worked around by converting to supported format (to use the FFmpeg-assisted playback).

Concerns:

• Some files have variable FPS, and frame timestamps (DTS/PTS) could vary based on things like the clock of the recorder (?), but in such case you would think that the frame timestamps returned by ffprobe show_entries packet=pts_time,flags are correct. Or should we instead use best_effort_timestamp_time?
• How does seeking work when we have audio with "audio keyframes" (or audio+video) Cut at the nearest audio (key-) frame instead of the next one #1433

Options:

• For keyframes: seek the start cut-point to a few frames after the keyframe (sometimes 1, sometimes 2, sometimes 3, sometimes even more). this is now implemented as an export option
• Use a timecode that is in the "middle" of a frame's lifetime

[iopq]

when I used normal cut, but not keyframe cut if you cut it wrong you get black frames if you cut from the wrong place because the keyframe got cut out

Why do I have to use the normal cut? Because I want to cut off some trailing frames at the end, since that's not an issue (we have the previous keyframe, just cutting more frames off won't always be problematic)

So you may want to treat "beginning" segments and "ending" markers differently

[DGrv]

Hi,

Thanks for your reply on #717. I deleted my post because I said something wrong.
Anyway I do not know where to post this but I guess this is the same topic.
I did a lot of test and the results are depending on a lot of parameters, especially the framerate and keyframes and certainly other thing that I have no clue.

My goal is to use the split method from losslesscut to cut exactly a video in 2. However this does not work well since it is not cutting exactly at the same place for both segments. Meaning that if you concat the 2 segments you will have at the trim position, duplicate frames, not always near each other.

I try to reproduce the behavior of losslesscut and try other things. I used smart cut.
In summary I found that :

• trim at 2 frames before the selected keyframe seems to get better results for seg1
• sseof do the same as ss (option from ffmpeg)
• changing frame for trim at seg2 does nothing (more logic since we use -ss)

Here a reproducible example with bash
All files and code example

Maybe this can be useful ...

in.mp4 is from a Gopro11, codec was converted to h264 (it was h265), this is the only thing I did with the original video.

Here the code and console output:

---

Lets take a random keyframe from in.mp4:

$$ kfn=20

kf=$(ffprobe -v error -select_streams v -show_frames -print_format csv in.mp4 | grep 'frame,video,0,1' | head -20 | tail -1 | perl -pe 's|frame,video,0,1,.?,(.?),.*|\1|')
$$ kf=3.803792

Lets select keyframe at 3.803792. Here are the timestamps of the all the frames from in.mp4 around this keyframe.

$$ ffprobe -v error -select_streams v -show_frames -print_format csv in.mp4 | perl -pe 's|frame,video,0,(.*?),.*?,(.*?),.*|\2 \1|' | perl -pe 's|(.*?) 1|\1\tKF|' | perl -pe 's|(.*?) 0|\1|' |grep -A 5 -B 5 --color 3.803792
3.720375
3.737083
3.753750
3.770417
3.787125
3.803792 KF
3.820500
3.837167
3.853833
3.870542
3.887208

Lets compare 2 methods of split: actual losslesscut 3.53.0 and another one

---

Method 1 - Losslesscut

if I use the split method at timeframe 3.803792 it will run

$$ ffmpeg -stats -v error -i in.mp4 -t 3.803792 -map 0:0 -c:0 copy -map 0:1 -c:1 copy -map_metadata 0 -movflags use_metadata_tags -movflags +faststart -ignore_unknown -f mp4 -y in_seg1.mp4
$$ ffmpeg -stats -v error -ss 3.803792 -i in.mp4 -avoid_negative_ts make_zero -map 0:0 -c:0 copy -map 0:1 -c:1 copy -map_metadata 0 -movflags use_metadata_tags -movflags +faststart -ignore_unknown -f mp4 -y in_seg2.mp4

Timestamps of in_seg1.mp4:

$$ ffprobe -v error -select_streams v -show_frames -print_format csv in_seg1.mp4 | perl -pe 's|frame,video,0,(.*?),.*?,(.*?),.*|\2 \1|' | perl -pe 's|(.*?) 1|\1\tKF|' | perl -pe 's|(.*?) 0|\1|' |grep -A 5 -B 5 --color 3.803792
3.720375
3.737083
3.753750
3.770417
3.787125
3.803792 KF

we see that it is taking 1 or 2 frames too much,always at least the KF

---

Method 2

The only good solution and reproducible that I found it to set the -t to 2 frames before the keyframe wanted

kf2=$(ffprobe -v error -select_streams v -show_frames -print_format csv in.mp4 | perl -pe 's|frame,video,0,.,.?,(.?),.*|\1|' | grep --color -B 2 3.803792 | head -1)

$$ kf2=3.770417

Lets take then 3.770417 instead of 3.803792

$$ ffprobe -v error -select_streams v -show_frames -print_format csv in.mp4 | perl -pe 's|frame,video,0,(.*?),.*?,(.*?),.*|\2 \1|' | perl -pe 's|(.*?) 1|\1\tKF|' | perl -pe 's|(.*?) 0|\1|' |grep -A 5 -B 5 --color 3.770417
3.687000
3.703708
3.720375
3.737083
3.753750
3.770417
3.787125
3.803792 KF
3.820500
3.837167
3.853833
$$ ffmpeg -stats -v error -i in.mp4 -t 3.770417 -map 0:0 -c:0 copy -map 0:1 -c:1 copy -map_metadata 0 -movflags use_metadata_tags -movflags +faststart -ignore_unknown -f mp4 -y in_seg1b.mp4

$$ ffprobe -v error -select_streams v -show_frames -print_format csv in_seg1b.mp4 | perl -pe 's|frame,video,0,(.*?),.*?,(.*?),.*|\2 \1|' | perl -pe 's|(.*?) 1|\1\tKF|' | perl -pe 's|(.*?) 0|\1|' |grep -A 5 -B 5 --color 3.770417
3.687000
3.703708
3.720375
3.737083
3.753750
3.770417
3.787125

now it it stopping just before the keyframe 3.803792.

---

Concat seg1 and seg2 from each method to compare.

$$ (echo file \'in_seg2.mp4\' & echo file \'in_seg1.mp4\' ) > list.txt
$$ ffmpeg -stats -v error -safe 0 -f concat -i list.txt -c copy -y out.mp4
$$ (echo file \'in_seg2.mp4\' & echo file \'in_seg1b.mp4\' ) > listb.txt
$$ ffmpeg -stats -v error -safe 0 -f concat -i listb.txt -c copy -y outb.mp4
$$ ffmpeg -stats -v error -i in.mp4 -i out.mp4 -filter_complex blend=all_mode=difference -c:a copy -y outdiff.mp4
$$ ffmpeg -stats -v error -i in.mp4 -i outb.mp4 -filter_complex blend=all_mode=difference -c:a copy -y outdiffb.mp4

---

Lets check the frame MD5 to see the difference between the 2 methods:

$$ ffmpeg -stats -v error -i in_seg1.mp4 -c copy -f framemd5 -y in_seg1.md5
$$ ffmpeg -stats -v error -i in_seg1b.mp4 -c copy -f framemd5 -y in_seg1b.md5
$$ ffmpeg -stats -v error -i in.mp4 -c copy -f framemd5 -y in.md5
$$ ffmpeg -stats -v error -i out.mp4 -c copy -f framemd5 -y out.md5
$$ ffmpeg -stats -v error -i outb.mp4 -c copy -f framemd5 -y outb.md5

$$ lastframe=df14f6ef759edaf25ce112301ca804d9

lastframe from in_seg1b.mp4 is df14f6ef759edaf25ce112301ca804d9
Lets find this frame in in_seg1.mp4, and in in.mp4

in_seg1.md5

$$ cat in_seg1.md5 | grep --color -B 5 -A 2 df14f6ef759edaf25ce112301ca804d9
1, 178176, 178176, 1024, 345, 6f543ce079298bd464e179b2f4ddd05a
0, 89289, 90891, 400, 21325, 11588ec4c0e943d8df6c1edb8de2d999
1, 179200, 179200, 1024, 344, 6260929978abe1af0deca162736bea8a
0, 89690, 90090, 400, 11299, c7d62f4b3cdbedff9a7a6391824facc6
0, 90090, 90490, 400, 13613, 8358c34183d7ab6f088ea56cd655f42b
1, 180224, 180224, 1024, 315, df14f6ef759edaf25ce112301ca804d9
0, 90490, 91291, 400, 284176, e9c6c1edea17faf8157fb42dd4b78f46
1, 181248, 181248, 1024, 306, 877029df74dbb5028ebdad7e6e664494

in_seg1b.md5

$$ cat in_seg1b.md5 | grep --color -B 5 -A 2 df14f6ef759edaf25ce112301ca804d9
1, 178176, 178176, 1024, 345, 6f543ce079298bd464e179b2f4ddd05a
0, 89289, 90891, 400, 21325, 11588ec4c0e943d8df6c1edb8de2d999
1, 179200, 179200, 1024, 344, 6260929978abe1af0deca162736bea8a
0, 89690, 90090, 400, 11299, c7d62f4b3cdbedff9a7a6391824facc6
0, 90090, 90490, 400, 13613, 8358c34183d7ab6f088ea56cd655f42b
1, 180224, 180224, 1024, 315, df14f6ef759edaf25ce112301ca804d9

in.md5

$$ cat in.md5 | grep --color -B 5 -A 5 df14f6ef759edaf25ce112301ca804d9
1, 178176, 178176, 1024, 345, 6f543ce079298bd464e179b2f4ddd05a
0, 89289, 90891, 400, 21325, 11588ec4c0e943d8df6c1edb8de2d999
1, 179200, 179200, 1024, 344, 6260929978abe1af0deca162736bea8a
0, 89690, 90090, 400, 11299, c7d62f4b3cdbedff9a7a6391824facc6
0, 90090, 90490, 400, 13613, 8358c34183d7ab6f088ea56cd655f42b
1, 180224, 180224, 1024, 315, df14f6ef759edaf25ce112301ca804d9
0, 90490, 91291, 400, 284176, e9c6c1edea17faf8157fb42dd4b78f46
1, 181248, 181248, 1024, 306, 877029df74dbb5028ebdad7e6e664494
0, 90891, 92893, 400, 53891, e47f5bf0d0bb4c82a10e2334f5ad6b09
1, 182272, 182272, 1024, 376, e0aeb413bf281fbd1fba0e866b08ba19
0, 91291, 92092, 400, 18068, aa6cf20b0cb7aee8265ae8acaf1b0d54

Lets check the next frame after lastframe in in.mp4 and find it again in the combine seg1 and seg2 in each method.
$$ lastframein=e9c6c1edea17faf8157fb42dd4b78f46

lastframe from in.mp4 is e9c6c1edea17faf8157fb42dd4b78f46

out.md5

$$ cat out.md5 | grep --color -B 5 -A 5 e9c6c1edea17faf8157fb42dd4b78f46

outb.md5

$$ cat outb.md5 | grep --color -B 5 -A 5 e9c6c1edea17faf8157fb42dd4b78f46

This one is not found in either of both ...
Lets check a bit more in detail in.mp4

Duplicates in out.mp4

$$ cat out.md5 | awk '{print $6}' | grep --color -A 15 df14f6ef759edaf25ce112301ca804d9 | sort | uniq -d
877029df74dbb5028ebdad7e6e664494
b2da0dda190820a8abb60b01faa2af39
df14f6ef759edaf25ce112301ca804d9
e0aeb413bf281fbd1fba0e866b08ba19
e47f5bf0d0bb4c82a10e2334f5ad6b09

Duplicates in outb.mp4

$$ cat outb.md5 | awk '{print $6}' | grep --color -A 15 df14f6ef759edaf25ce112301ca804d9 | sort | uniq -d
df14f6ef759edaf25ce112301ca804d9

[mifi]

next nightly build will have a new Export Option called "Shift all start times", it can be used to automatically shift all segment start times forward by one or more frames when cutting. This can be useful if the output video starts from the wrong (preceding) keyframe.

[DGrv]

next nightly build will have a new Export Option called "Shift all start times", it can be used to automatically shift all segment start times forward by one or more frames when cutting. This can be useful if the output video starts from the wrong (preceding) keyframe.

Is this already implemented ? I was looking everywhere in v3.64.0.0 but did not find it ? did I missed it ?
Thanks :)

[mifi]

You need to enable the export cofnrim dialog. Then it should be under «advanced»

[DGrv]

next nightly build will have a new Export Option called "Shift all start times", it can be used to automatically shift all segment start times forward by one or more frames when cutting. This can be useful if the output video starts from the wrong (preceding) keyframe.

I tried the options and did not get what I wanted but I am actually not sure what you expect and which condition.
Is it supposed to fix this #717