Lossless crop, flip, rotate, aspect ratio in video bitstream or container format metadata

[mifi]

Format metadata

This is metadata that affects presentation of the video, and is stored in the container format (not in the individual streams).

Display aspect ratio (-aspect)

From ffmpeg.org:

-aspect: If used together with -vcodec copy, it will affect the aspect ratio stored at container level, but not the aspect ratio stored in encoded frames, if it exists.

Supported in VLC and MacOS QuickLook/QuickTime (tested by me). See also #493

Matroska specific metadata

Crop

PixelCropBottom=40
PixelCropTop=0
PixelCropLeft=240
PixelCropRight=40

#504 (comment)

Actually, this is possible with a Matroska container. I've done it multiple times in the past with MKVToolNix GUI, and it plays perfectly fine in MPC-BE. For not supporting players like VLC, bug reports should be filed.

The only caveat is that the video data stays the same and the file size is not reduced. But for playing purposes, it's very useful.

In MKVToolNix GUI, import a video file, click the video track, and hover over the text box of "Video properties > Cropping" for help. There's no visual cropping tool, though. That's why Lossless Cut should have it.

I wish this would be implemented. You'd just have to communicate that the feature is limited to MKV output, and that it's just the cropping information that is saved, so the file size isn't reduced and the player is responsible for the actual cropping.

If FFMPEG is the problem (can't it set metadata in the output file?), maybe it must be adapted or a multi-tool solution with the command line equivalent of the MKVToolNix GUI header editor might be the way to go.

Aspect ratio / SAR / DAR

PixelWidth=320
PixelHeight=240
DisplayWidth=320
DisplayHeight=240

Status: Matroska crop and aspect metadata seems to be not supported by FFmpeg yet.

https://trac.ffmpeg.org/ticket/4489
https://trac.ffmpeg.org/ticket/5301
https://forum.videohelp.com/threads/346278-MKVToolnix-cropping-or-changing-display-width-height
https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2016-October/201451.html

MP4 specific metadata

CLAP atom / aperture width / height

Status: Seems to be supported in FFmpeg some time after 11 july 2024, however I can't find any sample file that has this, and I don't know how to use it.

https://trac.ffmpeg.org/ticket/7437
https://trac.ffmpeg.org/ticket/1485

Codec video stream metadata (-bsf)

This is metadata stored in the individual video streams themselves. aka. bitstream filters.

• H264: -bsf:v h264_metadata
• H265(HEVC): -bsf:v hevc_metadata
• MPEG-2: -bsf:v mpeg2_metadata

Crop (crop_left, crop_right, crop_top, crop_bottom)

Supported by H264 and H265.

#38 (comment)

Supported in windows media player

Rotate (rotate)

See also display_orientation.

Flip (flip)

See also display_orientation.

Sample aspect ratio (sample_aspect_ratio)

Supported by H264 and H265. #493

Display aspect ratio

• MPEG-2: display_aspect_ratio
• HEVC width/height ("Set width and height after crop" - not sure if the same)

[h-h-h-h]

Note that, in MKVToolNix GUI, additionally to "Cropping", "Display width/height" must be set to the new video size for distortion-free cropping (should also be possible via "Set aspect ratio").

(This brings us to the idea that Matroska medadata would also make possible the following: aspect ratio correction [spacial stretching/compressing]; adjusting audio delay; temporal stretching/compressing by setting stretch factors and/or FPS per track [even for synced combination of, say, 24-fps video track with English audio track of file 1 and sped-up German audio track of 25-fps video file 2].)

[shodanx2]

Hello,
I would like to signal interest in this feature.
I was searching for a "lossless crop" app as efficient as losslesscut but that doesn't appear to exist yet !
So, an arbitrary crop is not possible without a transcode because of the partial blocks where you're cropping.
For that reason, I think a limited crop that can only crop whole quantization blocks, could that work ?

You'd lose some accuracy on the crop dimensions depending on the size of the blocks, but I think that would still be very usable way to crop video and the benefits of not having to transcode are massive

[Craftyawesome]

For an alternative, some codecs like H264 and HEVC also have crop metadata. E.g. for H264 ffmpeg -i inputfile.mp4 -bsf:v h264_metadata=crop_left=128:crop_right=128:crop_top:0:crop_bottom=0 -codec copy outputfile.mp4
It actually seems to work in a lot of players, although VLC gets it wrong

[shodanx2]

Wow that would be great ! I just needed to re-frame a scene and didn't want to recompress, losing quality, this would be perfect, to draw a crop rectangle on the screen and move it around at various key frames !

[mifi]

nice find! i wonder if it works on the most popular video players and tools like

• chrome's built-in html5 video player
• safari's & ios safari's built in html5 video player
• firefox html5 video
• iphone photos app
• android built in video player
• android photos
• apple quicktime
• apple quicklook
• windows media player
• ffmpeg (e.g. read a losslessly cropped video and re-encode it, observing that the encoded video is cropped correctly)

[shodanx2]

Hello,

Please note where was a typo in @Craftyawesome 's command

ffmpeg -i inputfile.mp4 -bsf:v h264_metadata=crop_left=128:crop_right=128:crop_top:0:crop_bottom=0 -codec copy outputfile.mp4

should be

ffmpeg -i inputfile.mp4 -bsf:v h264_metadata=crop_left=128:crop_right=128:crop_top=0:crop_bottom=0 -codec copy outputfile.mp4

FFmpeg output

ffmpeg -hide_banner -i testvideo.mp4 -bsf:v h264_metadata=crop_left=128:crop_right=128:crop_top=0:crop_bottom=0 -codec copy testvideo.metacropped.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'testvideo.mp4':
  Metadata:
    major_brand     : dash
    minor_version   : 0
    compatible_brands: iso6avc1mp41
    creation_time   : 2023-02-14T13:25:33.000000Z
  Duration: 00:01:09.74, start: 0.000000, bitrate: 1985 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 7 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      creation_time   : 2023-02-14T13:25:33.000000Z
      handler_name    : ISO Media file produced by Google Inc.
      vendor_id       : [0][0][0][0]
Output #0, mp4, to 'testvideo.metacropped.mp4':
  Metadata:
    major_brand     : dash
    minor_version   : 0
    compatible_brands: iso6avc1mp41
    encoder         : Lavf60.11.100
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 7 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      creation_time   : 2023-02-14T13:25:33.000000Z
      handler_name    : ISO Media file produced by Google Inc.
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[out#0/mp4 @ 0000025d8cfe5200] video:16874kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.110405%
frame= 2090 fps=0.0 q=-1.0 Lsize=   16893kB time=00:01:09.66 bitrate=1986.3kbits/s speed=1.48e+03x

mediainfo

vlc

Hmm, I might need a more concrete test video, but it's doing something

[Craftyawesome]

Please note where was a typo in @Craftyawesome 's command

My bad, copied it from one of the related issues in that vlc issue link without checking myself.

Hmm, I might need a more concrete test video, but it's doing something

As mentioned in the issue link vlc seems to interpret left and top as an additional right and bottom. Try almost any other player, it seems to be quite compatible barring vlc.

[shodanx2]

Ok,

I have found a more suitable test file

testvideo.mp4

I have run the following commands to crop it different ways for testing

ffmpeg -hide_banner -i testvideo.mp4 -bsf:v h264_metadata=crop_left=500:crop_right=0:crop_top=0:crop_bottom=0 -codec copy testvideo.metacropped.cropleft500.mp4
ffmpeg -hide_banner -i testvideo.mp4 -bsf:v h264_metadata=crop_left=0:crop_right=500:crop_top=0:crop_bottom=0 -codec copy testvideo.metacropped.cropright500.mp4
ffmpeg -hide_banner -i testvideo.mp4 -bsf:v h264_metadata=crop_left=0:crop_right=0:crop_top=500:crop_bottom=0 -codec copy testvideo.metacropped.croptop500.mp4
ffmpeg -hide_banner -i testvideo.mp4 -bsf:v h264_metadata=crop_left=0:crop_right=0:crop_top=0:crop_bottom=500 -codec copy testvideo.metacropped.cropbottom500.mp4
ffmpeg -hide_banner -i testvideo.mp4 -bsf:v h264_metadata=crop_left=250:crop_right=250:crop_top=0:crop_bottom=0 -codec copy testvideo.metacropped.cropmiddlehorizon.mp4
ffmpeg -hide_banner -i testvideo.mp4 -bsf:v h264_metadata=crop_left=0:crop_right=0:crop_top=250:crop_bottom=250 -codec copy testvideo.metacropped.cropmiddlevertical.mp4
ffmpeg -hide_banner -i testvideo.mp4 -bsf:v h264_metadata=crop_left=250:crop_right=250:crop_top=250:crop_bottom=250 -codec copy testvideo.metacropped.cropcenter.mp4

Here is the console output of that
ffmpeg-metacrop.log

And here are the test files

testvideo.metacropped.cropbottom500.mp4

testvideo.metacropped.cropcenter.mp4

testvideo.metacropped.cropmiddlehorizon.mp4

testvideo.metacropped.cropmiddlevertical.mp4

testvideo.metacropped.cropright500.mp4

testvideo.metacropped.croptop500.mp4

testvideo.metacropped.cropleft500.mp4

Here is mediainfo about these files.

[shodanx2]

Here is how VLC handles those files.

[shodanx2]

Here is how MPC-HC abandonned version MPC-HC.1.7.13.x64.exe handles it

[shodanx2]

Here is how MPC-HC.1.9.24.x64.exe handles it

[shodanx2]

Here is how MPC-HC.2.0.0.x64.exe handles it

[shodanx2]

Here is how mpv-x86_64-v3-20230903-git-67368ac.7z handles it

[shodanx2]

How WACUP_Preview_v1_0_21_7236.exe handles it
Actually it is completely broken, it only shows the first frame then skips to the next time.

Even the original file does not play either.
Still I managed to capture the crop factor

[shodanx2]

How smplayer-23.6.0-x64-unsigned.exe handles it
This is using the MPV engine

[shodanx2]

How smplayer-23.6.0-x64-unsigned.exe handles it
But using the mplayer engine

[shodanx2]

How apprenticevideo-git.replay.20230729.181616.745a82e1-win32-x64.msi handles it

[shodanx2]

Here is how firefox 117.0 (64-bit) handles it

[shodanx2]

How Microsoft Photos 2019.19071.12548.0 handles it

[shodanx2]

How ffplay version 2023-09-04-git-f8503b4c33-full_build-www.gyan.dev handles it

[mifi]

thanks for researching. maybe if we could make a summary/table of each player and how well it works for them, then we could look into whether it's worth implementing

[shodanx2]

Yes, there needs to be a more visual test pattern, make all outputs standard sized and in the same order, so at a glance the difference between players can be seen.

I'm also very curious to know, if I cut up the test pattern video into multiple chunks, set different crop factors for each chunk and then re-assemble them, Will the crop factor change on the fly.

Also all these tests need to be tried with all codecs that support metadata cropping (at least h.265 and AV1) to see whether this feature is getting improvements over new codecs.

Having lossless cropping would be awesome, when I started doing videos it was my second most used action and 3rd was masking.

[Jamim]

Note that, in MKVToolNix GUI, additionally to "Cropping", "Display width/height" must be set to the new video size for distortion-free cropping (should also be possible via "Set aspect ratio").

(This brings us to the idea that Matroska medadata would also make possible the following: aspect ratio correction [spacial stretching/compressing]; adjusting audio delay; temporal stretching/compressing by setting stretch factors and/or FPS per track [even for synced combination of, say, 24-fps video track with English audio track of file 1 and sped-up German audio track of 25-fps video file 2].)

I found that this method works perfectly for YouTube since it respects metadata when transcoding uploaded videos. In my case, I've cut out exactly 108 pixels from the left. By the way, it seems like it's impossible to achieve the same result using h264_metadata.

So @h-h-h-h, thanks a lot!

[dr0biwan]

WACUP

Apologies bumping this but I assume there's stuff I still need to look into fixing for wacup? Just wanted to check which file(s) I should be testing against or if there's anything else I need to be thinking about, thanks.

[shodanx2]

Hello dr0biwan

If you scroll back enough, you will find the test files in this post.

This thread was an exploration in cropping with metadata, a lossless form of crop (but the data does stay behind, it doesn't get nulled out)

In this exploration you can see that support for this rather obscure feature is not uniform accross all players.

Reading my notes from back then. In the single test I had performed, WACUP failed to play the video instead getting stuck on the first frame.

Keep in mind this was more than a year ago, I imagine things have changed.

For this thread, once most players support this feature well, it would be interesting to see if we can change the crop metadata values over the timeline of the video.

Perhaps create something like a "dolly pan" "dolly zoom" effect and maybe other combination of effects