Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add vmaf --cuda option #178

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Add vmaf --cuda option #178

wants to merge 2 commits into from

Conversation

alexheretic
Copy link
Owner

@alexheretic alexheretic commented Jan 14, 2024

Add support for CUDA accelerated vmaf. See #163.

Run vmaf command with arg --cuda to enable.

This is experimental/incomplete. I can't test this properly myself as I lack nvidia hardware.

Comment on lines +131 to +137
"[0:v]scale_cuda=format={pix_fmt},scale_cuda={w}:{h},setpts=PTS-STARTPTS[dis];\
[1:v]scale_cuda=format={pix_fmt},scale_cuda={w}:{h},{ref_vf},setpts=PTS-STARTPTS[ref];[dis][ref]"
)
} else {
format!(
"[0:v]scale_cuda=format={pix_fmt},setpts=PTS-STARTPTS[dis];\
[1:v]scale_cuda=format={pix_fmt},{ref_vf}setpts=PTS-STARTPTS[ref];[dis][ref]"
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this correct for cuda vmaf?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the vmaf documentation isnt a great help here... maybe this?

let prefix = if let Some((w, h)) = self.vf_scale(model.unwrap_or_default(), distorted_res) {
format!(
"[0:v]scale_cuda=format={pix_fmt},scale={w}:{h},setpts=PTS-STARTPTS[dis];
[1:v]scale_cuda=format={pix_fmt},scale={w}:{h},{ref_vf}setpts=PTS-STARTPTS[ref];[dis][ref]"
)
} else {
format!(
"[0:v]scale_cuda=format={pix_fmt},setpts=PTS-STARTPTS[dis];
[1:v]scale_cuda=format={pix_fmt},{ref_vf}setpts=PTS-STARTPTS[ref];[dis][ref]"
)
};

src/vmaf.rs Outdated Show resolved Hide resolved
@bille747
Copy link

So I had a chance to test this version and was able to confirm that some things works and some don't.

What works

Running ab-av1 vmaf --cuda works perfectly fine (and extremely fast) when the reference and the distorted content uses the same codec.

For example, av1 => av1, x264 => x264, x265 => x265.

What doesn't work

Whenever the reference and distorted content doesn't use the same codec.

For example, x264 => av1, x265 => av1, x264 => x265

How to reproduce

I can reproduce this using the "Big Buck Bunny" test video:

  1. Acquire the 30 MB h264 "Big Buck Bunny" video from here
  2. Convert the file to av1 using ab-av1: ab-av1 auto-encode -i Big_Buck_Bunny_1080_10s_30MB.mp4
  3. Attempt to calculate the VMAF score using the following command: ab-av1 vmaf --cuda --reference Big_Buck_Bunny_1080_10s_30MB.mp4 --distorted Big_Buck_Bunny_1080_10s_30MB.av1.mp4

The command will fail with the following output:

Details

DEBUG: Using ffmpeg -filter_complex [0:v]scale_cuda=format=yuv420p10le,setpts=PTS-STARTPTS[dis];[1:v]scale_cuda=format=yuv420p10le,setpts=PTS-STARTPTS[ref];[dis][ref]libvmaf_cuda=
Error: ffmpeg vmaf exit code 183
---stderr---
d by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x558b1c858b40] No sequence header available
[libaom-av1 @ 0x558b1c858b40] Failed to decode frame: Bitstream not supported by this decoder
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x558b1c857800] Could not find codec parameters for stream 0 (Video: av1 (libaom-av1) (avc1 / 0x31637661), none, 1920x1080, 24559 kb/s): unspecified pixel format
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'Big_Buck_Bunny_1080_10s_30MB.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : Big Buck Bunny, Sunflower version
artist : Blender Foundation 2008, Janus Bager Kristensen 2013
composer : Sacha Goedegebure
encoder : Lavf57.63.100
comment : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
genre : Animation
Duration: 00:00:10.00, start: 0.000000, bitrate: 24563 kb/s
Stream #1:00x1: Video: av1 (libaom-av1) (avc1 / 0x31637661), none, 1920x1080, 24559 kb/s, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 15360 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
[h264_mp4toannexb @ 0x558b1da05580] Global extradata truncated, corrupted stream or invalid MP4/AVCC bitstream
[vist#0:0/h264 @ 0x558b1c829440] [dec:h264_cuvid @ 0x558b1c856380] Error while opening decoder: Invalid data found when processing input
[fc#0 @ 0x558b1c81c0c0] Error binding an input stream to complex filtergraph input scale_cuda:default.
Error binding filtergraph inputs/outputs: Invalid data found when processing input

If I use the ffmpeg command to achieve the same thing, I can get the exact same error if I specific both hwaccel codecs to av1_cuvid:

ffmpeg \
    -hwaccel cuda -hwaccel_output_format cuda -codec:v av1_cuvid -i Big_Buck_Bunny_1080_10s_30MB.av1.mp4 \
    -hwaccel cuda -hwaccel_output_format cuda -codec:v av1_cuvid -i Big_Buck_Bunny_1080_10s_30MB.mp4 \
    -filter_complex "
        [0:v]scale_cuda=format=yuv420p[dis]; \
        [1:v]scale_cuda=format=yuv420p[ref]; \
        [dis][ref]libvmaf_cuda=log_fmt=json:log_path=output.json
    " \
    -f null -
FFMPEG Full Error

ffmpeg version 7.0 Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
configuration: --disable-debug --disable-doc --disable-ffplay --enable-alsa --enable-cuvid --enable-ffprobe --enable-gpl --enable-libaom --enable-libass --enable-libfdk_aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libplacebo --enable-librav1e --enable-libshaderc --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nonfree --enable-nvdec --enable-nvenc --enable-cuda-llvm --enable-opencl --enable-openssl --enable-stripping --enable-vaapi --enable-vdpau --enable-version3 --enable-vulkan
libavutil 59. 8.100 / 59. 8.100
libavcodec 61. 3.100 / 61. 3.100
libavformat 61. 1.100 / 61. 1.100
libavdevice 61. 1.100 / 61. 1.100
libavfilter 10. 1.100 / 10. 1.100
libswscale 8. 1.100 / 8. 1.100
libswresample 5. 1.100 / 5. 1.100
libpostproc 58. 1.100 / 58. 1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Big_Buck_Bunny_1080_10s_30MB.av1.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomav01iso2mp41
title : Big Buck Bunny, Sunflower version
artist : Blender Foundation 2008, Janus Bager Kristensen 2013
composer : Sacha Goedegebure
encoder : Lavf61.1.100
comment : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
genre : Animation
Duration: 00:00:10.00, start: 0.000000, bitrate: 4893 kb/s
Stream #0:00x1: Video: av1 (libaom-av1) (Main) (av01 / 0x31307661), yuv420p10le(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 4891 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.3.100 libsvtav1
[libaom-av1 @ 0x563af837f0c0] obu_reserved_1bit out of range: 1, but must be in [0,0].
[libaom-av1 @ 0x563af837f0c0] Failed to parse extradata.
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[libaom-av1 @ 0x563af837f0c0] No sequence header available
[libaom-av1 @ 0x563af837f0c0] Failed to decode frame: Bitstream not supported by this decoder
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x563af837e200] Could not find codec parameters for stream 0 (Video: av1 (libaom-av1) (avc1 / 0x31637661), none, 1920x1080, 24559 kb/s): unspecified pixel format
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'Big_Buck_Bunny_1080_10s_30MB.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
title : Big Buck Bunny, Sunflower version
artist : Blender Foundation 2008, Janus Bager Kristensen 2013
composer : Sacha Goedegebure
encoder : Lavf57.63.100
comment : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
genre : Animation
Duration: 00:00:10.00, start: 0.000000, bitrate: 24563 kb/s
Stream #1:00x1: Video: av1 (libaom-av1) (avc1 / 0x31637661), none, 1920x1080, 24559 kb/s, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 15360 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
[av1_cuvid @ 0x563af94fd0c0] ctx->cvdl->cuvidParseVideoData(ctx->cuparser, &seq_pkt) failed -> CUDA_ERROR_UNKNOWN: unknown error
[vist#1:0/av1 @ 0x563af8381640] [dec:av1_cuvid @ 0x563af94fc500] Error while opening decoder: Generic error in an external library
[fc#0 @ 0x563af83680c0] Error binding an input stream to complex filtergraph input scale_cuda:default.
Error binding filtergraph inputs/outputs: Generic error in an external library

If I change the ffmpeg command to use h264_cuvid for the reference and av1_cuvid for the distorted video, the command completes successfully and I get a VMAF score:

ffmpeg \
    -hwaccel cuda -hwaccel_output_format cuda -codec:v av1_cuvid -i Big_Buck_Bunny_1080_10s_30MB.av1.mp4 \
    -hwaccel cuda -hwaccel_output_format cuda -codec:v h264_cuvid -i Big_Buck_Bunny_1080_10s_30MB.mp4 \
    -filter_complex "
        [0:v]scale_cuda=format=yuv420p[dis]; \
        [1:v]scale_cuda=format=yuv420p[ref]; \
        [dis][ref]libvmaf_cuda=log_fmt=json:log_path=output.json
    " \
    -f null -

Unfortunately, I have zero experience with programming in Rust, I only have limited experience with Python. However, I am available to help test this further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants