SuperRes filter is great, but output-res option is not suitable for it

[rainman74]

Since many videos do not always have exact formats, the SuperRes filter usually does not work. Even if I calculate the size exactly according to the resize factor.

Example:
Input 720x404 --> --output-res=1440x808 --> Suitable ratio for nvvfx-superres not found!

nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 1.5, retrying with other ratios...
nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 2.0, retrying with other ratios...
nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 3.0, retrying with other ratios...
nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 4.0, retrying with other ratios...
resize: Suitable ratio for nvvfx-superres not found.
resize: Failed to init nvvfx filter.

Upscaling from 1280x720 to 1920x1080 (2x) works but I don't need that often.

Therefore, the following alternative option to --output-res would be better for SuperRes:
--output-scale

Examples:
--output-scale 1.5
--output-scale 2
--output-scale 3
--output-scale 4

[quamt]

Hello @rigaya,

I get similar error messages with the new filters.
I am not sure if it is the resolution or the source material.
Is MPEG2 720x480

(The Upscaler feature supports any input resolution, and the following scaling factors: 4/3x (~1.33x), 1.5x, 2x, 3x and 4x.
https://github.com/NVIDIA/MAXINE-VFX-SDK)

Trying to denoise, upscale with superres and artefact reduction I get the error messages below.

nvvfx-denoise: Failed to set input image: A CUDA pitch is not within the acceptable range..
nvvfx-artifact-reduction: Failed to set input image: A CUDA pitch is not within the acceptable range..

nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 4.0, retrying with other ratios...
resize: Suitable ratio for nvvfx-superres not found.
resize: Failed to init nvvfx filter.

or

nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 3.0, retrying with other ratios...
nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 4.0, retrying with other ratios...
resize: Suitable ratio for nvvfx-superres not found.
resize: Failed to init nvvfx filter.

or

nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 2.0, retrying with other ratios...
nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 3.0, retrying with other ratios...
nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 4.0, retrying with other ratios...
resize: Suitable ratio for nvvfx-superres not found.
resize: Failed to init nvvfx filter.

However, I noticed it seems as if it tries to upscale the 720x480 to 1922x1080 when using -2x1080 in the upsize settings.
Shouldnt it go to 2160x1440 and from there resize with the Spline to 1920x1080 as I seen with other material?

[quamt]

I did some more testing and it is the resolution that seem to give the problem.
Upscaling in Handbrake to 720p and reencode with denoise, upscale with superres and artefact reduction works without problem.

However, there is this strange upscaling again.

Vpp Filters    cspconv(nv12 -> yv12(16bit))
               nvvfx-denoise: strength 0
               **resize(spline16): 1280x720 -> 1904x1080**
               cspconv(yv12(16bit) -> p010)

And the video gets this strange look

compared to

Vpp Filters    cspconv(nv12 -> yv12(16bit))
               nvvfx-denoise: strength 0
               resize: nvvfx-superres 1280x720 -> 1920x1080
                       spline36 1920x1080 -> 1904x1080
               cspconv(yv12(16bit) -> p010)

[rigaya]

@quamt The line artifact showing up was due to spline16 bug in NVEnc 7.15, shall be fixed in NVEnc 7.16.

I also think "A CUDA pitch is not within the acceptable range" problem is due to input resolution. as I have no problem with at least 720p and 1080p. The problem is that I could not find from the documents which is the acceptable range (or value).

vpp-resize nvvfx-superres is using "Super Resolution" and not "Upscaler" from the MAXINE-VFX-SDK. As scaling factor is limited to 4/3x (~1.33x), 1.5x, 2x, 3x and 4x, NVEnc will try using these ratios to starting from the ratio near --output-res, and then resize to the exact --output-res value. Therefore, I think --output-res will be enough and adding another --output-scale option will be more confusing.

[rainman74]

@rigaya Then SuperRes is unfortunately not usable for me with NVEnc, because even a simple resize from a 720x400 (16:9) video fails :-(

nvencc64.exe --avhw -i input.mkv -c hevc --profile main --level auto --preset quality --qvbr 28 --aq-temporal --aq-strength 0 --lookahead 32 --bref-mode middle --output-res 1920x1080 [-2x1080] --vpp-resize algo=nvvfx-superres,superres-mode=1,superres-strength=0.5 --audio-copy -o output.mkv

nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 3.0, retrying with other ratios...
nvvfx-superres: Failed to set input image: A CUDA pitch is not within the acceptable range..
resize: Failed to init nvvfx-superres with ratio 4.0, retrying with other ratios...
resize: Suitable ratio for nvvfx-superres not found.
resize: Failed to init nvvfx filter.

Magnifying a 1280x720p or 1920x1080p with SuperRes is relatively useless, but yes, I can also confirm that this works.

Update: 640x480p ->1440x1080 works, too!

[quamt]

@rigaya

The Super Resolution feature supports between 90p to 2160p as input resolutions.
https://github.com/NVIDIA/MAXINE-VFX-SDK

So it seems only Progressive material is supported.
But even after deinterlaced 720x480i in handbrake it gave the error.

Testet 640x480p and it works.

[Alex-Balay]

It seems the width must be a multiple of 128.

[rainman74]

MAXINE-VFX-SDK

yes looks like it! But this doesn't seem to be a limitation of the MAXINE-VFX-SDK (at least according to the documentation).

[rigaya]

@Alex-Balay Thanks, I've did some tests and can confirm width must be multiple of 128, and height seems to have no restriction.

Below is result for --vpp-nvvfx-denoise, "Y" was fine, but "N" went into error.

Input Resolution	Result
320x180	N
320x240	N
384x288	Y
384x216	Y
480x360	N
512x288	Y
512x384	Y
640x360	Y
640x480	Y
704x396	N
704x480	N
720x480	N
768x432	Y
768x576	Y
800x600	N
832x468	N
896x504	Y
896x672	Y
960x720	N
1024x576	Y
1024x768	Y
1152x648	Y
1152x864	Y
1280x720	Y
1280x960	Y
1408x792	Y
1408x1056	Y
1440x810	N
1440x1080	N
1536x864	Y
1920x1080	Y

[rainman74]

@rigaya The fault lies with the SDK, I just tried it with the sample app and there are limitations here that were not mentioned.

So, to make sense of this filter, NVEnc would first have to scale to the next smaller or larger supported width (divisible by 128) if a different width is detected and then apply the SuperRes filter.

Of course only from my point of view, because I don't think anyone needs this filter for resizing 720p and 1080p content.

[Alex-Balay]

The fault lies with the SDK

No, try setting the output resolution to 2x height of the input.

VideoEffectsApp.exe --in_file=000000244.png --out_file=000000244-1.png --effect=SuperRes --resolution=1152 --mode=1 --show

where 000000244.png is 720x576

@rigaya
I'm not sure but maybe the last parameter in the NvCVImage constructor should be 1 ?

NVEncFilterNvvfx.cpp

m_srcImg = std::make_unique<NvCVImage>(pParam->frameIn.width, pParam->frameIn.height,
    NVCV_BGR, NVCV_F32, NVCV_PLANAR, NVCV_GPU  
   , 1 
);

From API:
alignment [in]
Type: unsigned
The row byte alignment, which specifies the alignment of the first pixel in each scan line.
Set this parameter to 0 or a power of 2.
‣ 1: Specifies no gap between scan lines.
A byte alignment of 1 is required by all GPU buffers used by the video effect filters.

[rainman74]

@Alex-Balay oh, I had overlooked that, you're right!

With the SDK it works. So the error lies with NVEnc :-(

[rainman74]

[rigaya]

NVEnc 7.17 should input width which cannot be divided by 128.

What I changed is very simple...

MAXINE-VFX-SDK uses NvCVImage structure as filter input and it's memory can be allocated in 2 ways, one ways is to use NvCVImage C++ constructer and another is to use NvCVImage_Alloc function.

NVEnc 7.17 switches to use NvCVImage_Alloc function instead of C++ constructer as same as in the sample code, and that seems to avoid the error.

I was successfully able to resize 720x480 → 1920x1080.

[rainman74]

@rigaya Thank you! :-)