Add support for arbitrary ncnn, PyTorch or ONNX models

[Senshou]

Hello, i love Video2x, but i think there is few models to upscale different types of footage. Would it possible to add more models, like from openmodeldb.info for example? Thanks in advance!

[k4yt3x]

I can probably look into this in the long run. At the moment, it only supports ncnn models at best. It'll be great to be able to support PyTorch and ONNX formats.

[Pete4K]

Is it possible to use these models by xinntao:

• RealESR General WDN x4 v3 https://openmodeldb.info/models/4x-realesr-general-wdn-x4v3
• RealESR General x4 v3 https://openmodeldb.info/models/4x-realesr-general-x4v3

These are some of the last models he had created.

[k4yt3x]

Yeah I am aware. The current issue is that the PyTorch to ncnn conversion isn't working. If you try the realesrgan-plus-x2 model, it produces black frames.

[Pete4K]

Damn. I would like to train a model, a Combination of Real ESRGAN Animevideo v3 and Real ESRGAN Anime Plus would be great, because Anime Plus is a little too Sharp and too slow at the moment.

[Pete4K]

Could it be possible to change the Real ESRGAN Anime Plus-Model to 2x (as 2nd option to 4x), when you use such a command like xinntao wrote in an issue tab:
"For now, you can use another argument --outscale. For example, you can add --outscale 2 to upsample x2." xinntao/Real-ESRGAN#88

[k4yt3x]

@Pete4K I took a look at the code. --outscale seems to be just a lanczos filter:

I don't think it's gonna help processing speed?

[Pete4K]

I don't know if that works, because my programming language knowledge is more than worse ;). I think when this would work, you can have a clearer quality with a smaller file size. But I don't know what affects the processing speed. I hope it will be possible in the future. But I don't know this for Anime plus x4, because I saw that this model is trained for anime pictures. But it works totally great for anime and it seems to produce the clearest textures.

[k4yt3x]

If we have a 4x model then its output will always be 4x. Doing the rescaling like this actually takes extra time to complete. If you want the output to be smaller, you can rescale it with lanczos yourself after the processing is completed. You're effectively discarding some work and resizing its resolution to a lower one.

I'm working on #1196 which will add support for FFmpeg filters in v2x.

[Pete4K]

That seems to be a great Idea with the Filters. Thanks!

[avan06]

Yeah I am aware. The current issue is that the PyTorch to ncnn conversion isn't working. If you try the realesrgan-plus-x2 model, it produces black frames.

Hi k4yt3x,

First of all, thank you for creating the practical tool, video2x.

Regarding the realesr-general-x4v3 model,
attached below is the ncnn model I converted from the Python version using the chaiNNer tool.

Based on preliminary testing, I believe video2x should be able to use it.
Since I didn't modify the video2x code, my testing method was simply renaming the model to realesr-animevideov3-x4.

Please feel free to take a look. Thank you!

realesr-general-x4v3.bin.zip

[k4yt3x]

@avan06 Awesome, thanks for sharing! I'll give it a try and add more models using this method if it works well.

[avan06]

Currently, only RealESRGAN_x2plus.pth fails to convert during testing. All x4 realesr models have been tested and work fine. Below are the chaiNNer conversion settings.

[k4yt3x]

Neat! Thanks for letting me know. I tried converting it with pnnx following their official documentations. That didn't work either. Not sure what's wrong here.

[Pete4K]

Would this be a model to try out? https://openmodeldb.info/models/4x-UltraSharp

[Pete4K]

Or these models here: https://github.com/sunny2109/SAFMN

[k4yt3x]

@Pete4K This thread isn't for adding any specific models. Ideally we build a framework that will allow loading any models like chaiNNer or ComfyUI.

[Pete4K]

Ok.

[k4yt3x]

@avan06 your issue is now tracked in #1319