RealESRGANv2

RealESRGANv2 is a practical image super-resolution neural network that removes common image artifacts.

Link:

• (stable) https://github.com/AmusementClub/vs-mlrt/releases/download/model-20211209/RealESRGANv2_v1.7z

Models

Includes small models optimized for anime videos:

• RealESRGANv2: requires RGB input
  • animevideo-xsx2, animevideo-xsx4 (2x / 4x upscaling)

vsmlrt.py wrapper Usage

In order to simplify usage, we provided a Python wrapper module vsmlrt:

from vsmlrt import RealESRGANv2, RealESRGANv2Model, Backend

src = core.std.BlankClip(format=vs.RGBS)

# backend could be:
#  - CPU Backend.OV_CPU(): the recommended CPU backend; generally faster than ORT-CPU.
#  - CPU Backend.ORT_CPU(num_streams=1, verbosity=2): vs-ort cpu backend.
#  - GPU Backend.ORT_CUDA(device_id=0, cudnn_benchmark=True, num_streams=1, verbosity=2)
#     - use device_id to select device
#     - set cudnn_benchmark=False to reduce script reload latency when debugging, but with slight throughput performance penalty.
#  - GPU Backend.TRT(fp16=True, device_id=0, num_streams=1): TensorRT runtime, the fastest NV GPU runtime.
flt = RealESRGANv2(src, model=RealESRGANv2Model.animevideo_xsx2, backend=Backend.ORT_CUDA())

Raw Model Usage

src = core.std.BlankClip(width=1920, height=1080, format=vs.RGBS)
flt = core.ov.Model(src, "RealESRGANv2/RealESRGANv2-animevideo-xsx2")

Benchmarking

Measurements: FPS / Device Memory (GB)

Device memory:

• CPU: private memory including VapourSynth
• GPU: device memory including context

Tesla V100

Software: VapourSynth R57, Windows Server 2019, Graphics Driver 511.23

Input size: 1920x1080

Backends

1. vs-mlrt v6
2. vs-realesrgan v2.0.0, PyTorch 1.10.1+cu113

Performance

FP32

Model	[1] ort-cuda	[1] trt	[1] trt (2 streams)	[2] cuda
xsx2	5.27 / 2213	6.07 / 1835	6.86 / 2999	3.64 / 6799

FP16

Model	[1] ort-cuda	[1] trt	[1] trt (2 streams)	[2] cuda
xsx2	8.90 / 2981	11.8 / 1637	15.5 / 2539	4.72 / 4291

Tesla A10

Software: VapourSynth R57, Windows Server 2019, Graphics Driver 511.23, lock the GPU clocks at max frequency.

Input size: 1920x1080

Backends

1. vs-mlrt v6
2. vs-realesrgan v2.0.0, PyTorch 1.10.1+cu113

Performance

FP32

Model	[1] ort-cuda	[1] trt	[1] trt (2 streams)	[2] cuda
xsx2	4.15 / 2817	4.97 / 1965	5.26 / 3125	3.47 / 7075

FP16

Model	[1] ort-cuda	[1] trt	[1] trt (2 streams)	[2] cuda
xsx2	7.16 / 3585	11.5 / 1881	12.1 / 2769	4.90 / 4585

Icelake Server

Hardware: Xeon Icelake Server 32C64T @2.90 GHz

Software: VapourSynth R57, Windows Server 2019

Input size: 1920x1080

Backends

1. vs-mlrt v6

Performance

FP32

Model	[1] ov-cpu
xsx2	1.49 / 16751

EPYC Milan

Hardware: EPYC Milan 32C64T @2.55 GHz

Software: VapourSynth R57, Windows Server 2019

Input size: 1920x1080

Backends

1. vs-mlrt v6

Performance

FP32

Model	[1] ov-cpu
xsx2	0.53 / 16750