RIFE with OpenVINO, NCNN with Alpha

[Selur]

Just wanted to give some feedback, using:

core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/vs-mlrt/vsncnn.dll")
import imp
vsmlrt = imp.load_source('vsmlrt', 'i:/Hybrid/64bit/vsfilters/vs-mlrt/vsmlrt.py')
from vsmlrt import Backend
clip = RIFE(clip, multi=2, model=RIFEModel.v4_6, backend=Backend.NCNN_VK)

I end up with:

GridSample not supported yet!
  # align_corners=1
  # mode=bilinear
  # padding_mode=border
GridSample not supported yet!
  # align_corners=1
  # mode=bilinear
  # padding_mode=border
GridSample not supported yet!
  # align_corners=1
  # mode=bilinear
  # padding_mode=border
GridSample not supported yet!
  # align_corners=1
  # mode=bilinear
  # padding_mode=border
Unsupported split axis !
[0 NVIDIA GeForce RTX 4080]  queueC=2[8]  queueG=0[16]  queueT=1[2]
[0 NVIDIA GeForce RTX 4080]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 NVIDIA GeForce RTX 4080]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 NVIDIA GeForce RTX 4080]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
[1 Intel(R) Arc(TM) A380 Graphics]  queueC=1[1]  queueG=0[1]  queueT=2[1]
[1 Intel(R) Arc(TM) A380 Graphics]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[1 Intel(R) Arc(TM) A380 Graphics]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[1 Intel(R) Arc(TM) A380 Graphics]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
[2 AMD Radeon(TM) Graphics]  queueC=1[2]  queueG=0[1]  queueT=2[1]
[2 AMD Radeon(TM) Graphics]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[2 AMD Radeon(TM) Graphics]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[2 AMD Radeon(TM) Graphics]  subgroup=64  basic=1  vote=1  ballot=1  shuffle=1
layer GridSample not exists or registered

and when using:

core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/vs-mlrt/vsov.dll")
import imp
vsmlrt = imp.load_source('vsmlrt', 'i:/Hybrid/64bit/vsfilters/vs-mlrt/vsmlrt.py')
from vsmlrt import Backend
clip = RIFE(clip, multi=2, model=RIFEModel.v4_6, backend=Backend.OV_GPU)

I end up with:

clip = RIFE(clip, multi=2, model=RIFEModel.v4_6, backend=Backend.OV_GPU)
File "i:/Hybrid/64bit/vsfilters/vs-mlrt/vsmlrt.py", line 945, in RIFE
output0 = RIFEMerge(
File "i:/Hybrid/64bit/vsfilters/vs-mlrt/vsmlrt.py", line 827, in RIFEMerge
return inference_with_fallback(
File "i:/Hybrid/64bit/vsfilters/vs-mlrt/vsmlrt.py", line 1444, in inference_with_fallback
raise e
File "i:/Hybrid/64bit/vsfilters/vs-mlrt/vsmlrt.py", line 1423, in inference_with_fallback
return _inference(
File "i:/Hybrid/64bit/vsfilters/vs-mlrt/vsmlrt.py", line 1342, in _inference
clip = core.ov.Model(
File "src\cython\vapoursynth.pyx", line 2612, in vapoursynth.Function.__call__
vapoursynth.Error: operator (): Operation: GridSample_55 of type GridSample(op::v0) is not supported

see also: https://forum.videohelp.com/threads/409482-vs-mlrt-a-bit-after-Getting-started-Backend-trouble

I'm aware that a downloaded an alpha release and this problem might be due to my setup, but in case this is a bug, I wanted to give some feedback.

Side note: I really like the speed boost through vstrt.

Cu Selur

[hooke007]

https://github.com/AmusementClub/vs-mlrt/releases/tag/v12

Here said no support for rife in ncnnvk. And seems no update since then.

[Selur]

Ah okay, missed that. That explains the NCNN_VK issue, but it should work with OV_GPU, right?
Thanks for the info.

[hooke007]

I found this https://github.com/AmusementClub/vs-mlrt/releases/tag/model-20220923

Specifically, ov-gpu and ncnn-vk do not support RIFE due to missing gridsample op.

[Selur]

Top! That solved that, mystery.
So for both it's a known restriction, I simply didn't know of. :)
Thanks.

[WolframRhodium]

RIFE is supported in OV_GPU since vs-mlrt v13.1.

[hooke007]

Is there any other limitations ?

import vapoursynth as vs
from vapoursynth import core
import vsmlrt
from vsmlrt import RIFEModel, Backend

#backend = Backend.OV_CPU()
backend = Backend.OV_GPU(fp16=False, num_streams=2)
src = "test.mp4"
src = vs.core.lsmas.LWLibavSource(src)
ph = (src.height + 31) // 32 * 32 - src.height
pw = (src.width  + 31) // 32 * 32 - src.width
padded = src.std.AddBorders(right=pw, bottom=ph).resize.Bicubic(format=vs.RGBS, matrix_in_s="709")
flt = vsmlrt.RIFE(padded, model=RIFEModel.v4_6, backend=backend)
res = flt.resize.Bicubic(format=vs.YUV420P8, matrix_s="709").std.Crop(right=pw, bottom=ph)
res.set_output()

Script evaluation done in 3.86 seconds
Error: Failed to retrieve frame 1 with error: operator (): [IE exception] Create inference request: [GPU] clEnqueueNDRangeKernel, error code: -52
Output 32 frames in 0.51 seconds (62.69 fps)

fp16 & rgbh doesn't work neither, show the same error as #47 (comment)

[WolframRhodium]

OV_GPU requires Broadwell+ CPUs with integrated graphics or discrete graphics like DG1/Arc, tested on skylake and Arc.

[Selur]

Okay, then it should work with my "Intel(R) Arc(TM) A380 Graphics", right?

[WolframRhodium]

Yes, have you upgraded to vs-mlrt v13.1?

[Selur]

Ah, sorry, thought the nightly at the top was the latest,.. will do and report back.

[WolframRhodium]

Nightly was the latest, both should work.

[Selur]

Using the nightly and:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")

# source: 'G:\TestClips&Co\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive

# adjusting color format to RGBS for custom section
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")

# side note: since my input is already mod32, I do not need any padding



core.std.LoadPlugin(path="i:/Hybrid/64bit/vs-mlrt/vsov.dll")
import imp
vsmlrt = imp.load_source('vsmlrt', 'i:/Hybrid/64bit/vs-mlrt/vsmlrt.py')
from vsmlrt import Backend
from vsmlrt import RIFE, RIFEModel, RIFEMerge
clip = RIFE(clip, multi=2, model=RIFEModel.v4_6, backend=Backend.OV_GPU)


# Output
clip.set_output()

I get:

Failed to evaluate the script:
Python exception: operator (): Operation: GridSample_55 of type GridSample(op::v0) is not supported

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2866, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 2867, in vapoursynth._vpy_evaluate
File "C:\Users\Selur\Desktop\test_1.vpy", line 36, in 
clip = RIFE(clip, multi=2, model=RIFEModel.v4_6, backend=Backend.OV_GPU)
File "i:/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 945, in RIFE
output0 = RIFEMerge(
File "i:/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 827, in RIFEMerge
return inference_with_fallback(
File "i:/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 1444, in inference_with_fallback
raise e
File "i:/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 1423, in inference_with_fallback
return _inference(
File "i:/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 1342, in _inference
clip = core.ov.Model(
File "src\cython\vapoursynth.pyx", line 2612, in vapoursynth.Function.__call__
vapoursynth.Error: operator (): Operation: GridSample_55 of type GridSample(op::v0) is not supported

since the Intel Arc card is device 1, I also tried clip = RIFE(clip, multi=2, model=RIFEModel.v4_6, backend=Backend.OV_GPU(device_id=1)), that returns:

Failed to evaluate the script:
Python exception: operator (): Config for device with 1 ID is not registered in GPU plugin

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2866, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 2867, in vapoursynth._vpy_evaluate
File "C:\Users\Selur\Desktop\test_1.vpy", line 36, in 
clip = RIFE(clip, multi=2, model=RIFEModel.v4_6, backend=Backend.OV_GPU(device_id=1))
File "i:/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 945, in RIFE
output0 = RIFEMerge(
File "i:/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 827, in RIFEMerge
return inference_with_fallback(
File "i:/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 1444, in inference_with_fallback
raise e
File "i:/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 1423, in inference_with_fallback
return _inference(
File "i:/Hybrid/64bit/vs-mlrt/vsmlrt.py", line 1342, in _inference
clip = core.ov.Model(
File "src\cython\vapoursynth.pyx", line 2612, in vapoursynth.Function.__call__
vapoursynth.Error: operator (): Config for device with 1 ID is not registered in GPU plugin

I'm using the latest 31.0.101.4255 Intel® Graphics Driver (in case it matters).

[WolframRhodium]

What is the return value of core.ov.Version()?

[Selur]

that show version to be v13-0-g5b7840f -> must have mixed something up.
Using the vsov from v13.1 fixes the problem.
Thanks!