Issues with long videos. (1530 frames). Video Quality (720p).

[charchit7]

Hey, @hkchengrex I was runnning the demo colab script to test out an object (sofa) in my case. And it failed after few frames.
Do you know why is that the case?
I am assuming because a human subject came into the picture and went away the memory propagation failed? ( issues with the original Xmem paper).

Please let me know your thoughts on it. (check_frame_human is the image when the human came into the frame).

[hkchengrex]

I am not sure what I am looking at here. Most of these masks look decent -- or am I parsing the scene wrong?
Secondly you mentioned that it fails after a few frames, but these frames do not look continuous.

[charchit7]

Yeah, I shared intermediate frames for reference. Failed case is if you look at the first image given above.
Sharing result and input videos with you : Drive Link
Input : obj1_720p.mp4, result : results_obj1.mp4

Please let me know whenever you get time to check. @hkchengrex Thanks :)
Have two questions wrt results :

• Why is the output video not segmenting only one object which is the sofa. ( you can see the color variations in the video frames).
• You can see the segmentation getting distorted in the later frames. I used your colab tutorial, how can I segment the masks separately for visualisation.

[hkchengrex]

Thank you for providing the videos.
To preface, all models make mistakes and Cutie is no exception. Most of the frames look fairly reasonable to me. There might be a few things that can improve the performance of the model, but your millage might vary:

1. It seems that you are mixing up RGB and BGR orderings in the visualization. Cutie uses RGB.
2. The result video has a lower framerate than the source video. Use high frame rate input for Cutie if possible.
3. The only noticeable error I see comes from a finger blocking the camera. That is out of the training distribution, in general leads to a low data quality, and should be avoided if possible.

[hkchengrex]

Feel free to re-open if there are any updates.

[charchit7]

Got it. thank you for the detailed response, @hkchengrex.
I got stuck at work so couldn't update you, will update you on this after trying the changes. Thanks :)