Add Video Swin Transformer Model

[innat]

Short Description

Video Swin Transformer is a pure transformer based video modeling algorithm, attained top accuracy on the major video recognition benchmarks.

Papers

https://arxiv.org/abs/2106.13230
published in 2021, Cited by 1154 (until now).

Existing Implementations

• PyTorch (official): https://github.com/SwinTransformer/Video-Swin-Transformer
• TorchVision : https://pytorch.org/vision/main/models/video_swin_transformer.html
• Keras 2: https://github.com/innat/VideoSwin.
• Keras 3: https://github.com/innat/VideoSwin/tree/feat_kerasv3

Other Information

[divyashreepathihalli]

@innat Thanks for filing the issue! Are you interested in contributing?

[innat]

@divyashreepathihalli

@innat Thanks for filing the issue! Are you interested in contributing?

Unfortunately I don't have long bandwidth to keep working on this feature, (I've noticed there are many pending PR). Therefore, unless there is a high-priority inclusion of this feature in kerascv's current roadmap, I am willing to offer guidance to any contributor interested. Thank you for your understanding.

[simeetnayan81]

Hey @innat @divyashreepathihalli. This project seems interesting and I wish to contribute. Will require some guidance too since I am new to Keras codebase.

[innat]

@simeetnayan81
As you're new to keras-cv, first take a look how they iimplemented backbone and image classification task. According to that, you may start adding video swin as backbone and create video classifier as high level task. For model implementation in keras-v3, please check the first post.

[divyashreepathihalli]

Thank you @simeetnayan81 for your interest and thank you @innat for your help! The team currently does not have bandwidth for this. We appreciate the help!!

[ID6109]

Hey @innat @divyashreepathihalli! I'd love to add this model to the codebase. I have prior experience with handling the models implemented in KerasCV as well. Thanks!

[innat]

@ID6109 @simeetnayan81
Thank you both. Feel free to start working. You guys can collaborate to each other. Check out the resource I shared in the first post.

Note, unlike image model which only have imagenet weight currently, video mdoels often comes with pretrained weight for mutliple dataset, i.e. kinetrics, something something. Also, their rescaling can be different. At first, you don't need to worry about weight, just start adding backbone and high level classifier.

[divyashreepathihalli]

Created a branch - https://github.com/keras-team/keras-cv/tree/video-swin-transformer
please collaborate on this branch and then we can open a PR to master from here.

[innat]

@divyashreepathihalli
Is it acceptable to take guideline from practitioner?