videomambasuite

Video Mamba Suite

Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding
Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang, Kunchang Li, Tong Lu, Limin Wang

Abstract

Understanding videos is one of the fundamental directions in computer vision research, with extensive efforts dedicated to exploring various architectures such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state space model, e.g., Mamba, shows promising traits to extend its success in long sequence modeling to video modeling. To assess whether Mamba can be a viable alternative to Transformers in the video understanding domain, in this work, we conduct a comprehensive set of studies, probing different roles Mamba can play in modeling videos, while investigating diverse tasks where Mamba could exhibit superiority. We categorize Mamba into four roles for modeling videos, deriving a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12 video understanding tasks. Our extensive experiments reveal the strong potential of Mamba on both video-only and video-language tasks while showing promising efficiency-performance trade-offs. We hope this work could provide valuable data points and insights for future research on video understanding.

Usage

Before running the TAD experiments, go to video-mamba-suite official repo and install the mamba module as the library.

Results and Models

ActivityNet-1.3 with InternVideo2 classifier.

Features	Setting	mAP@0.5	mAP@0.75	mAP@0.95	ave. mAP	Config	Download
InternVideo2-6B	DBM	63.13	44.36	10.36	42.80	config	model | log

• The validation dataset we used has 4,728 videos, which is the same number as in BMN but less than ActionFormer's implementation. Consequently, this result is slightly higher than VideoMambaSuite's official result. You can check README for more details.

THUMOS-14

Features	Setting	mAP@0.3	mAP@0.4	mAP@0.5	mAP@0.6	mAP@0.7	ave. mAP	Config	Download
InternVideo2-6B	DBM	87.30	82.95	77.17	67.06	51.74	73.24	config	model | log

• Following VSGN, we additionally delete video_test_0000270 during testing due to wrong annotation. Consequently, this result is slightly higher than VideoMambaSuite's official result. You can check README for more details.

FineAction

Features	Setting	mAP@0.5	mAP@0.75	mAP@0.95	ave. mAP	Config	Download
InternVideo2-1B	DBM	45.72	29.37	5.37	29.13	config	model | log

HACS with InternVideo2 classifier.

Features	Setting	mAP@0.5	mAP@0.75	mAP@0.95	ave. mAP	Config	Download
InternVideo2-6B	DBM	64.16	46.08	13.86	44.81	config	model | log

MultiTHUMOS

Features	Setting	mAP@0.2	mAP@0.5	mAP@0.7	ave. mAP (0.1:0.9:0.1)	Config	Download
InternVideo2-6B	DBM	65.71	51.57	31.09	44.58	config	model | log

Train

You can use the following command to train a model.

torchrun --nnodes=1 --nproc_per_node=1 --rdzv_backend=c10d --rdzv_endpoint=localhost:0 tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train VideoMambaSuite on ActivityNet dataset.

torchrun --nnodes=1 --nproc_per_node=1 --rdzv_backend=c10d --rdzv_endpoint=localhost:0 tools/train.py configs/videomambasuite/videomambasuite_internvideo6b.py

For more details, you can refer to the Training part in the Usage.

Test

You can use the following command to test a model.

torchrun --nnodes=1 --nproc_per_node=1 --rdzv_backend=c10d --rdzv_endpoint=localhost:0 tools/test.py ${CONFIG_FILE} --checkpoint ${CHECKPOINT_FILE} [optional arguments]

Example: test VideoMambaSuite on ActivityNet dataset.

torchrun --nnodes=1 --nproc_per_node=1 --rdzv_backend=c10d --rdzv_endpoint=localhost:0 tools/test.py configs/videomambasuite/videomambasuite_internvideo6b.py --checkpoint exps/anet/videomambasuite_internvideo6b/gpu1_id0/checkpoint/epoch_11.pth

For more details, you can refer to the Test part in the Usage.

Citation

@misc{2024videomambasuite,
      title={Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding}, 
      author={Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang, Kunchang Li, Tong Lu, Limin Wang},
      year={2024},
      eprint={2403.09626},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}