Champion Solutions repository for Perception Test challenges in ICCV2023 workshop.
We achieves the best performance in Temporal Sound Localisation task and runner-up in Temporal Action Localisation task. In this repository, we provide the pretrained video&audio features, checkpoints, and codes for feature extraction, training, and inference.
Please refer to INSTALL.md to install the prerequisite packages.
For the video features, we use the UMT large model pre-trained on Something Something-V2 and the VideoMAE model pre-trained on Ego4D-Verb dataset. The weights of Ego4d can be found here. These two features are concatenated before putting into the ActionFormer model during both training and inference stages.
For the audio features, we use the BEATs model as feature extractor and adopt its iter3+ checkpoints pre-trained on the AudioSet-2M dataset. we provide scripts to extract BEATs and CAV-MAE (although not used), please use python audio_feat_extract.py to extract audio features.
For the video feature, we use the UMT large model pre-trained on Something Something-V2 and fine-tuned on the perception test temporal action localisation training set.
For the audio features, we use the BEATs model as feature extractor and adopt its iter3+ checkpoints pre-trained on the AudioSet-2M dataset. we provide scripts to extract BEATs and CAV-MAE (although not used), please use python audio_feat_extract.py to extract audio features.
| Features | Modality | Task | Download Link |
|---|---|---|---|
| BEATs_iter2 | Audio | TAL&TSL | Download |
| Ego4d_verb | Video | TAL | Download |
| UMT-L Sth Sth-V2 | Video | TAL | Download |
| UMT-L Sth Sth-V2 ft | Video | TSL | Download |
cd ./tsl/
python train.py configs/perception_tsl_multi_train.yaml
Inference on the validation set:
cd ./tsl/
python eval.py configs/perception_tsl_multi_valid.yaml ./ckpt/XXX -epoch=XX
Inference on the test set:
cd ./tsl/
python eval.py configs/perception_tsl_multi_test.yaml ./ckpt/XXX -epoch=XX --saveonly
cd ./tal/
python train.py configs/perception_tal_multi_train.yaml
Inference on the validation set:
cd ./tal/
python eval.py configs/perception_tal_multi_valid.yaml ./ckpt/XXX -epoch=XX
Inference on the test set:
cd ./tal/
python eval.py configs/perception_tal_multi_test.yaml ./ckpt/XXX -epoch=XX --saveonly
We release the checkpoint in the below table.
| Method | Task | mAP (Valid) | Download |
|---|---|---|---|
| BEATs + UMT | tsl | 26.70 | ckpt |
| BEATs + UMT ft | tsl | 39.25 | ckpt |
| BEATs + UMT | tal | 44.14 | ckpt |
| BEATs + UMT&VideoMAE | tal | 46.75 | ckpt |
If you have any questions, please contact Jiashuo Yu and Guo Chen