This repository contains the PyTorch implementation of the paper DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval. It provides code for the knowledge distillation training of coarse- and fine-grained student networks based on similarities calculated from a teacher and the selector network. Also, the scripts for the training of the selector network are included. Finally, to facilitate the reproduction of the paper's results, the evaluation code, the extracted features for the employed video datasets, and pre-trained networks for the various students and selectors are available.
- Python 3
- PyTorch >= 1.1
- Torchvision >= 0.4
- Clone this repo:
git clone https://github.com/mever-team/distill-and-select
cd distill-and-select- You can install all the dependencies by
pip install -r requirements.txtor
conda install --file requirements.txt-
We provide our extracted features for all datasets to facilitate reproducibility for future research.
-
Download the feature files of the dataset you want:
-
All feature files are in HDF5 format
We provide the code for training and evaluation of our student models.
- To train a fine-grained student, run the
train_student.pygivenfine-grainedas value to the--student_typeargument, as in the following command:
python train_student.py --student_type fine-grained --experiment_path experiments/DnS_students --trainset_hdf5 /path/to/dns_100k.hdf5- You can train an attention or binarization fine-grained students by setting either the
--attentionor--binarizationflags totrue, respectively.
For fine-grained attention students:
python train_student.py --student_type fine-grained --binarization false --attention true --experiment_path /path/to/experiment/ --trainset_hdf5 /path/to/dns_100k.hdf5For fine-grained binarization students:
python train_student.py --student_type fine-grained --binarization true --attention false --experiment_path /path/to/experiment/ --trainset_hdf5 /path/to/dns_100k.hdf5- To train a coarse-grained students, provide
coarse-grainedto the--student_typeargument:
python train_student.py --student_type coarse-grained --experiment_path /path/to/experiment/ --trainset_hdf5 /path/to/dns_100k.hdf5 --attention true --learning_rate 1e-5- Provide one of the
teacher,fg_att_student_iter1,fg_att_student_iter2to the--teacherargument in odrder to train a student with a different teacher:
python train_student.py --teacher fg_att_student_iter2 --experiment_path /path/to/experiment/ --trainset_hdf5 /path/to/dns_100k.hdf5- You can optionally perform validation with FIVR-5K by providing its HDF5 file to the
--val_hdf5and choosing one of the DSVR, CSVR, ISVR sets with the--val_setargument:
python train_student.py --student_type coarse-grained --val_hdf5 /path/to/fivr_5k.hdf5 --val_set ISVR --experiment_path /path/to/experiment/ --trainset_hdf5 /path/to/dns_100k.hdf5 --learning_rate 1e-5-
Choose one of the
FIVR-5K,FIVR-200K,CC_WEB_VIDEO,SVD, orEVVEdatasets to evaluate your models. -
For the evaluation of the students, run the
evaluation_student.pyscript by providing the path to the.pthmodel to the--student_pathargument, as in the following command:
python evaluation_student.py --student_path experiments/DnS_students/model_fg_att_student.pth --dataset FIVR-5K --dataset_hdf5 /path/to/fivr_200k.hdf5- If you don't pass any value to the
--student_path, a pretrained model will be selected:
python evaluation_student.py --student_type fine-grained --attention true --dataset FIVR-5K --dataset_hdf5 /path/to/fivr_200k.hdf5We also provide the code for training of the selector network and the evaluation of our overall DnS framework.
- To train a selector network, run the
train_selector.pyas in the following command:
python train_selector.py --experiment_path experiments/DnS_students --trainset_hdf5 /path/to/dns_100k.hdf5- Provide different values to
--thresholdargument to train the selector network with different label functions.
- For the evaluation of the DnS framework, run the
evaluation_dns.pyscript by providing the path to the.pthmodel to the corresponding network arguments, as in the following command:
python evaluation_dns.py --selector_network_path experiments/DnS_students/model_selector_network.pth --dataset FIVR-5K --dataset_hdf5 /path/to/fivr_200k.hdf5- If you don't pass any value to the network path argument, then the pretrained model will be selected. E.g. to evalute DnS with the Fine-grained Attention Student:
python evaluation_dns.py --attention true --dataset FIVR-5K --dataset_hdf5 /path/to/fivr_200k.hdf5- Provide different values to
--percentageargument to sent different number of video pairs for reranking to the Fine-grained student. Given the valueall, it runs evaluation for all dataset percentages.
We also provide our pretrained models trained with the fg_att_student_iter2 teacher.
- Load our pretrained models as follows:
from model.feature_extractor import FeatureExtractor
from model.students import FineGrainedStudent, CoarseGrainedStudent
from model.selector import SelectorNetwork
# The feature extraction network used in out experiments
feature_extractor = FeatureExtractor(dims=512).eval()
# Our Fine-grained Students
fg_att_student = FineGrainedStudent(pretrained=True, attention=True).eval()
fg_bin_student = FineGrainedStudent(pretrained=True, binarization=True).eval()
# Our Coarse-grained Students
cg_student = CoarseGrainedStudent(pretrained=True).eval()
# Our Selector Networks
selector_att = SelectorNetwork(pretrained=True, attention=True).eval()
selector_bin = SelectorNetwork(pretrained=True, binarization=True).eval()- First, extract video features by providing a video tensor to feature extractor (similar as here)
video_features = feature_extractor(video_tensor)- Use the
index_video()function providing video features to extract video representations for the student and selector networks
fg_features = fg_att_student.index_video(video_features)
cg_features = cg_student.index_video(video_features)
sn_features = selector_att.index_video(video_features)- Use the
calculate_video_similarity()function providing query and target features to calculate similarity based on the student networks.
fine_similarity = fg_att_student.calculate_video_similarity(query_fg_features, target_fg_features)
coarse_similarity = cg_student.calculate_video_similarity(query_cg_features, target_cg_features)- To calculate the selector's score for a video pair, call the selector network by providing the features extracted for each video and their coarse similarity
selector_features = torch.cat([query_sn_features, target_sn_features, coarse_similarity], 1)
selector_scores = selector_att(selector_features)If you use this code for your research, please consider citing our papers:
@article{kordopatis2022dns,
title={{DnS}: {Distill-and-Select} for Efficient and Accurate Video Indexing and Retrieval},
author={Kordopatis-Zilos, Giorgos and Tzelepis, Christos and Papadopoulos, Symeon and Kompatsiaris, Ioannis and Patras, Ioannis},
journal={International Journal of Computer Vision},
year={2022}
}
@inproceedings{kordopatis2019visil,
title={{ViSiL}: Fine-grained Spatio-Temporal Video Similarity Learning},
author={Kordopatis-Zilos, Giorgos and Papadopoulos, Symeon and Patras, Ioannis and Kompatsiaris, Ioannis},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
year={2019}
}ViSiL - here you can find our teacher model
FIVR-200K - download our FIVR-200K dataset
This work has been supported by the projects WeVerify and MediaVerse, partially funded by the European Commission under contract number 825297 and 957252, respectively, and DECSTER funded by EPSRC under contract number EP/R025290/1.
This project is licensed under the Apache License 2.0 - see the LICENSE file for details
Giorgos Kordopatis-Zilos (georgekordopatis@iti.gr)