We propose the first benchmark for image query performance prediction (iQPP). First, we establish a set of four data sets (PASCAL VOC 2012, Caltech-101, ROxford5k and RParis6k) and estimate the ground-truth difficulty of each query as the average precision or the precision@𝑘, using two state-of-the-art image retrieval models. Next, we propose and evaluate twelve pre-retrieval and post-retrieval query performance predictors. We release our code as an open source, under the MIT license. However, the retrieval methods and the datasets each have their own open source license.
- Radenovic et al. [TPAMI 2019] - released under MIT License.
- Revaud et al. [ICCV 2019] - released under BSD-3 Clause license.
- PASCAL VOC 2012 (Download) - Flickr terms of use
- Caltech-101 (Download) - Caltech Data terms of use
- ROxford5k (Download) - Flickr terms of use and Dataset Terms of Access
- RParis6k (Download) - Flickr terms of use and Dataset Terms of Access
This repository holds the source code four our paper "Image Query Performance Prediction". It holds the label generation scripts, modifications to the retrieval models and implementations of the studied models. Due to the naming of the retrieval model git repositories, you will often see the retrieval methods described as "cnnimageretrieval" and "deepretrieval". These are in fact the repository names of the work of Radenovic et al. , respectively Revaud et al.
The project is structured as follows:
IQPP/
Datasets/ (contains an extensive description of the dataset format and a short tutorial in order to download the content and prepare it for CBIR)
Embeddings/ (folder which will be populated with the embeddings generated by the retrieval methods)
Folds/ ( contains the 5-folds used for training the supervised models in order for further researcher to obtain consistent replication)
QPP_Methods/ (the codebase for the employed content-based image retrieval models)
Results/ (folder which will contain csv files containing predicted scores after running the QPP methods)
Retrieval_Methods/ (folder which contains the studied retrieval method implementation. these are forks of the original git repository of the authors)
readme.md ( A readme file which contains the current tutorial)
Prerequisites can be seen in the requirements.txt text file. However I would like to point out that Revaud et al. retrieval model cannot be run on an newer scikit-learn version. As per the issue the authors recommend the version 0.20.2.
Run the following command in order to install the prerequisites:
pip install -r requirements.txt
There are multiple steps involved in running the benchmark. In order to replicate our results you will have to follow the next steps:
- Download the datasets from the provided links.
- Copy all the images in the corresponding dataset folder under the "jpg" subfolder. For example, in the case of ROxford5K you must copy the images in "Datasets/ROxford5k/jpg".
- Run the retrieval methods. The methods do not have a unified interface, and as such, require method specific arguments and changes. All of these are described in detail below.
3.1.2. Decide on which dataset you want to run the retrieval model. Available options are:
roxford5k
roxford5k_drift_cnn
rparis6k
rparis6k_drift_cnn
pascalvoc_700
pascalvoc_700_train
pascalvoc_700_drift_cnn
caltech101_700
caltech101_700_train
caltech101_700_drift_cnn
The methods marked as drift denote the queries were identified during the adapted query drift technique.
The embeddings of the query and dataset images will be saved in "Embeddings/CNN_Image_Retrieval/".
The top results will be saved in the folder "Results/".
3.1.3. Select the metric you want to show (p@100 or ap) by changing which value you want to be printed. In "Retrieval_Methods\Radenovic-et-al\radenovic-et-al\cirtorch\utils\evaluate.py" comment one of the lines 135/137. By default, AP is selected.
3.1.4. Run the model with the following instructions:
python -m cirtorch.examples.test --gpu-id "0" --network-path "retrievalSfM120k-resnet101-gem" --datasets "caltech101_700" \ --whitening "retrieval-SfM-120k" --multiscale "[1, 1/2**(1/2), 1/2]"
3.2.1. Decide on which dataset you want to run the retrieval model. Available options are:
ROxford5K
ROxford5K_Drift
RParis6K
RParis6K_Drift
PascalVOC_700
PascalVOC_700_Train
PascalVOC_700_Drift
Caltech101_700
Caltech101_700_Train
Caltech101_700_Drift
3.2.2. Select which metric you want to use. In "Retrieval_Methods\Revaud-et-al\deep-image-retrieval\dirtorch\test_dir.py", call either
eval_query_AP or eval_query_top_k by commenting the specific line (182 or 183, depending on your choice).
3.2.3. Run the model with the following command:
python -m dirtorch.test_dir --dataset 'Caltech101_700' --checkpoint '/notebooks/deep-image-retrieval/Resnet101-AP-GeM.pt'
You must follow the next steps in order to run the query performance prediction methods:
4.1. cd QPP_Methods/Score_Variance
4.2. Run the model with the following command:
python3 main.py --dataset roxford5k --method cnnimageretrieval --metric ap
Available options for methods are:
cnnimageretrieval
deepretrieval
Available options for metrics are:
ap
p@100
- Pytorch - Deep Learning Library.
@inproceedings{poesina2023iqpp,
title={iQPP: A Benchmark for Image Query Performance Prediction},
author={Poesina, Eduard and Ionescu, Radu Tudor and Mothe, Josiane},
booktitle={Proceedings of SIGIR},
year={2023}
}
We thank all researchers and data annotators behind the datasets and the content-based image retrieval methods!
