How to do Instance Retrieval like the demo?

[mau-io]

Issue Title: How to do Instance Retrieval like the demo?

Issue Description:

Hello, I have recently come across the demo on instance retrieval in this repository, and I'm very interested in implementing a similar feature in my own project. However, I'm having some difficulties understanding the exact steps and required components to achieve this.

Can you please provide some guidance or documentation on how to replicate the instance retrieval functionality as demonstrated in the demo? Specifically, I'd like to know about the following:

1. Which part of the codebase is responsible for instance retrieval?
2. Any examples or tutorials that could help me better understand the implementation?

I appreciate any help or direction you can provide. Thank you!

[MarcSzafraniec]

Hi ! To answer your question: the part for instance retrieval is not public, so it is not in this code base.

However, the steps to reproduce it are pretty simple: if you have a dataset that you want to retrieve in ("base dataset") and a "query" image, what you need to do is:

• first embed every image in the base dataset with the forward method of one of our models. That will give you a tensor for each image
• You can transform these tensors in a numpy array, then in a faiss index
• Finally, embed the query image with the same model, and query it in the faiss index

It should not be very complicated once you know how to use faiss ! And if the base dataset is relatively small, it should be pretty fast

[patricklabatut]

the part for instance retrieval is not public

The k-NN evaluation code does provide functionality close to that though (with a k-NN classifier).

To complement @MarcSzafraniec's answer with a possibly higher-level description of the retrieval process:

• first, map all the images (both query images and database images) to feature vectors representing these images. At the most basic level, this is done by just applying the model - see this function in the evaluation code,
• then, for each query image and their respective feature vector, determine the closest database images by searching for the closest database features vectors (closest being according to some similarity measure, e.g. cosine similarity). This can be done in vanilla PyTorch (by computing the cosine similarity, i.e. the dot product between normalized query and database vectors and extracting the top-k database vectors).

For some use cases (e.g. the demo), the above could (and should) be done (partially) offline by first indexing of all the database feature vectors to facilitate subsequent queries. And then querying this index on each request. As noted above, the Faiss library can be used to implement such indexing and search logic.

[sherylwang]

Hi ! To answer your question: the part for instance retrieval is not public, so it is not in this code base.

However, the steps to reproduce it are pretty simple: if you have a dataset that you want to retrieve in ("base dataset") and a "query" image, what you need to do is:

• first embed every image in the base dataset with the forward method of one of our models. That will give you a tensor for each image
• You can transform these tensors in a numpy array, then in a faiss index
• Finally, embed the query image with the same model, and query it in the faiss index

It should not be very complicated once you know how to use faiss ! And if the base dataset is relatively small, it should be pretty fast

It seems there will be 3 output features "x_norm_clstoken","x_norm_patchtokens" and "x_prenorm" which should be used for retrieval?

[woctezuma]

For image retrieval, you can directly use :

dinov2/dinov2/eval/utils.py

Line 122 in fc49f49

features_rank = model(samples).float()

This would probably be the equivalent of the CLS token, so x_norm_clstoken which you mentioned.

See:

• Is this the right way to do inference? #2 (comment)
• Is this the right way to do inference? #2 (comment)
• How can i exttract features from vision?  dino#72

[abdelkareemkobo]

This is may help small demo

[patricklabatut]

Closing and keeping track as part of #95.