Filtering Support on Index Created using IndexIDMap

[navneet1v]

Description

I am from OpenSearch Team. We are using Faiss Library C++ code to enable the K-NN search in OpenSearch via k-NN Plugin. At the bottom we create the faiss index and then do the search on those indices.

Question

Currently we have different types of indices(hnsw, IVF and IVF-PQ) we create based on customer parameters, and we want to support filtering using IDSelector on those indices. But currently when I tried to enable it I am getting exception which is coming from here.

Based on the above context I have few questions:

1. Why filtering is not supported on indices created using: IndexIDMap?
2. Is there any plan to support the filtering on indices created using IndexIDMap?

[mdouze]

There is no particular reason other than the we did not have a use case for it so far.
I will implement this with a wrapped IDSelector that uses the id_map to translate indices.

[navneet1v]

@mdouze thanks this will be very useful. Any ETA that I can track for this change?

[vamshin]

@mdouze in the context of supporting pre filters on faiss indexes, this is very important feature for us. Any inputs on the timeline would be super helpful.

[navneet1v]

Thanks @mdouze for raising the PR.

I can see the PR here: #2848

cc: @vamshin