Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

Support for cosine similarity search on sparse vectors #112

Open
vamshin opened this issue May 15, 2020 · 4 comments
Open

Support for cosine similarity search on sparse vectors #112

vamshin opened this issue May 15, 2020 · 4 comments
Assignees
Labels
Features New functionality added k-NN Items related to k-NN plugin

Comments

@vamshin
Copy link
Member

vamshin commented May 15, 2020

By default k-NN plugin uses dense vector for Cosine similarity measure. NMS library has support for sparse vector space type. We could enable this support in k-NN plugin.

@jmazanec15
Copy link
Member

jmazanec15 commented Jun 8, 2020

In order to implement Sparse Vector Support, we would need to make a modification to our object creation in JNI code. Currently, the dataset is represented as float * and that pointer is passed in to create the corresponding object. However, for sparse vector support, we would need to represent the Object's data set as SparseVectElem.

Additionally, we need to think of the appropriate UX for users to input data for this format. For example, users could input data in the following format:
{<index>: <value>,..}

@sean-zheng-amazon sean-zheng-amazon added the k-NN Items related to k-NN plugin label Jun 10, 2020
@chenqi0805 chenqi0805 self-assigned this Jul 12, 2020
@chenqi0805
Copy link
Contributor

I can take on this issue if it is not blocking. Together with #109

Additionally, we need to think of the appropriate UX for users to input data for this format.

@vamshin @jmazanec15 Shall we come up with a design of an appropriate sparse vector input format or there is UX designer doing the job?

@chenqi0805
Copy link
Contributor

BTW, in nmslib, sparse space type does not support optmized index, either(#161 )

@chenqi0805
Copy link
Contributor

IMO, we should add a new field type knn_sparse_vector. This is worth a design doc before execution. also applies to #109

@vamshin vamshin added the Features New functionality added label Sep 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Features New functionality added k-NN Items related to k-NN plugin
Projects
None yet
Development

No branches or pull requests

4 participants