Support for inner product similarity measures (dot product)

[vamshin]

https://github.com/nmslib/nmslib/blob/master/manual/spaces.md#inner-product-spaces.

Feel free to +1 if you land up here looking for this feature :)

[chenqi0805]

By experimenting with negdotprod I find a bug in load index:

Check failed: The number of stored elements 3 doesn't match the number of data points ! Did you forget to re-load data?
java.lang.Exception: Check failed: The number of stored elements 3 doesn't match the number of data points ! Did you forget to re-load data?
	at __randomizedtesting.SeedInfo.seed([45BD2309FB22BD07:E1AD9DFC2D566FE4]:0)
	at com.amazon.opendistroforelasticsearch.knn.index.v1736.KNNIndex.init(Native Method)
	at com.amazon.opendistroforelasticsearch.knn.index.v1736.KNNIndex.loadIndex(KNNIndex.java:123)
	at com.amazon.opendistroforelasticsearch.knn.index.KNNJNITests.testAddAndQueryHnswIndexNegDotProd(KNNJNITests.java:192)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)

This issue is supposed to be fixed in nmslib 1.8 according to nmslib/nmslib#356

I propose an upgrade of the external library before tackling this. Thought? @vamshin @jmazanec15

[vamshin]

@chenqi0805 sounds good to me. We need to probably upgrade the library version first. Also do we have data points on the backward compatibility. Does nmslib 1.8 version can load/query indices created on nmslib 1.7 version?

[chenqi0805]

Also do we have data points on the backward compatibility. Does nmslib 1.8 version can load/query indices created on nmslib 1.7 version?

@vamshin Need further investigation. Overall upgrade is worth a separate issue.

[vamshin]

Sounds good. Created issue #128

[chenqi0805]

After further look at nmslib/nmslib#356, upgrade might not be a prerequisite. One just need to store ObjectVector into a separate '.dat' file for spaces other than l2 and cosine:

https://github.com/nmslib/nmslib/blob/cd79cffafbcd60727510cd610ed4adb96301f5b4/python_bindings/nmslib.cc#L122

But then there will be issue of mismatch between .dat file names in saveIndex and loadIndex due to current way of handling header/footer issue in KNNDocValuesConsumer:

k-NN/src/main/java/com/amazon/opendistroforelasticsearch/knn/index/codec/KNN80Codec/KNN80DocValuesConsumer.java

Line 106 in 1c80b62

KNNIndex.saveIndex(pair.docs, pair.vectors, tempIndexPath, algoParams, spaceType);

Looking further into design doc...

[vamshin]

@chenqi0805 If we can achieve this from plugin upgrade, my suggestion would be to better focus efforts on upgrade the library?

[chenqi0805]

@vamshin I found that upgrade alone does not resolve this issue(the bug posted above). If one follows the fix: https://github.com/nmslib/nmslib/blob/cd79cffafbcd60727510cd610ed4adb96301f5b4/python_bindings/nmslib.cc#L115 where object vector data is saved and loaded for spaces other than l2 and cosine,
we still need upgrade in order to use WriteObjectVectorBinData that does not exist in v1736.

Save and load object vector data will probably lead to re-design of a few APIs in kNN since now we have to deal with the new .dat files. I am not familiar with how codec works but I guess the logic on header, footer and compound files will be affected.

[vamshin]

Now that we have optimized index support built into nmslib for dot prodcut, we should be able to add this feature to our future release.

[prog8]

Hi. @vamshin do you have a rough time estimate when the future release (with dot product) is planned? I tried to find PR which covers this but I didn't. Thanks.

[vamshin]

@prog8, we would plan to have this feature available for next odfe release. Would update the timeline soon. Thanks

[prog8]

@vamshin. Thank you for information

[chenqi0805]

Resolved via #324 #328 .