Support dot product similarity measure and change scoring function (Backward Compatible) #161

chenqi0805 · 2020-07-07T23:03:26Z

Issue #, if available:
#114 #171

Description of changes:
This PR supports the new space type negdotprod. The main issue in supporting the new space type is that nmslib only supports l2 and cosinesimil for optimized index: https://github.com/nmslib/nmslib/blob/32e6e69a574b678da0d37bece8dbe6b1b250b660/python_bindings/README.md#saving-indexes-and-data

while for all other space types Object vector data has to be saved and loaded explicitly from .dat file. This PR deals with those new .dat files. Changes include

JNI code
Write footer for .dat file
Compound file format
Change scoring function into 1-sigmoid(raw KNN similarity score) in KNNWeight to cover the whole real axis.

Note:

This PR should be reviewed after #160 since it is based off the nmslib latest version.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

chenqi0805 · 2020-07-07T23:10:39Z

@jmazanec15 @vamshin The current state of the PR treats indices for all supported space types as non-optimized indices. I realized that this works OK for new data, but has significant backward compatibility issue with old doc saved under optimized indices for l2 and cosinesimil, e.g.

POST /yourindex/_search
{
    "size" : 10,
    "query": {
        "knn": {
            "my_vector1": {
                "vector": [1, 25],
                "k": 2
            }
        }
    }
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "runtime_exception",
        "reason" : "java.util.concurrent.ExecutionException: java.lang.Exception: Check failed: Cannot open file '/Users/qchea/Downloads/elasticsearch-7.8.0/data/nodes/0/indices/YLGjlb_7TA-wzqJFeJ6Urg/0/index/_0_1736_my_vector1.hnswc.dat' for reading"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "yourindex",
        "node" : "UWTvby-lTSWRsQcPoeFv-g",
        "reason" : {
          "type" : "runtime_exception",
          "reason" : "java.util.concurrent.ExecutionException: java.lang.Exception: Check failed: Cannot open file '/Users/qchea/Downloads/elasticsearch-7.8.0/data/nodes/0/indices/YLGjlb_7TA-wzqJFeJ6Urg/0/index/_0_1736_my_vector1.hnswc.dat' for reading",
          "caused_by" : {
            "type" : "execution_exception",
            "reason" : "java.lang.Exception: Check failed: Cannot open file '/Users/qchea/Downloads/elasticsearch-7.8.0/data/nodes/0/indices/YLGjlb_7TA-wzqJFeJ6Urg/0/index/_0_1736_my_vector1.hnswc.dat' for reading",
            "caused_by" : {
              "type" : "exception",
              "reason" : "Check failed: Cannot open file '/Users/qchea/Downloads/elasticsearch-7.8.0/data/nodes/0/indices/YLGjlb_7TA-wzqJFeJ6Urg/0/index/_0_1736_my_vector1.hnswc.dat' for reading"
            }
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "runtime_exception",
      "reason" : "java.util.concurrent.ExecutionException: java.lang.Exception: Check failed: Cannot open file '/Users/qchea/Downloads/elasticsearch-7.8.0/data/nodes/0/indices/YLGjlb_7TA-wzqJFeJ6Urg/0/index/_0_1736_my_vector1.hnswc.dat' for reading",
      "caused_by" : {
        "type" : "runtime_exception",
        "reason" : "java.util.concurrent.ExecutionException: java.lang.Exception: Check failed: Cannot open file '/Users/qchea/Downloads/elasticsearch-7.8.0/data/nodes/0/indices/YLGjlb_7TA-wzqJFeJ6Urg/0/index/_0_1736_my_vector1.hnswc.dat' for reading",
        "caused_by" : {
          "type" : "execution_exception",
          "reason" : "java.lang.Exception: Check failed: Cannot open file '/Users/qchea/Downloads/elasticsearch-7.8.0/data/nodes/0/indices/YLGjlb_7TA-wzqJFeJ6Urg/0/index/_0_1736_my_vector1.hnswc.dat' for reading",
          "caused_by" : {
            "type" : "exception",
            "reason" : "Check failed: Cannot open file '/Users/qchea/Downloads/elasticsearch-7.8.0/data/nodes/0/indices/YLGjlb_7TA-wzqJFeJ6Urg/0/index/_0_1736_my_vector1.hnswc.dat' for reading"
          }
        }
      }
    }
  },
  "status" : 500
}

To resolve that, I think one option is to treat space types that supports optimized index separately from those do not support optimized index. But before going further, I would like to hear from your comments and suggestions.

chenqi0805 · 2020-07-11T20:38:52Z

Finished separation between optimized(l2, cosinesimil) and non-optimized(negdotprod, etc) space types for KNN index.

…asure * master: FIX: Pass -march=x86-64 to build JNI library (opendistro-for-elasticsearch#164) FIX: Added resetState for uTs so state does not spill over (opendistro-for-elasticsearch#159)

chenqi0805 · 2020-07-18T05:38:08Z

A failed test case scenario:

public void testQueryIndexWithNonOptimizeSpace() throws IOException {
        Settings settings = Settings.builder()
                .put(getKNNDefaultIndexSettings())
                .put("index.knn.space_type", "negdotprod")
                .put("index.knn.algo_param.ef_search", 300)
                .build();
        createKnnIndex("hisindex", settings, createKnnIndexMapping("my_vector1", 2));

        // add doc
        Float[] vector = {1.0f, 1.0f};
        addKnnDoc("hisindex", "1", "my_vector1", vector);

        // search
        float[] queryVector = {1.5f, 2.5f}; // vector to be queried
        int k = 1; //  nearest 1 neighbor
        KNNQueryBuilder knnQueryBuilder = new KNNQueryBuilder("my_vector1", queryVector, k);
        searchKNNIndex("hisindex", knnQueryBuilder, k);
    }

root cause: lucene does not allow negative score in score collector:

=== Standard error of node `node{::integTest-0}` ===
»   ↓ last 40 non error or warning messages from /Users/qchea/IdeaProjects/k-NN/build/testclusters/integTest-0/logs/es.stderr.log ↓
» fatal error in thread [elasticsearch[integTest-0][search][T#1]], exiting
»  java.lang.AssertionError
»       at org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1.collect(TopScoreDocCollector.java:79)
»       at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:242)
»       at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:229)
»       at org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:56)
»       at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
»       at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:212)
»       at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:185)
»       at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
»       at org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:344)
»       at org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:299)
»       at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)
»       at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:361)
»       at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)
»       at org.elasticsearch.search.SearchService.access$200(SearchService.java:135)
»       at org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)
»       at org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)
»       at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
»       at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695)
»       at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
»       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
»       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
»       at java.base/java.lang.Thread.run(Thread.java:832)

We need to change the following scoring formula for negdotprod:

result -> 1/(1 + result.getScore())

…dot-product-similarity-measure * enh/171-better-scoring-function: change function and test case Add release notes automation (opendistro-for-elasticsearch#168) FIX: fix versioning for lib artifacts (opendistro-for-elasticsearch#166)

chenqi0805 · 2020-07-21T19:39:55Z

Modified scoring function according to discussion in #171 .

vamshin · 2020-07-22T18:38:00Z

Hi @chenqi0805,

Thanks for the awesome work. We will get back to this PR soon.

jmazanec15

Hey @chenqi0805 , did you get a chance to investigate why they don't support optimized index for negative dot product in NMSLIB?

chenqi0805 · 2020-07-23T21:58:58Z

Hey @chenqi0805 , did you get a chance to investigate why they don't support optimized index for negative dot product in NMSLIB?

@jmazanec15 Not yet. I will first check issues in nmslib and then post a question in the nmslib lobby.

jmazanec15 · 2020-07-23T23:38:44Z

Cool, saw your comment @chenqi0805 . We could look to contribute if possible.

vamshin · 2021-01-07T19:49:19Z

We now have nmslib supporting optimized version of negative dot product starting version 2.0.8. At this point we would close this PR and come up with new PR which reads optimized version of negative dot product. Thanks for the hard work and contribution @chenqi0805.

jgq2008303393 · 2021-09-01T06:34:43Z

We now have nmslib supporting optimized version of negative dot product starting version 2.0.8. At this point we would close this PR and come up with new PR which reads optimized version of negative dot product. Thanks for the hard work and contribution @chenqi0805.

I would like to ask if there is any progress about the new PR？

chenqi0805 added 6 commits May 29, 2020 15:51

ENH: update nmslib and add regression tests

edaf62e

FEAT: save objectvector

8ade879

Merge current master and delete obsolete artifact

883d4ca

Working solution for compound files

1f7ba59

REF: variables on .dat

9549e84

Resolve conflict

8bcb9af

chenqi0805 requested review from vamshin and jmazanec15 July 7, 2020 23:03

chenqi0805 added 6 commits July 11, 2020 11:26

ENH: separate logic for non-optimized space type

4f3c774

MNT: snakeeyes to camel

86fec5d

Separate logic in add footer

2cd1aa6

FIX: KNNJNITests

7356df0

MAINT: CodecTestCase

6af215f

enrich KNNESIT to cover non-optimized index

17a4193

chenqi0805 changed the title ~~[Backward Compatibility]FEAT: support dot product similarity measure~~ FEAT: support dot product similarity measure(Backward Compatible) Jul 12, 2020

This was referenced Jul 12, 2020

Support for cosine similarity search on sparse vectors #112

Open

Upgrade nmslib to v2.0.6 #160

Merged

Merge branch 'master' into feat/114-support-dot-product-similarity-me…

e4dfd83

…asure * master: FIX: Pass -march=x86-64 to build JNI library (opendistro-for-elasticsearch#164) FIX: Added resetState for uTs so state does not spill over (opendistro-for-elasticsearch#159)

jmazanec15 added the Features New functionality added label Jul 16, 2020

jmazanec15 changed the title ~~FEAT: support dot product similarity measure(Backward Compatible)~~ Support dot product similarity measure(Backward Compatible) Jul 16, 2020

chenqi0805 changed the title ~~Support dot product similarity measure(Backward Compatible)~~ [WIP]Support dot product similarity measure(Backward Compatible) Jul 18, 2020

chenqi0805 mentioned this pull request Jul 18, 2020

Better scoring function in KNN weight #171

Closed

chenqi0805 added 5 commits July 19, 2020 15:07

change function and test case

f0eeee8

TST: nonoptimized space query

abab51b

MNT: test case name

672e7a6

MAINT: optimize test suite

230dcc2

chenqi0805 changed the title ~~[WIP]Support dot product similarity measure(Backward Compatible)~~ Support dot product similarity measure and change scoring function (Backward Compatible) Jul 19, 2020

chenqi0805 added 5 commits July 19, 2020 16:08

STY: unused import

3676559

TST: more test coverage in KNNMapperSearcherIT

c0baca2

RMV: unused variable

be3a419

MNT: line length

5d6934c

Invert scoring by analytically continuate old score function

0604ff6

jmazanec15 reviewed Jul 23, 2020

View reviewed changes

This was referenced Dec 18, 2020

Add nmslib's bit_hamming spaces into plugin #283

Open

[On Hold] Add bit hamming #284

Open

vamshin changed the title ~~Support dot product similarity measure and change scoring function (Backward Compatible)~~ [On hold]Support dot product similarity measure and change scoring function (Backward Compatible) Jan 6, 2021

vamshin changed the title ~~[On hold]Support dot product similarity measure and change scoring function (Backward Compatible)~~ [On Hold]Support dot product similarity measure and change scoring function (Backward Compatible) Jan 6, 2021

vamshin changed the title ~~[On Hold]Support dot product similarity measure and change scoring function (Backward Compatible)~~ [On Hold] Support dot product similarity measure and change scoring function (Backward Compatible) Jan 6, 2021

vamshin closed this Jan 7, 2021

vamshin changed the title ~~[On Hold] Support dot product similarity measure and change scoring function (Backward Compatible)~~ Support dot product similarity measure and change scoring function (Backward Compatible) Jan 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support dot product similarity measure and change scoring function (Backward Compatible) #161

Support dot product similarity measure and change scoring function (Backward Compatible) #161

chenqi0805 commented Jul 7, 2020 •

edited

Loading

chenqi0805 commented Jul 7, 2020

chenqi0805 commented Jul 11, 2020 •

edited

Loading

chenqi0805 commented Jul 18, 2020

chenqi0805 commented Jul 21, 2020

vamshin commented Jul 22, 2020

jmazanec15 left a comment

chenqi0805 commented Jul 23, 2020

jmazanec15 commented Jul 23, 2020

vamshin commented Jan 7, 2021 •

edited

Loading

jgq2008303393 commented Sep 1, 2021

Support dot product similarity measure and change scoring function (Backward Compatible) #161

Support dot product similarity measure and change scoring function (Backward Compatible) #161

Conversation

chenqi0805 commented Jul 7, 2020 • edited Loading

chenqi0805 commented Jul 7, 2020

chenqi0805 commented Jul 11, 2020 • edited Loading

chenqi0805 commented Jul 18, 2020

chenqi0805 commented Jul 21, 2020

vamshin commented Jul 22, 2020

jmazanec15 left a comment

Choose a reason for hiding this comment

chenqi0805 commented Jul 23, 2020

jmazanec15 commented Jul 23, 2020

vamshin commented Jan 7, 2021 • edited Loading

jgq2008303393 commented Sep 1, 2021

chenqi0805 commented Jul 7, 2020 •

edited

Loading

chenqi0805 commented Jul 11, 2020 •

edited

Loading

vamshin commented Jan 7, 2021 •

edited

Loading