Exclude msmarco from IT tests #708
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation and see the Github Action logs for details |
|
Perhaps we need to backport #610 first ? |
|
ah ha, yes - i hadnt checked, just assumed it was already in 8.15 |
gareth-ellis
added a commit
to gareth-ellis/rally-tracks
that referenced
this pull request
Dec 6, 2024
gareth-ellis
added a commit
to gareth-ellis/rally-tracks
that referenced
this pull request
Dec 6, 2024
gareth-ellis
added a commit
that referenced
this pull request
Dec 16, 2024
* Add recall and NDCG operations in msmarco-v2-vector (#610) This change adds an operation called knn-recall that computes the following metrics: * Recall * NDCG * Avg number of nodes visited during search Given the size of the corpus, the true top N values used for recall operations have been approximated offline for each query as follows: ``` { "knn": { "field": "emb", "query_vector": query['emb'], "k": 10000, "num_candidates": 10000 }, "rescore": { "window_size": 10000, "query": { "query_weight": 0, "rescore_query": { "script_score": { "query": { "match_all": {} }, "script": { "source": "double value = dotProduct(params.query_vector, 'emb'); return sigmoid(1, Math.E, -value);", "params": { "query_vector": vec } } } } } } } ``` This means that the computed recall is measured against the system's best possible approximate neighbor run rather than the actual top N. For the relevance metrics, the `qrels.tsv` file contains annotations for all the queries listed in `queries.json`. This file is generated from the original training data available at [ir_datasets/msmarco_passage_v2](https://ir-datasets.com/msmarco-passage-v2.html#msmarco-passage-v2/train). (cherry picked from commit b6f3535) * Exclude msmarco from IT tests (#708) --------- Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Whilst checking the output from another PR, i spotted that msmarco-v2-vector takes around 50 minutes to run even in test-mode.
Checking track.py there's a very suspicious loop where we read 12000 lines from a file, which we decompress on the fly. This is run many times during the track, as we load it for each operation. I ran on a very large instance I had running and it took 25 minutes to run, the machines we're using for CI are obviously a lot slower...
It would be good if we could get the test-mode config down to the paramsource etc, and then consider doing something different for this track (or others), but it doesnt seem thats too simple right now. I'll create a ticket in rally to address, but in the meantime I think this track should be excluded from system tests.
It would be nice if we could add a comment we could make that would run a specific test (e.g when making changes to some tracks that are excluded).