-
Notifications
You must be signed in to change notification settings - Fork 478
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improvements to flat vector search (#2512)
+ Two separate codecs: AnseriniLucene99FlatVectorFormat and AnseriniLucene99ScalarQuantizedVectorsFormat.
+ Hooked everything up to regressions for all BEIR datasets: {cached, ONNX} x {original, int8}.- Loading branch information
Showing
608 changed files
with
22,461 additions
and
503 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 2 additions & 2 deletions
4
docs/regressions/regressions-beir-v1.0.0-arguana-bge-base-en-v1.5-hnsw.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
82 changes: 82 additions & 0 deletions
82
...egressions/regressions-beir-v1.0.0-arguana.bge-base-en-v1.5.flat-int8.cached.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| # Anserini Regressions: BEIR (v1.0.0) — ArguAna | ||
|
|
||
| **Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with quantized flat indexes (using cached queries) | ||
|
|
||
| This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on [BEIR (v1.0.0) — ArguAna](http://beir.ai/), as described in the following paper: | ||
|
|
||
| > Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. [C-Pack: Packaged Resources To Advance General Chinese Embedding.](https://arxiv.org/abs/2309.07597) _arXiv:2309.07597_, 2023. | ||
| In these experiments, we are using cached queries (i.e., cached results of query encoding). | ||
|
|
||
| The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-arguana.bge-base-en-v1.5.flat-int8.cached.yaml). | ||
| Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-arguana.bge-base-en-v1.5.flat-int8.cached.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. | ||
|
|
||
| From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: | ||
|
|
||
| ``` | ||
| python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana.bge-base-en-v1.5.flat-int8.cached | ||
| ``` | ||
|
|
||
| All the BEIR corpora, encoded by the BGE-base-en-v1.5 model, are available for download: | ||
|
|
||
| ```bash | ||
| wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-bge-base-en-v1.5.tar -P collections/ | ||
| tar xvf collections/beir-v1.0.0-bge-base-en-v1.5.tar -C collections/ | ||
| ``` | ||
|
|
||
| The tarball is 294 GB and has MD5 checksum `e4e8324ba3da3b46e715297407a24f00`. | ||
| After download and unpacking the corpora, the `run_regression.py` command above should work without any issue. | ||
|
|
||
| ## Indexing | ||
|
|
||
| Sample indexing command, building quantized flat indexes: | ||
|
|
||
| ``` | ||
| bin/run.sh io.anserini.index.IndexCollection \ | ||
| -collection JsonDenseVectorCollection \ | ||
| -input /path/to/beir-v1.0.0.bge-base-en-v1.5 \ | ||
| -generator DenseVectorDocumentGenerator \ | ||
| -index indexes/lucene-flat-int8.beir-v1.0.0-arguana.bge-base-en-v1.5/ \ | ||
| -threads 16 -quantize.int8 \ | ||
| >& logs/log.beir-v1.0.0.bge-base-en-v1.5 & | ||
| ``` | ||
|
|
||
| The path `/path/to/beir-v1.0.0.bge-base-en-v1.5/` should point to the corpus downloaded above. | ||
|
|
||
| ## Retrieval | ||
|
|
||
| Topics and qrels are stored [here](https://github.com/castorini/anserini-tools/tree/master/topics-and-qrels), which is linked to the Anserini repo as a submodule. | ||
|
|
||
| After indexing has completed, you should be able to perform retrieval as follows: | ||
|
|
||
| ``` | ||
| bin/run.sh io.anserini.search.SearchCollection \ | ||
| -index indexes/lucene-flat-int8.beir-v1.0.0-arguana.bge-base-en-v1.5/ \ | ||
| -topics tools/topics-and-qrels/topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.gz \ | ||
| -topicReader JsonStringVector \ | ||
| -output runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat-int8.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt \ | ||
| -generator VectorQueryGenerator -topicField vector -removeQuery -threads 16 -hits 1000 & | ||
| ``` | ||
|
|
||
| Evaluation can be performed using `trec_eval`: | ||
|
|
||
| ``` | ||
| bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat-int8.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt | ||
| bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat-int8.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt | ||
| bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat-int8.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt | ||
| ``` | ||
|
|
||
| ## Effectiveness | ||
|
|
||
| With the above commands, you should be able to reproduce the following results: | ||
|
|
||
| | **nDCG@10** | **BGE-base-en-v1.5**| | ||
| |:-------------------------------------------------------------------------------------------------------------|-----------| | ||
| | BEIR (v1.0.0): ArguAna | 0.6361 | | ||
| | **R@100** | **BGE-base-en-v1.5**| | ||
| | BEIR (v1.0.0): ArguAna | 0.9915 | | ||
| | **R@1000** | **BGE-base-en-v1.5**| | ||
| | BEIR (v1.0.0): ArguAna | 0.9964 | | ||
|
|
||
| The above figures are from running brute-force search with cached queries on non-quantized indexes. | ||
| With quantized indexes, results may differ slightly, but the nDCG@10 score should generally be within 0.004 of the result reported above (with a small number of outliers). |
82 changes: 82 additions & 0 deletions
82
.../regressions/regressions-beir-v1.0.0-arguana.bge-base-en-v1.5.flat-int8.onnx.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| # Anserini Regressions: BEIR (v1.0.0) — ArguAna | ||
|
|
||
| **Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with quantized flat indexes (using ONNX for on-the-fly query encoding) | ||
|
|
||
| This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on [BEIR (v1.0.0) — ArguAna](http://beir.ai/), as described in the following paper: | ||
|
|
||
| > Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. [C-Pack: Packaged Resources To Advance General Chinese Embedding.](https://arxiv.org/abs/2309.07597) _arXiv:2309.07597_, 2023. | ||
| In these experiments, we are using ONNX to perform query encoding on the fly. | ||
|
|
||
| The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-arguana.bge-base-en-v1.5.flat-int8.onnx.yaml). | ||
| Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-arguana.bge-base-en-v1.5.flat-int8.onnx.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. | ||
|
|
||
| From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: | ||
|
|
||
| ``` | ||
| python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana.bge-base-en-v1.5.flat-int8.onnx | ||
| ``` | ||
|
|
||
| All the BEIR corpora, encoded by the BGE-base-en-v1.5 model, are available for download: | ||
|
|
||
| ```bash | ||
| wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-bge-base-en-v1.5.tar -P collections/ | ||
| tar xvf collections/beir-v1.0.0-bge-base-en-v1.5.tar -C collections/ | ||
| ``` | ||
|
|
||
| The tarball is 294 GB and has MD5 checksum `e4e8324ba3da3b46e715297407a24f00`. | ||
| After download and unpacking the corpora, the `run_regression.py` command above should work without any issue. | ||
|
|
||
| ## Indexing | ||
|
|
||
| Sample indexing command, building quantized flat indexes: | ||
|
|
||
| ``` | ||
| bin/run.sh io.anserini.index.IndexCollection \ | ||
| -collection JsonDenseVectorCollection \ | ||
| -input /path/to/beir-v1.0.0.bge-base-en-v1.5 \ | ||
| -generator DenseVectorDocumentGenerator \ | ||
| -index indexes/lucene-flat-int8.beir-v1.0.0-arguana.bge-base-en-v1.5/ \ | ||
| -threads 16 -quantize.int8 \ | ||
| >& logs/log.beir-v1.0.0.bge-base-en-v1.5 & | ||
| ``` | ||
|
|
||
| The path `/path/to/beir-v1.0.0.bge-base-en-v1.5/` should point to the corpus downloaded above. | ||
|
|
||
| ## Retrieval | ||
|
|
||
| Topics and qrels are stored [here](https://github.com/castorini/anserini-tools/tree/master/topics-and-qrels), which is linked to the Anserini repo as a submodule. | ||
|
|
||
| After indexing has completed, you should be able to perform retrieval as follows: | ||
|
|
||
| ``` | ||
| bin/run.sh io.anserini.search.SearchCollection \ | ||
| -index indexes/lucene-flat-int8.beir-v1.0.0-arguana.bge-base-en-v1.5/ \ | ||
| -topics tools/topics-and-qrels/topics.beir-v1.0.0-arguana.test.tsv.gz \ | ||
| -topicReader TsvString \ | ||
| -output runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat-int8-onnx.topics.beir-v1.0.0-arguana.test.txt \ | ||
| -generator VectorQueryGenerator -topicField vector -removeQuery -threads 16 -hits 1000 -encoder BgeBaseEn15 & | ||
| ``` | ||
|
|
||
| Evaluation can be performed using `trec_eval`: | ||
|
|
||
| ``` | ||
| bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat-int8-onnx.topics.beir-v1.0.0-arguana.test.txt | ||
| bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat-int8-onnx.topics.beir-v1.0.0-arguana.test.txt | ||
| bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat-int8-onnx.topics.beir-v1.0.0-arguana.test.txt | ||
| ``` | ||
|
|
||
| ## Effectiveness | ||
|
|
||
| With the above commands, you should be able to reproduce the following results: | ||
|
|
||
| | **nDCG@10** | **BGE-base-en-v1.5**| | ||
| |:-------------------------------------------------------------------------------------------------------------|-----------| | ||
| | BEIR (v1.0.0): ArguAna | 0.6361 | | ||
| | **R@100** | **BGE-base-en-v1.5**| | ||
| | BEIR (v1.0.0): ArguAna | 0.9915 | | ||
| | **R@1000** | **BGE-base-en-v1.5**| | ||
| | BEIR (v1.0.0): ArguAna | 0.9964 | | ||
|
|
||
| The above figures are from running brute-force search with cached queries on non-quantized indexes. | ||
| With quantized indexes and on-the-fly ONNX query encoding, results may differ slightly, but the nDCG@10 score should generally be within 0.005 of the result reported above (with a small number of outliers). |
81 changes: 81 additions & 0 deletions
81
docs/regressions/regressions-beir-v1.0.0-arguana.bge-base-en-v1.5.flat.cached.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| # Anserini Regressions: BEIR (v1.0.0) — ArguAna | ||
|
|
||
| **Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with flat indexes (using cached queries) | ||
|
|
||
| This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on [BEIR (v1.0.0) — ArguAna](http://beir.ai/), as described in the following paper: | ||
|
|
||
| > Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. [C-Pack: Packaged Resources To Advance General Chinese Embedding.](https://arxiv.org/abs/2309.07597) _arXiv:2309.07597_, 2023. | ||
| In these experiments, we are using cached queries (i.e., cached results of query encoding). | ||
|
|
||
| The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/beir-v1.0.0-arguana.bge-base-en-v1.5.flat.cached.yaml). | ||
| Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-arguana.bge-base-en-v1.5.flat.cached.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. | ||
|
|
||
| From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: | ||
|
|
||
| ``` | ||
| python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana.bge-base-en-v1.5.flat.cached | ||
| ``` | ||
|
|
||
| All the BEIR corpora, encoded by the BGE-base-en-v1.5 model, are available for download: | ||
|
|
||
| ```bash | ||
| wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-bge-base-en-v1.5.tar -P collections/ | ||
| tar xvf collections/beir-v1.0.0-bge-base-en-v1.5.tar -C collections/ | ||
| ``` | ||
|
|
||
| The tarball is 294 GB and has MD5 checksum `e4e8324ba3da3b46e715297407a24f00`. | ||
| After download and unpacking the corpora, the `run_regression.py` command above should work without any issue. | ||
|
|
||
| ## Indexing | ||
|
|
||
| Sample indexing command, building flat indexes: | ||
|
|
||
| ``` | ||
| bin/run.sh io.anserini.index.IndexCollection \ | ||
| -collection JsonDenseVectorCollection \ | ||
| -input /path/to/beir-v1.0.0.bge-base-en-v1.5 \ | ||
| -generator DenseVectorDocumentGenerator \ | ||
| -index indexes/lucene-flat.beir-v1.0.0-arguana.bge-base-en-v1.5/ \ | ||
| -threads 16 \ | ||
| >& logs/log.beir-v1.0.0.bge-base-en-v1.5 & | ||
| ``` | ||
|
|
||
| The path `/path/to/beir-v1.0.0.bge-base-en-v1.5/` should point to the corpus downloaded above. | ||
|
|
||
| ## Retrieval | ||
|
|
||
| Topics and qrels are stored [here](https://github.com/castorini/anserini-tools/tree/master/topics-and-qrels), which is linked to the Anserini repo as a submodule. | ||
|
|
||
| After indexing has completed, you should be able to perform retrieval as follows: | ||
|
|
||
| ``` | ||
| bin/run.sh io.anserini.search.SearchCollection \ | ||
| -index indexes/lucene-flat.beir-v1.0.0-arguana.bge-base-en-v1.5/ \ | ||
| -topics tools/topics-and-qrels/topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.gz \ | ||
| -topicReader JsonStringVector \ | ||
| -output runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt \ | ||
| -generator VectorQueryGenerator -topicField vector -removeQuery -threads 16 -hits 1000 & | ||
| ``` | ||
|
|
||
| Evaluation can be performed using `trec_eval`: | ||
|
|
||
| ``` | ||
| bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt | ||
| bin/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt | ||
| bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0.bge-base-en-v1.5.bge-flat.topics.beir-v1.0.0-arguana.test.bge-base-en-v1.5.jsonl.txt | ||
| ``` | ||
|
|
||
| ## Effectiveness | ||
|
|
||
| With the above commands, you should be able to reproduce the following results: | ||
|
|
||
| | **nDCG@10** | **BGE-base-en-v1.5**| | ||
| |:-------------------------------------------------------------------------------------------------------------|-----------| | ||
| | BEIR (v1.0.0): ArguAna | 0.6361 | | ||
| | **R@100** | **BGE-base-en-v1.5**| | ||
| | BEIR (v1.0.0): ArguAna | 0.9915 | | ||
| | **R@1000** | **BGE-base-en-v1.5**| | ||
| | BEIR (v1.0.0): ArguAna | 0.9964 | | ||
|
|
||
| Note that since we're running brute-force search, the results should be reproducible _exactly_. |
Oops, something went wrong.