diff --git a/README.md b/README.md index dafcc37187..6aacf99ebf 100644 --- a/README.md +++ b/README.md @@ -67,40 +67,42 @@ See individual pages for details! ### MS MARCO V1 Passage Regressions -| | dev | DL19 | DL20 | -|--------------------------------------------|:------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------:| -| **Unsupervised Sparse** | | | | -| BoW baselines | [+](docs/regressions/regressions-msmarco-passage.md) | [+](docs/regressions/regressions-dl19-passage.md) | [+](docs/regressions/regressions-dl20-passage.md) | -| Quantized BM25 | [✓](docs/regressions/regressions-msmarco-passage-bm25-b8.md) | [✓](docs/regressions/regressions-dl19-passage-bm25-b8.md) | [✓](docs/regressions/regressions-dl20-passage-bm25-b8.md) | -| WP baselines | [+](docs/regressions/regressions-msmarco-passage-wp.md) | [+](docs/regressions/regressions-dl19-passage-wp.md) | [+](docs/regressions/regressions-dl20-passage-wp.md) | -| Huggingface WP baselines | [+](docs/regressions/regressions-msmarco-passage-hgf-wp.md) | [+](docs/regressions/regressions-dl19-passage-hgf-wp.md) | [+](docs/regressions/regressions-dl20-passage-hgf-wp.md) | -| doc2query | [+](docs/regressions/regressions-msmarco-passage-doc2query.md) | | | -| doc2query-T5 | [+](docs/regressions/regressions-msmarco-passage-docTTTTTquery.md) | [+](docs/regressions/regressions-dl19-passage-docTTTTTquery.md) | [+](docs/regressions/regressions-dl20-passage-docTTTTTquery.md) | -| **Learned Sparse (uniCOIL family)** | | | | -| uniCOIL noexp | [✓](docs/regressions/regressions-msmarco-passage-unicoil-noexp.md) | [✓](docs/regressions/regressions-dl19-passage-unicoil-noexp.md) | [✓](docs/regressions/regressions-dl20-passage-unicoil-noexp.md) | -| uniCOIL with doc2query-T5 | [✓](docs/regressions/regressions-msmarco-passage-unicoil.md) | [✓](docs/regressions/regressions-dl19-passage-unicoil.md) | [✓](docs/regressions/regressions-dl20-passage-unicoil.md) | -| uniCOIL with TILDE | [✓](docs/regressions/regressions-msmarco-passage-unicoil-tilde-expansion.md) | | | -| **Learned Sparse (other)** | | | | -| DeepImpact | [✓](docs/regressions/regressions-msmarco-passage-deepimpact.md) | | | -| SPLADEv2 | [✓](docs/regressions/regressions-msmarco-passage-distill-splade-max.md) | | | -| SPLADE++ CoCondenser-EnsembleDistil | [✓](docs/regressions/regressions-msmarco-passage-splade-pp-ed.md) | [✓](docs/regressions/regressions-dl19-passage-splade-pp-ed.md) | [✓](docs/regressions/regressions-dl20-passage-splade-pp-ed.md) | -| SPLADE++ CoCondenser-EnsembleDistil (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-splade-pp-ed-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-splade-pp-ed-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-splade-pp-ed-onnx.md) | -| SPLADE++ CoCondenser-SelfDistil | [✓](docs/regressions/regressions-msmarco-passage-splade-pp-sd.md) | [✓](docs/regressions/regressions-dl19-passage-splade-pp-sd.md) | [✓](docs/regressions/regressions-dl20-passage-splade-pp-sd.md) | -| SPLADE++ CoCondenser-SelfDistil (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-splade-pp-sd-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-splade-pp-sd-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-splade-pp-sd-onnx.md) | -| **Learned Dense** (HNSW) | | | | -| cosDPR-distil w/ HNSW fp32 | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw.md) | -| cosDPR-distil w/ HSNW fp32 (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-onnx.md) | -| cosDPR-distil w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8.md) | -| cosDPR-distil w/ HSNW int8 (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8-onnx.md) | -| BGE-base-en-v1.5 w/ HNSW fp32 | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw.md) | -| BGE-base-en-v1.5 w/ HNSW fp32 (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-onnx.md) | -| BGE-base-en-v1.5 w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8.md) | -| BGE-base-en-v1.5 w/ HNSW int8 (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.md) | -| OpenAI Ada2 w/ HNSW fp32 | [✓](docs/regressions/regressions-msmarco-passage-openai-ada2.md) | [✓](docs/regressions/regressions-dl19-passage-openai-ada2.md) | [✓](docs/regressions/regressions-dl20-passage-openai-ada2.md) | -| OpenAI Ada2 w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-passage-openai-ada2-int8.md) | [✓](docs/regressions/regressions-dl19-passage-openai-ada2-int8.md) | [✓](docs/regressions/regressions-dl20-passage-openai-ada2-int8.md) | -| **Learned Dense** (Inverted; experimental) | | | | -| cosDPR-distil w/ "fake words" | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-fw.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-fw.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-fw.md) | -| cosDPR-distil w/ "LexLSH" | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-lexlsh.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-lexlsh.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-lexlsh.md) | +| | dev | DL19 | DL20 | +|--------------------------------------------|:--------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------:| +| **Unsupervised Sparse** | | | | +| BoW baselines | [+](docs/regressions/regressions-msmarco-passage.md) | [+](docs/regressions/regressions-dl19-passage.md) | [+](docs/regressions/regressions-dl20-passage.md) | +| Quantized BM25 | [✓](docs/regressions/regressions-msmarco-passage-bm25-b8.md) | [✓](docs/regressions/regressions-dl19-passage-bm25-b8.md) | [✓](docs/regressions/regressions-dl20-passage-bm25-b8.md) | +| WP baselines | [+](docs/regressions/regressions-msmarco-passage-wp.md) | [+](docs/regressions/regressions-dl19-passage-wp.md) | [+](docs/regressions/regressions-dl20-passage-wp.md) | +| Huggingface WP baselines | [+](docs/regressions/regressions-msmarco-passage-hgf-wp.md) | [+](docs/regressions/regressions-dl19-passage-hgf-wp.md) | [+](docs/regressions/regressions-dl20-passage-hgf-wp.md) | +| doc2query | [+](docs/regressions/regressions-msmarco-passage-doc2query.md) | | | +| doc2query-T5 | [+](docs/regressions/regressions-msmarco-passage-docTTTTTquery.md) | [+](docs/regressions/regressions-dl19-passage-docTTTTTquery.md) | [+](docs/regressions/regressions-dl20-passage-docTTTTTquery.md) | +| **Learned Sparse (uniCOIL family)** | | | | +| uniCOIL noexp | [✓](docs/regressions/regressions-msmarco-passage-unicoil-noexp.md) | [✓](docs/regressions/regressions-dl19-passage-unicoil-noexp.md) | [✓](docs/regressions/regressions-dl20-passage-unicoil-noexp.md) | +| uniCOIL with doc2query-T5 | [✓](docs/regressions/regressions-msmarco-passage-unicoil.md) | [✓](docs/regressions/regressions-dl19-passage-unicoil.md) | [✓](docs/regressions/regressions-dl20-passage-unicoil.md) | +| uniCOIL with TILDE | [✓](docs/regressions/regressions-msmarco-passage-unicoil-tilde-expansion.md) | | | +| **Learned Sparse (other)** | | | | +| DeepImpact | [✓](docs/regressions/regressions-msmarco-passage-deepimpact.md) | | | +| SPLADEv2 | [✓](docs/regressions/regressions-msmarco-passage-distill-splade-max.md) | | | +| SPLADE++ CoCondenser-EnsembleDistil | [✓](docs/regressions/regressions-msmarco-passage-splade-pp-ed.md) | [✓](docs/regressions/regressions-dl19-passage-splade-pp-ed.md) | [✓](docs/regressions/regressions-dl20-passage-splade-pp-ed.md) | +| SPLADE++ CoCondenser-EnsembleDistil (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-splade-pp-ed-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-splade-pp-ed-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-splade-pp-ed-onnx.md) | +| SPLADE++ CoCondenser-SelfDistil | [✓](docs/regressions/regressions-msmarco-passage-splade-pp-sd.md) | [✓](docs/regressions/regressions-dl19-passage-splade-pp-sd.md) | [✓](docs/regressions/regressions-dl20-passage-splade-pp-sd.md) | +| SPLADE++ CoCondenser-SelfDistil (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-splade-pp-sd-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-splade-pp-sd-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-splade-pp-sd-onnx.md) | +| **Learned Dense** (HNSW) | | | | +| cosDPR-distil w/ HNSW fp32 | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw.md) | +| cosDPR-distil w/ HSNW fp32 (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-onnx.md) | +| cosDPR-distil w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8.md) | +| cosDPR-distil w/ HSNW int8 (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8-onnx.md) | +| BGE-base-en-v1.5 w/ HNSW fp32 | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw.md) | +| BGE-base-en-v1.5 w/ HNSW fp32 (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-onnx.md) | +| BGE-base-en-v1.5 w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8.md) | +| BGE-base-en-v1.5 w/ HNSW int8 (ONNX) | [✓](docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.md) | [✓](docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.md) | [✓](docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.md) | +| OpenAI Ada2 w/ HNSW fp32 | [✓](docs/regressions/regressions-msmarco-passage-openai-ada2.md) | [✓](docs/regressions/regressions-dl19-passage-openai-ada2.md) | [✓](docs/regressions/regressions-dl20-passage-openai-ada2.md) | +| OpenAI Ada2 w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-passage-openai-ada2-int8.md) | [✓](docs/regressions/regressions-dl19-passage-openai-ada2-int8.md) | [✓](docs/regressions/regressions-dl20-passage-openai-ada2-int8.md) | +| Cohere English v3 w/ HNSW fp32 | [✓](docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3-hnsw.md) | | | +| Cohere English v3 w/ HNSW int8 | [✓](docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3-hnsw-int8.md) | | | +| **Learned Dense** (Inverted; experimental) | | | | +| cosDPR-distil w/ "fake words" | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-fw.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-fw.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-fw.md) | +| cosDPR-distil w/ "LexLSH" | [✓](docs/regressions/regressions-msmarco-passage-cos-dpr-distil-lexlsh.md) | [✓](docs/regressions/regressions-dl19-passage-cos-dpr-distil-lexlsh.md) | [✓](docs/regressions/regressions-dl20-passage-cos-dpr-distil-lexlsh.md) | ### Available Corpora for Download diff --git a/docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.md b/docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.md index c853ecd481..2ebdc1f48c 100644 --- a/docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.md +++ b/docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.md @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2019 Deep Learning Track (Passage) -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8.md b/docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8.md index a8bfc19bdc..c6c0a11186 100644 --- a/docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8.md +++ b/docs/regressions/regressions-dl19-passage-bge-base-en-v1.5-hnsw-int8.md @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2019 Deep Learning Track (Passage) -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using pre-encoded queries) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8-onnx.md b/docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8-onnx.md index 647a65a5ba..0824c49085 100644 --- a/docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8-onnx.md +++ b/docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8-onnx.md @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2019 Deep Learning Track (Passage) -**Model**: cosDPR-distil with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: cosDPR-distil with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8.md b/docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8.md index 734cbf6693..ee8edb1010 100644 --- a/docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8.md +++ b/docs/regressions/regressions-dl19-passage-cos-dpr-distil-hnsw-int8.md @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2019 Deep Learning Track (Passage) -**Model**: cosDPR-distil with HNSW indexes (using pre-encoded queries) +**Model**: cosDPR-distil with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/docs/regressions/regressions-dl19-passage-openai-ada2-int8.md b/docs/regressions/regressions-dl19-passage-openai-ada2-int8.md index 16b0920376..8549e64e34 100644 --- a/docs/regressions/regressions-dl19-passage-openai-ada2-int8.md +++ b/docs/regressions/regressions-dl19-passage-openai-ada2-int8.md @@ -2,7 +2,7 @@ **NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. -**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW indexes +**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.md b/docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.md index c6876bae61..0372d238c4 100644 --- a/docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.md +++ b/docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.md @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2020 Deep Learning Track (Passage) -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8.md b/docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8.md index 80cea35f50..c20b059f88 100644 --- a/docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8.md +++ b/docs/regressions/regressions-dl20-passage-bge-base-en-v1.5-hnsw-int8.md @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2020 Deep Learning Track (Passage) -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using pre-encoded queries) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8-onnx.md b/docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8-onnx.md index e71541216f..ef9f8c5307 100644 --- a/docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8-onnx.md +++ b/docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8-onnx.md @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2020 Deep Learning Track (Passage) -**Model**: cosDPR-distil with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: cosDPR-distil with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8.md b/docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8.md index ae940e2309..7d76ca14d6 100644 --- a/docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8.md +++ b/docs/regressions/regressions-dl20-passage-cos-dpr-distil-hnsw-int8.md @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2020 Deep Learning Track (Passage) -**Model**: cosDPR-distil with HNSW indexes (using pre-encoded queries) +**Model**: cosDPR-distil with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/docs/regressions/regressions-dl20-passage-openai-ada2-int8.md b/docs/regressions/regressions-dl20-passage-openai-ada2-int8.md index e8e315a955..37b74f67e8 100644 --- a/docs/regressions/regressions-dl20-passage-openai-ada2-int8.md +++ b/docs/regressions/regressions-dl20-passage-openai-ada2-int8.md @@ -2,7 +2,7 @@ **NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. -**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW indexes +**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.md b/docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.md index 553f93ebf0..661ad5a0b4 100644 --- a/docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.md +++ b/docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.md @@ -1,6 +1,6 @@ # Anserini Regressions: MS MARCO Passage Ranking -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8.md b/docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8.md index fb473238ac..abeda2404d 100644 --- a/docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8.md +++ b/docs/regressions/regressions-msmarco-passage-bge-base-en-v1.5-hnsw-int8.md @@ -1,6 +1,6 @@ # Anserini Regressions: MS MARCO Passage Ranking -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using pre-encoded queries) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3-hnsw-int8.md b/docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3-hnsw-int8.md new file mode 100644 index 0000000000..826d1fab41 --- /dev/null +++ b/docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3-hnsw-int8.md @@ -0,0 +1,111 @@ +# Anserini Regressions: MS MARCO Passage Ranking + +**Model**: [Cohere embed-english-v3.0](https://docs.cohere.com/reference/embed) with HNSW quantized indexes (using pre-encoded queries) + +This page describes regression experiments, integrated into Anserini's regression testing framework, using the [Cohere embed-english-v3.0](https://docs.cohere.com/reference/embed) model on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking). + +In these experiments, we are using pre-encoded queries (i.e., cached results of query encoding). + +The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/msmarco-passage-cohere-embed-english-v3-hnsw-int8.yaml). +Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3-hnsw-int8.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. + +From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: + +```bash +python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cohere-embed-english-v3-hnsw-int8 +``` + +We make available a version of the MS MARCO Passage Corpus that has already been encoded with Cohere embed-english-v3.0. + +From any machine, the following command will download the corpus and perform the complete regression, end to end: + +```bash +python src/main/python/run_regression.py --download --index --verify --search --regression msmarco-passage-cohere-embed-english-v3-hnsw-int8 +``` + +The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results. + +## Corpus Download + +Download the corpus and unpack into `collections/`: + +```bash +wget https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-cohere-embed-english-v3.tar -P collections/ +tar xvf collections/msmarco-passage-cohere-embed-english-v3.tar -C collections/ +``` + +To confirm, `msmarco-passage-cohere-embed-english-v3.tar` is 38 GB and has MD5 checksum `6b7d9795806891b227378f6c290464a9`. +With the corpus downloaded, the following command will perform the remaining steps below: + +```bash +python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cohere-embed-english-v3-hnsw-int8 \ + --corpus-path collections/msmarco-passage-cohere-embed-english-v3 +``` + +## Indexing + +Sample indexing command, building HNSW indexes: + +```bash +target/appassembler/bin/IndexHnswDenseVectors \ + -collection JsonDenseVectorCollection \ + -input /path/to/msmarco-passage-cohere-embed-english-v3 \ + -generator HnswDenseVectorDocumentGenerator \ + -index indexes/lucene-hnsw.msmarco-passage-cohere-embed-english-v3-int8/ \ + -threads 16 -M 16 -efC 100 -noMerge -quantize.int8 \ + >& logs/log.msmarco-passage-cohere-embed-english-v3 & +``` + +The path `/path/to/msmarco-passage-cohere-embed-english-v3/` should point to the corpus downloaded above. +Upon completion, we should have an index with 8,841,823 documents. + +Note that here we are explicitly using Lucene's `NoMergePolicy` merge policy, which suppresses any merging of index segments. +This is because merging index segments is a costly operation and not worthwhile given our query set. +Furthermore, we are using Lucene's [Automatic Byte Quantization](https://www.elastic.co/search-labs/blog/articles/scalar-quantization-in-lucene) feature, which increase the on-disk footprint of the indexes since we're storing both the int8 quantized vectors and the float32 vectors, but only the int8 quantized vectors need to be loaded into memory. +See [issue #2292](https://github.com/castorini/anserini/issues/2292) for some experiments reporting the performance impact. + +## Retrieval + +Topics and qrels are stored [here](https://github.com/castorini/anserini-tools/tree/master/topics-and-qrels), which is linked to the Anserini repo as a submodule. +The regression experiments here evaluate on the 6980 dev set questions; see [this page](../../docs/experiments-msmarco-passage.md) for more details. + +After indexing has completed, you should be able to perform retrieval as follows using HNSW indexes: + +```bash +target/appassembler/bin/SearchHnswDenseVectors \ + -index indexes/lucene-hnsw.msmarco-passage-cohere-embed-english-v3-int8/ \ + -topics tools/topics-and-qrels/topics.msmarco-passage.dev-subset.cohere-embed-english-v3.jsonl.gz \ + -topicReader JsonIntVector \ + -output runs/run.msmarco-passage-cohere-embed-english-v3.cohere-embed-english-v3.topics.msmarco-passage.dev-subset.cohere-embed-english-v3.jsonl.txt \ + -generator VectorQueryGenerator -topicField vector -threads 16 -hits 1000 -efSearch 1000 & +``` + +Evaluation can be performed using `trec_eval`: + +```bash +target/appassembler/bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-cohere-embed-english-v3.cohere-embed-english-v3.topics.msmarco-passage.dev-subset.cohere-embed-english-v3.jsonl.txt +target/appassembler/bin/trec_eval -c -m map tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-cohere-embed-english-v3.cohere-embed-english-v3.topics.msmarco-passage.dev-subset.cohere-embed-english-v3.jsonl.txt +target/appassembler/bin/trec_eval -c -M 10 -m recip_rank tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-cohere-embed-english-v3.cohere-embed-english-v3.topics.msmarco-passage.dev-subset.cohere-embed-english-v3.jsonl.txt +target/appassembler/bin/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-cohere-embed-english-v3.cohere-embed-english-v3.topics.msmarco-passage.dev-subset.cohere-embed-english-v3.jsonl.txt +``` + +## Effectiveness + +With the above commands, you should be able to reproduce the following results: + +| **nDCG@10** | **cohere-embed-english-v3**| +|:-------------------------------------------------------------------------------------------------------------|-----------| +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.428 | +| **AP@1000** | **cohere-embed-english-v3**| +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.371 | +| **RR@10** | **cohere-embed-english-v3**| +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.365 | +| **R@1000** | **cohere-embed-english-v3**| +| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.974 | + +Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. +Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/msmarco-passage-cohere-embed-english-v3-hnsw-int8.yaml). + +## Reproduction Log[*](../../docs/reproducibility.md) + +To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3-hnsw-int8.template) and run `bin/build.sh` to rebuild the documentation. diff --git a/docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3.md b/docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3-hnsw.md similarity index 75% rename from docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3.md rename to docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3-hnsw.md index f745943d64..1202775092 100644 --- a/docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3.md +++ b/docs/regressions/regressions-msmarco-passage-cohere-embed-english-v3-hnsw.md @@ -6,8 +6,24 @@ This page describes regression experiments, integrated into Anserini's regressio In these experiments, we are using pre-encoded queries (i.e., cached results of query encoding). -The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/msmarco-passage-cohere-embed-english-v3.yaml). -Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. +The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/msmarco-passage-cohere-embed-english-v3-hnsw.yaml). +Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3-hnsw.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. + +From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: + +```bash +python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cohere-embed-english-v3-hnsw +``` + +We make available a version of the MS MARCO Passage Corpus that has already been encoded with Cohere embed-english-v3.0. + +From any machine, the following command will download the corpus and perform the complete regression, end to end: + +```bash +python src/main/python/run_regression.py --download --index --verify --search --regression msmarco-passage-cohere-embed-english-v3-hnsw +``` + +The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results. ## Corpus Download @@ -19,6 +35,12 @@ tar xvf collections/msmarco-passage-cohere-embed-english-v3.tar -C collections/ ``` To confirm, `msmarco-passage-cohere-embed-english-v3.tar` is 38 GB and has MD5 checksum `6b7d9795806891b227378f6c290464a9`. +With the corpus downloaded, the following command will perform the remaining steps below: + +```bash +python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cohere-embed-english-v3-hnsw \ + --corpus-path collections/msmarco-passage-cohere-embed-english-v3 +``` ## Indexing @@ -30,13 +52,16 @@ target/appassembler/bin/IndexHnswDenseVectors \ -input /path/to/msmarco-passage-cohere-embed-english-v3 \ -generator HnswDenseVectorDocumentGenerator \ -index indexes/lucene-hnsw.msmarco-passage-cohere-embed-english-v3/ \ - -threads 16 -M 16 -efC 100 \ + -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge \ >& logs/log.msmarco-passage-cohere-embed-english-v3 & ``` The path `/path/to/msmarco-passage-cohere-embed-english-v3/` should point to the corpus downloaded above. Upon completion, we should have an index with 8,841,823 documents. +Note that here we are explicitly using Lucene's `NoMergePolicy` merge policy, which suppresses any merging of index segments. +This is because merging index segments is a costly operation and not worthwhile given our query set. + ## Retrieval Topics and qrels are stored [here](https://github.com/castorini/anserini-tools/tree/master/topics-and-qrels), which is linked to the Anserini repo as a submodule. @@ -77,8 +102,8 @@ With the above commands, you should be able to reproduce the following results: | [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking) | 0.974 | Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. -Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/msmarco-passage-cohere-embed-english-v3.yaml). +Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/msmarco-passage-cohere-embed-english-v3-hnsw.yaml). ## Reproduction Log[*](../../docs/reproducibility.md) -To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3.template) and run `bin/build.sh` to rebuild the documentation. +To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3-hnsw.template) and run `bin/build.sh` to rebuild the documentation. diff --git a/docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8-onnx.md b/docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8-onnx.md index 40a098be7b..dce1b0cb6b 100644 --- a/docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8-onnx.md +++ b/docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8-onnx.md @@ -1,6 +1,6 @@ # Anserini Regressions: MS MARCO Passage Ranking -**Model**: cosDPR-distil with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: cosDPR-distil with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8.md b/docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8.md index 8742d62044..66bdb89bf6 100644 --- a/docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8.md +++ b/docs/regressions/regressions-msmarco-passage-cos-dpr-distil-hnsw-int8.md @@ -1,6 +1,6 @@ # Anserini Regressions: MS MARCO Passage Ranking -**Model**: cosDPR-distil with HNSW indexes (using pre-encoded queries) +**Model**: cosDPR-distil with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/docs/regressions/regressions-msmarco-passage-openai-ada2-int8.md b/docs/regressions/regressions-msmarco-passage-openai-ada2-int8.md index e637cd7d0b..bf1c658994 100644 --- a/docs/regressions/regressions-msmarco-passage-openai-ada2-int8.md +++ b/docs/regressions/regressions-msmarco-passage-openai-ada2-int8.md @@ -2,7 +2,7 @@ **NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. -**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW indexes +**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.template b/src/main/resources/docgen/templates/dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.template index be21dd1db0..23f2ba30b1 100644 --- a/src/main/resources/docgen/templates/dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.template +++ b/src/main/resources/docgen/templates/dl19-passage-bge-base-en-v1.5-hnsw-int8-onnx.template @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2019 Deep Learning Track (Passage) -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl19-passage-bge-base-en-v1.5-hnsw-int8.template b/src/main/resources/docgen/templates/dl19-passage-bge-base-en-v1.5-hnsw-int8.template index 643e2e75eb..0c2c2b1a44 100644 --- a/src/main/resources/docgen/templates/dl19-passage-bge-base-en-v1.5-hnsw-int8.template +++ b/src/main/resources/docgen/templates/dl19-passage-bge-base-en-v1.5-hnsw-int8.template @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2019 Deep Learning Track (Passage) -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using pre-encoded queries) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl19-passage-cos-dpr-distil-hnsw-int8-onnx.template b/src/main/resources/docgen/templates/dl19-passage-cos-dpr-distil-hnsw-int8-onnx.template index fd1fa8fa91..8ca952d8ca 100644 --- a/src/main/resources/docgen/templates/dl19-passage-cos-dpr-distil-hnsw-int8-onnx.template +++ b/src/main/resources/docgen/templates/dl19-passage-cos-dpr-distil-hnsw-int8-onnx.template @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2019 Deep Learning Track (Passage) -**Model**: cosDPR-distil with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: cosDPR-distil with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl19-passage-cos-dpr-distil-hnsw-int8.template b/src/main/resources/docgen/templates/dl19-passage-cos-dpr-distil-hnsw-int8.template index 0900647c40..248b8f6a36 100644 --- a/src/main/resources/docgen/templates/dl19-passage-cos-dpr-distil-hnsw-int8.template +++ b/src/main/resources/docgen/templates/dl19-passage-cos-dpr-distil-hnsw-int8.template @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2019 Deep Learning Track (Passage) -**Model**: cosDPR-distil with HNSW indexes (using pre-encoded queries) +**Model**: cosDPR-distil with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl19-passage-openai-ada2-int8.template b/src/main/resources/docgen/templates/dl19-passage-openai-ada2-int8.template index 99454b9d31..274bfc48e5 100644 --- a/src/main/resources/docgen/templates/dl19-passage-openai-ada2-int8.template +++ b/src/main/resources/docgen/templates/dl19-passage-openai-ada2-int8.template @@ -2,7 +2,7 @@ **NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. -**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW indexes +**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [TREC 2019 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.template b/src/main/resources/docgen/templates/dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.template index 8cb472a69f..daf66a3bab 100644 --- a/src/main/resources/docgen/templates/dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.template +++ b/src/main/resources/docgen/templates/dl20-passage-bge-base-en-v1.5-hnsw-int8-onnx.template @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2020 Deep Learning Track (Passage) -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl20-passage-bge-base-en-v1.5-hnsw-int8.template b/src/main/resources/docgen/templates/dl20-passage-bge-base-en-v1.5-hnsw-int8.template index b1bff69692..8ec6277285 100644 --- a/src/main/resources/docgen/templates/dl20-passage-bge-base-en-v1.5-hnsw-int8.template +++ b/src/main/resources/docgen/templates/dl20-passage-bge-base-en-v1.5-hnsw-int8.template @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2020 Deep Learning Track (Passage) -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using pre-encoded queries) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl20-passage-cos-dpr-distil-hnsw-int8-onnx.template b/src/main/resources/docgen/templates/dl20-passage-cos-dpr-distil-hnsw-int8-onnx.template index 9179919a51..df3c3a82c8 100644 --- a/src/main/resources/docgen/templates/dl20-passage-cos-dpr-distil-hnsw-int8-onnx.template +++ b/src/main/resources/docgen/templates/dl20-passage-cos-dpr-distil-hnsw-int8-onnx.template @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2020 Deep Learning Track (Passage) -**Model**: cosDPR-distil with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: cosDPR-distil with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl20-passage-cos-dpr-distil-hnsw-int8.template b/src/main/resources/docgen/templates/dl20-passage-cos-dpr-distil-hnsw-int8.template index e9e46d5fd7..6f2ec5fdec 100644 --- a/src/main/resources/docgen/templates/dl20-passage-cos-dpr-distil-hnsw-int8.template +++ b/src/main/resources/docgen/templates/dl20-passage-cos-dpr-distil-hnsw-int8.template @@ -1,6 +1,6 @@ # Anserini Regressions: TREC 2020 Deep Learning Track (Passage) -**Model**: cosDPR-distil with HNSW indexes (using pre-encoded queries) +**Model**: cosDPR-distil with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/dl20-passage-openai-ada2-int8.template b/src/main/resources/docgen/templates/dl20-passage-openai-ada2-int8.template index eea224f03e..e205721637 100644 --- a/src/main/resources/docgen/templates/dl20-passage-openai-ada2-int8.template +++ b/src/main/resources/docgen/templates/dl20-passage-openai-ada2-int8.template @@ -2,7 +2,7 @@ **NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. -**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW indexes +**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [TREC 2020 Deep Learning Track passage ranking task](https://trec.nist.gov/data/deep2019.html), as described in the following paper: diff --git a/src/main/resources/docgen/templates/msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.template b/src/main/resources/docgen/templates/msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.template index 8ab9f6edd3..83df36bfc5 100644 --- a/src/main/resources/docgen/templates/msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.template +++ b/src/main/resources/docgen/templates/msmarco-passage-bge-base-en-v1.5-hnsw-int8-onnx.template @@ -1,6 +1,6 @@ # Anserini Regressions: MS MARCO Passage Ranking -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/src/main/resources/docgen/templates/msmarco-passage-bge-base-en-v1.5-hnsw-int8.template b/src/main/resources/docgen/templates/msmarco-passage-bge-base-en-v1.5-hnsw-int8.template index 6164caca75..59436c7610 100644 --- a/src/main/resources/docgen/templates/msmarco-passage-bge-base-en-v1.5-hnsw-int8.template +++ b/src/main/resources/docgen/templates/msmarco-passage-bge-base-en-v1.5-hnsw-int8.template @@ -1,6 +1,6 @@ # Anserini Regressions: MS MARCO Passage Ranking -**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW indexes (using pre-encoded queries) +**Model**: [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the [BGE-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) model on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3-hnsw-int8.template b/src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3-hnsw-int8.template new file mode 100644 index 0000000000..1ff40fa029 --- /dev/null +++ b/src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3-hnsw-int8.template @@ -0,0 +1,89 @@ +# Anserini Regressions: MS MARCO Passage Ranking + +**Model**: [Cohere embed-english-v3.0](https://docs.cohere.com/reference/embed) with HNSW quantized indexes (using pre-encoded queries) + +This page describes regression experiments, integrated into Anserini's regression testing framework, using the [Cohere embed-english-v3.0](https://docs.cohere.com/reference/embed) model on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking). + +In these experiments, we are using pre-encoded queries (i.e., cached results of query encoding). + +The exact configurations for these regressions are stored in [this YAML file](${yaml}). +Note that this page is automatically generated from [this template](${template}) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. + +From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: + +```bash +python src/main/python/run_regression.py --index --verify --search --regression ${test_name} +``` + +We make available a version of the MS MARCO Passage Corpus that has already been encoded with Cohere embed-english-v3.0. + +From any machine, the following command will download the corpus and perform the complete regression, end to end: + +```bash +python src/main/python/run_regression.py --download --index --verify --search --regression ${test_name} +``` + +The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results. + +## Corpus Download + +Download the corpus and unpack into `collections/`: + +```bash +wget ${download_url} -P collections/ +tar xvf collections/${corpus}.tar -C collections/ +``` + +To confirm, `${corpus}.tar` is 38 GB and has MD5 checksum `${download_checksum}`. +With the corpus downloaded, the following command will perform the remaining steps below: + +```bash +python src/main/python/run_regression.py --index --verify --search --regression ${test_name} \ + --corpus-path collections/${corpus} +``` + +## Indexing + +Sample indexing command, building HNSW indexes: + +```bash +${index_cmds} +``` + +The path `/path/to/${corpus}/` should point to the corpus downloaded above. +Upon completion, we should have an index with 8,841,823 documents. + +Note that here we are explicitly using Lucene's `NoMergePolicy` merge policy, which suppresses any merging of index segments. +This is because merging index segments is a costly operation and not worthwhile given our query set. +Furthermore, we are using Lucene's [Automatic Byte Quantization](https://www.elastic.co/search-labs/blog/articles/scalar-quantization-in-lucene) feature, which increase the on-disk footprint of the indexes since we're storing both the int8 quantized vectors and the float32 vectors, but only the int8 quantized vectors need to be loaded into memory. +See [issue #2292](https://github.com/castorini/anserini/issues/2292) for some experiments reporting the performance impact. + +## Retrieval + +Topics and qrels are stored [here](https://github.com/castorini/anserini-tools/tree/master/topics-and-qrels), which is linked to the Anserini repo as a submodule. +The regression experiments here evaluate on the 6980 dev set questions; see [this page](${root_path}/docs/experiments-msmarco-passage.md) for more details. + +After indexing has completed, you should be able to perform retrieval as follows using HNSW indexes: + +```bash +${ranking_cmds} +``` + +Evaluation can be performed using `trec_eval`: + +```bash +${eval_cmds} +``` + +## Effectiveness + +With the above commands, you should be able to reproduce the following results: + +${effectiveness} + +Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run. +Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](${yaml}). + +## Reproduction Log[*](${root_path}/docs/reproducibility.md) + +To add to this reproduction log, modify [this template](${template}) and run `bin/build.sh` to rebuild the documentation. diff --git a/src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3.template b/src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3-hnsw.template similarity index 66% rename from src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3.template rename to src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3-hnsw.template index e3a1e1af99..7370c4fc7b 100644 --- a/src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3.template +++ b/src/main/resources/docgen/templates/msmarco-passage-cohere-embed-english-v3-hnsw.template @@ -9,6 +9,22 @@ In these experiments, we are using pre-encoded queries (i.e., cached results of The exact configurations for these regressions are stored in [this YAML file](${yaml}). Note that this page is automatically generated from [this template](${template}) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation. +From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end: + +```bash +python src/main/python/run_regression.py --index --verify --search --regression ${test_name} +``` + +We make available a version of the MS MARCO Passage Corpus that has already been encoded with Cohere embed-english-v3.0. + +From any machine, the following command will download the corpus and perform the complete regression, end to end: + +```bash +python src/main/python/run_regression.py --download --index --verify --search --regression ${test_name} +``` + +The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results. + ## Corpus Download Download the corpus and unpack into `collections/`: @@ -19,6 +35,12 @@ tar xvf collections/${corpus}.tar -C collections/ ``` To confirm, `${corpus}.tar` is 38 GB and has MD5 checksum `${download_checksum}`. +With the corpus downloaded, the following command will perform the remaining steps below: + +```bash +python src/main/python/run_regression.py --index --verify --search --regression ${test_name} \ + --corpus-path collections/${corpus} +``` ## Indexing @@ -31,6 +53,9 @@ ${index_cmds} The path `/path/to/${corpus}/` should point to the corpus downloaded above. Upon completion, we should have an index with 8,841,823 documents. +Note that here we are explicitly using Lucene's `NoMergePolicy` merge policy, which suppresses any merging of index segments. +This is because merging index segments is a costly operation and not worthwhile given our query set. + ## Retrieval Topics and qrels are stored [here](https://github.com/castorini/anserini-tools/tree/master/topics-and-qrels), which is linked to the Anserini repo as a submodule. diff --git a/src/main/resources/docgen/templates/msmarco-passage-cos-dpr-distil-hnsw-int8-onnx.template b/src/main/resources/docgen/templates/msmarco-passage-cos-dpr-distil-hnsw-int8-onnx.template index a7e21103f1..de54bd521e 100644 --- a/src/main/resources/docgen/templates/msmarco-passage-cos-dpr-distil-hnsw-int8-onnx.template +++ b/src/main/resources/docgen/templates/msmarco-passage-cos-dpr-distil-hnsw-int8-onnx.template @@ -1,6 +1,6 @@ # Anserini Regressions: MS MARCO Passage Ranking -**Model**: cosDPR-distil with HNSW indexes (using ONNX for on-the-fly query encoding) +**Model**: cosDPR-distil with HNSW quantized indexes (using ONNX for on-the-fly query encoding) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/src/main/resources/docgen/templates/msmarco-passage-cos-dpr-distil-hnsw-int8.template b/src/main/resources/docgen/templates/msmarco-passage-cos-dpr-distil-hnsw-int8.template index ae04c97479..8489b89171 100644 --- a/src/main/resources/docgen/templates/msmarco-passage-cos-dpr-distil-hnsw-int8.template +++ b/src/main/resources/docgen/templates/msmarco-passage-cos-dpr-distil-hnsw-int8.template @@ -1,6 +1,6 @@ # Anserini Regressions: MS MARCO Passage Ranking -**Model**: cosDPR-distil with HNSW indexes (using pre-encoded queries) +**Model**: cosDPR-distil with HNSW quantized indexes (using pre-encoded queries) This page describes regression experiments, integrated into Anserini's regression testing framework, using the cosDPR-distil model on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/src/main/resources/docgen/templates/msmarco-passage-openai-ada2-int8.template b/src/main/resources/docgen/templates/msmarco-passage-openai-ada2-int8.template index b9e3a3c5e5..fec329dff5 100644 --- a/src/main/resources/docgen/templates/msmarco-passage-openai-ada2-int8.template +++ b/src/main/resources/docgen/templates/msmarco-passage-openai-ada2-int8.template @@ -2,7 +2,7 @@ **NOTE:** We're currently having issues with this regression, which throws "Retried waiting for GCLocker too often" errors. -**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW indexes +**Model**: OpenAI-ada2 embeddings (using pre-encoded queries) with HNSW quantized indexes This page describes regression experiments, integrated into Anserini's regression testing framework, using OpenAI-ada2 embeddings on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking), as described in the following paper: diff --git a/src/main/resources/regression/msmarco-passage-cohere-embed-english-v3-hnsw-int8.yaml b/src/main/resources/regression/msmarco-passage-cohere-embed-english-v3-hnsw-int8.yaml new file mode 100644 index 0000000000..e4d65aef10 --- /dev/null +++ b/src/main/resources/regression/msmarco-passage-cohere-embed-english-v3-hnsw-int8.yaml @@ -0,0 +1,65 @@ +--- +corpus: msmarco-passage-cohere-embed-english-v3 +corpus_path: collections/msmarco/msmarco-passage-cohere-embed-english-v3/ + +download_url: https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-cohere-embed-english-v3.tar +download_checksum: 6b7d9795806891b227378f6c290464a9 + +index_path: indexes/lucene-hnsw.msmarco-passage-cohere-embed-english-v3-int8/ +index_type: hnsw +collection_class: JsonDenseVectorCollection +generator_class: HnswDenseVectorDocumentGenerator +index_threads: 16 +index_options: -M 16 -efC 100 -noMerge -quantize.int8 + +metrics: + - metric: nDCG@10 + command: target/appassembler/bin/trec_eval + params: -c -m ndcg_cut.10 + separator: "\t" + parse_index: 2 + metric_precision: 4 + can_combine: false + - metric: AP@1000 + command: target/appassembler/bin/trec_eval + params: -c -m map + separator: "\t" + parse_index: 2 + metric_precision: 4 + can_combine: false + - metric: RR@10 + command: target/appassembler/bin/trec_eval + params: -c -M 10 -m recip_rank + separator: "\t" + parse_index: 2 + metric_precision: 4 + can_combine: false + - metric: R@1000 + command: target/appassembler/bin/trec_eval + params: -c -m recall.1000 + separator: "\t" + parse_index: 2 + metric_precision: 4 + can_combine: false + +topic_reader: JsonIntVector +topics: + - name: "[MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking)" + id: dev + path: topics.msmarco-passage.dev-subset.cohere-embed-english-v3.jsonl.gz + qrel: qrels.msmarco-passage.dev-subset.txt + +models: + - name: cohere-embed-english-v3 + display: cohere-embed-english-v3 + type: hnsw + params: -generator VectorQueryGenerator -topicField vector -threads 16 -hits 1000 -efSearch 1000 + results: + nDCG@10: + - 0.4275 + AP@1000: + - 0.3706 + RR@10: + - 0.3648 + R@1000: + - 0.9735 diff --git a/src/main/resources/regression/msmarco-passage-cohere-embed-english-v3.yaml b/src/main/resources/regression/msmarco-passage-cohere-embed-english-v3-hnsw.yaml similarity index 96% rename from src/main/resources/regression/msmarco-passage-cohere-embed-english-v3.yaml rename to src/main/resources/regression/msmarco-passage-cohere-embed-english-v3-hnsw.yaml index 7c283b1748..a74f9a3bde 100644 --- a/src/main/resources/regression/msmarco-passage-cohere-embed-english-v3.yaml +++ b/src/main/resources/regression/msmarco-passage-cohere-embed-english-v3-hnsw.yaml @@ -10,7 +10,7 @@ index_type: hnsw collection_class: JsonDenseVectorCollection generator_class: HnswDenseVectorDocumentGenerator index_threads: 16 -index_options: -M 16 -efC 100 +index_options: -M 16 -efC 100 -memoryBuffer 65536 -noMerge metrics: - metric: nDCG@10 diff --git a/tools b/tools index fe0d5e7776..19c1984cda 160000 --- a/tools +++ b/tools @@ -1 +1 @@ -Subproject commit fe0d5e7776da48ea8a2ea12acc9bfc474cea7a17 +Subproject commit 19c1984cda8329cd3a09b9257771c47420042a26