Add MS MARCO (V2) regressions for doc2query-T5 (#1744)

castorini · Jan 20, 2022 · 1be47b9 · 1be47b9
1 parent d7a5d6c
commit 1be47b9
Show file tree

Hide file tree

Showing 40 changed files with 1,823 additions and 24 deletions.
diff --git a/README.md b/README.md
@@ -65,14 +65,20 @@ For the most part, these runs are based on [_default_ parameter settings](https:
   + Document ranking, complete doc[*](docs/experiments-msmarco-doc-doc2query-details.md): [baselines](docs/regressions-dl20-doc.md), [doc2query-T5](docs/regressions-dl20-doc-docTTTTTquery.md)
   + Document ranking, segmented doc[*](docs/experiments-msmarco-doc-doc2query-details.md): [baselines](docs/regressions-dl20-doc-segmented.md), [doc2query-T5](docs/regressions-dl20-doc-segmented-docTTTTTquery.md)
 + Regressions for MS MARCO (V2) Passage Ranking:
-  + Bag-of-words models: [baselines](docs/regressions-msmarco-v2-passage.md), [on augmented corpus](docs/regressions-msmarco-v2-passage-augmented.md)
+  + Bag-of-words models, original corpus: [baselines](docs/regressions-msmarco-v2-passage.md), [doc2query-T5](docs/regressions-msmarco-v2-passage-d2q-t5.md)
+  + Bag-of-words models, augmented corpus: [baselines](docs/regressions-msmarco-v2-passage-augmented.md), [doc2query-T5](docs/regressions-msmarco-v2-passage-augmented-d2q-t5.md)
   + Sparse learned models: [uniCOIL noexp zero-shot](docs/regressions-msmarco-v2-passage-unicoil-noexp-0shot.md)
 + Regressions for MS MARCO (V2) Document Ranking:
-  + Bag-of-words models: [baselines](docs/regressions-msmarco-v2-doc.md), [on segmented corpus](docs/regressions-msmarco-v2-doc-segmented.md)
+  + Bag-of-words, complete doc: [baselines](docs/regressions-msmarco-v2-doc.md), [doc2query-T5](docs/regressions-msmarco-v2-doc-d2q-t5.md)
+  + Bag-of-words, segmented doc: [baselines](docs/regressions-msmarco-v2-doc-segmented.md), [doc2query-T5](docs/regressions-msmarco-v2-doc-segmented-d2q-t5.md)
   + Sparse learned models: [uniCOIL noexp zero-shot](docs/regressions-msmarco-v2-doc-segmented-unicoil-noexp-0shot.md)
 + Regressions for TREC 2021 Deep Learning Track:
-  + Passage ranking: [baselines](docs/regressions-dl21-passage.md), [on augmented corpus](docs/regressions-dl21-passage-augmented.md), [uniCOIL noexp zero-shot](docs/regressions-dl21-passage-unicoil-noexp-0shot.md)
-  + Document ranking: [baselines](docs/regressions-dl21-doc.md), [on segmented corpus](docs/regressions-dl21-doc-segmented.md), [uniCOIL noexp zero-shot](docs/regressions-dl21-doc-segmented-unicoil-noexp-0shot.md)
+  + Passage ranking, original corpus: [baselines](docs/regressions-dl21-passage.md), [doc2query-T5](docs/regressions-dl21-passage-d2q-t5.md)
+  + Passage ranking, augmented corpus: [baselines](docs/regressions-dl21-passage-augmented.md), [doc2query-T5](docs/regressions-dl21-passage-augmented-d2q-t5.md)
+  + Passage ranking, sparse learned models: [uniCOIL noexp zero-shot](docs/regressions-dl21-passage-unicoil-noexp-0shot.md)
+  + Document ranking, complete doc: [baselines](docs/regressions-dl21-doc.md), [doc2query-T5](docs/regressions-dl21-doc-d2q-t5.md)
+  + Document ranking, segmented doc: [baselines](docs/regressions-dl21-doc-segmented.md), [doc2query-T5](docs/regressions-dl21-doc-segmented-d2q-t5.md)
+  + Document ranking, sparse learned models: [uniCOIL noexp zero-shot](docs/regressions-dl21-doc-segmented-unicoil-noexp-0shot.md)
 + Regressions for TREC News Tracks (Background Linking Task): [2018](docs/regressions-backgroundlinking18.md), [2019](docs/regressions-backgroundlinking19.md), [2020](docs/regressions-backgroundlinking20.md)
 + Regressions for [FEVER Fact Verification](docs/regressions-fever.md)
 + Regressions for [NTCIR-8 ACLIA (IR4QA subtask, Monolingual Chinese)](docs/regressions-ntcir8-zh.md)

diff --git a/docs/regressions-dl21-doc-d2q-t5.md b/docs/regressions-dl21-doc-d2q-t5.md
@@ -0,0 +1,106 @@
+# Anserini: Regressions for [DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning.html)
+
+This page describes experiments, integrated into Anserini's regression testing framework, for the TREC 2021 Deep Learning Track (Document Ranking Task) on the MS MARCO V2 document collection (with doc2query-T5 expansions) using relevance judgments from NIST.
+
+At the time this regression was created (November 2021), the qrels are only available to TREC participants.
+You must download the qrels from NIST's "active participants" password-protected site and place at `src/main/resources/topics-and-qrels/qrels.dl21-doc.txt`.
+The qrels will be added to Anserini when they are publicly released in Spring 2022.
+
+Note that the NIST relevance judgments provide far more relevant documents per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast).
+For additional instructions on working with MS MARCO V2 document collection, refer to [this page](experiments-msmarco-v2.md).
+
+Note that there are four different regression conditions for this task, and this page describes the following:
+
++ **Indexing Condition:** each document in the MS MARCO V2 document collection is treated as a unit of indexing
++ **Expansion Condition:** doc2query-T5
+
+The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl21-doc-d2q-t5.yaml).
+Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/dl21-doc-d2q-t5.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.
+
+## Indexing
+
+Typical indexing command:
+
+```
+target/appassembler/bin/IndexCollection \
+  -collection MsMarcoV2DocCollection \
+  -input /path/to/msmarco-v2-doc-d2q-t5 \
+  -index indexes/lucene-index.msmarco-v2-doc-d2q-t5/ \
+  -generator DefaultLuceneDocumentGenerator \
+  -threads 24 -storePositions -storeDocvectors -storeRaw \
+  >& logs/log.msmarco-v2-doc-d2q-t5 &
+```
+
+The value of `-input` should be a directory containing the compressed `jsonl` files that comprise the corpus.
+See [this page](experiments-msmarco-v2.md) for additional details.
+
+For additional details, see explanation of [common indexing options](common-indexing-options.md).
+
+## Retrieval
+
+Topics and qrels are stored in [`src/main/resources/topics-and-qrels/`](../src/main/resources/topics-and-qrels/).
+The regression experiments here evaluate on the 57 topics for which NIST has provided judgments as part of the TREC 2021 Deep Learning Track.
+<!-- The original data can be found [here](https://trec.nist.gov/data/deep2021.html). -->
+
+After indexing has completed, you should be able to perform retrieval as follows:
+
+```
+target/appassembler/bin/SearchCollection \
+  -index indexes/lucene-index.msmarco-v2-doc-d2q-t5/ \
+  -topics src/main/resources/topics-and-qrels/topics.dl21.txt -topicreader TsvInt \
+  -output runs/run.msmarco-v2-doc-d2q-t5.bm25-default.topics.dl21.txt \
+  -hits 1000 -bm25 &
+
+target/appassembler/bin/SearchCollection \
+  -index indexes/lucene-index.msmarco-v2-doc-d2q-t5/ \
+  -topics src/main/resources/topics-and-qrels/topics.dl21.txt -topicreader TsvInt \
+  -output runs/run.msmarco-v2-doc-d2q-t5.bm25-default+rm3.topics.dl21.txt \
+  -hits 1000 -bm25 -rm3 &
+```
+
+Evaluation can be performed using `trec_eval`:
+
+```
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-d2q-t5.bm25-default.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-d2q-t5.bm25-default.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-d2q-t5.bm25-default.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-d2q-t5.bm25-default.topics.dl21.txt
+
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-d2q-t5.bm25-default+rm3.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-d2q-t5.bm25-default+rm3.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-d2q-t5.bm25-default+rm3.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-d2q-t5.bm25-default+rm3.topics.dl21.txt
+```
+
+## Effectiveness
+
+With the above commands, you should be able to reproduce the following results:
+
+MAP@100                                 | BM25 (default)| +RM3      |
+:---------------------------------------|-----------|-----------|
+[DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning)| 0.2387    | 0.2608    |
+
+
+MRR@100                                 | BM25 (default)| +RM3      |
+:---------------------------------------|-----------|-----------|
+[DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning)| 0.8866    | 0.8342    |
+
+
+nDCG@10                                 | BM25 (default)| +RM3      |
+:---------------------------------------|-----------|-----------|
+[DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning)| 0.5792    | 0.5392    |
+
+
+R@100                                   | BM25 (default)| +RM3      |
+:---------------------------------------|-----------|-----------|
+[DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning)| 0.3443    | 0.3580    |
+
+
+R@1000                                  | BM25 (default)| +RM3      |
+:---------------------------------------|-----------|-----------|
+[DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning)| 0.7066    | 0.7572    |
+
+Some of these regressions correspond to official TREC 2021 Deep Learning Track "baseline" submissions:
+
++ `d_bm25` = BM25 (default), `k1=0.9`, `b=0.4`
++ `d_bm25rm3` = BM25 (default) + RM3, `k1=0.9`, `b=0.4`
diff --git a/docs/regressions-dl21-doc-segmented-d2q-t5.md b/docs/regressions-dl21-doc-segmented-d2q-t5.md
@@ -0,0 +1,106 @@
+# Anserini: Regressions for [DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning.html)
+
+This page describes experiments, integrated into Anserini's regression testing framework, for the TREC 2021 Deep Learning Track (Document Ranking Task) on the MS MARCO V2 _segmented_ document collection (with doc2query-T5 expansions) using relevance judgments from NIST.
+
+At the time this regression was created (November 2021), the qrels are only available to TREC participants.
+You must download the qrels from NIST's "active participants" password-protected site and place at `src/main/resources/topics-and-qrels/qrels.dl21-doc.txt`.
+The qrels will be added to Anserini when they are publicly released in Spring 2022.
+
+Note that the NIST relevance judgments provide far more relevant documents per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast).
+For additional instructions on working with MS MARCO V2 document collection, refer to [this page](experiments-msmarco-v2.md).
+
+Note that there are four different regression conditions for this task, and this page describes the following:
+
++ **Indexing Condition:** each segment in the MS MARCO V2 _segmented_ document collection is treated as a unit of indexing
++ **Expansion Condition:** doc2query-T5
+
+The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl21-doc-segmented-d2q-t5.yaml).
+Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/dl21-doc-segmented-d2q-t5.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.
+
+## Indexing
+
+Typical indexing command:
+
+```
+target/appassembler/bin/IndexCollection \
+  -collection MsMarcoV2DocCollection \
+  -input /path/to/msmarco-v2-doc-segmented-d2q-t5 \
+  -index indexes/lucene-index.msmarco-v2-doc-segmented-d2q-t5/ \
+  -generator DefaultLuceneDocumentGenerator \
+  -threads 24 -storePositions -storeDocvectors -storeRaw \
+  >& logs/log.msmarco-v2-doc-segmented-d2q-t5 &
+```
+
+The value of `-input` should be a directory containing the compressed `jsonl` files that comprise the corpus.
+See [this page](experiments-msmarco-v2.md) for additional details.
+
+For additional details, see explanation of [common indexing options](common-indexing-options.md).
+
+## Retrieval
+
+Topics and qrels are stored in [`src/main/resources/topics-and-qrels/`](../src/main/resources/topics-and-qrels/).
+The regression experiments here evaluate on the 57 topics for which NIST has provided judgments as part of the TREC 2021 Deep Learning Track.
+<!-- The original data can be found [here](https://trec.nist.gov/data/deep2021.html). -->
+
+After indexing has completed, you should be able to perform retrieval as follows:
+
+```
+target/appassembler/bin/SearchCollection \
+  -index indexes/lucene-index.msmarco-v2-doc-segmented-d2q-t5/ \
+  -topics src/main/resources/topics-and-qrels/topics.dl21.txt -topicreader TsvInt \
+  -output runs/run.msmarco-v2-doc-segmented-d2q-t5.bm25-default.topics.dl21.txt \
+  -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 1000 -bm25 &
+
+target/appassembler/bin/SearchCollection \
+  -index indexes/lucene-index.msmarco-v2-doc-segmented-d2q-t5/ \
+  -topics src/main/resources/topics-and-qrels/topics.dl21.txt -topicreader TsvInt \
+  -output runs/run.msmarco-v2-doc-segmented-d2q-t5.bm25-default+rm3.topics.dl21.txt \
+  -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 1000 -bm25 -rm3 &
+```
+
+Evaluation can be performed using `trec_eval`:
+
+```
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-segmented-d2q-t5.bm25-default.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-segmented-d2q-t5.bm25-default.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-segmented-d2q-t5.bm25-default.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-segmented-d2q-t5.bm25-default.topics.dl21.txt
+
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-segmented-d2q-t5.bm25-default+rm3.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-segmented-d2q-t5.bm25-default+rm3.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-segmented-d2q-t5.bm25-default+rm3.topics.dl21.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl21-doc.txt runs/run.msmarco-v2-doc-segmented-d2q-t5.bm25-default+rm3.topics.dl21.txt
+```
+
+## Effectiveness
+
+With the above commands, you should be able to reproduce the following results:
+
+MAP@100                                 | BM25 (default)| +RM3      |
+:---------------------------------------|-----------|-----------|
+[DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning)| 0.2683    | 0.3192    |
+
+
+MRR@100                                 | BM25 (default)| +RM3      |
+:---------------------------------------|-----------|-----------|
+[DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning)| 0.9454    | 0.8960    |
+
+
+nDCG@10                                 | BM25 (default)| +RM3      |
+:---------------------------------------|-----------|-----------|
+[DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning)| 0.6289    | 0.6555    |
+
+
+R@100                                   | BM25 (default)| +RM3      |
+:---------------------------------------|-----------|-----------|
+[DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning)| 0.3656    | 0.4119    |
+
+
+R@1000                                  | BM25 (default)| +RM3      |
+:---------------------------------------|-----------|-----------|
+[DL21 (Doc)](https://microsoft.github.io/msmarco/TREC-Deep-Learning)| 0.7202    | 0.7941    |
+
+Some of these regressions correspond to official TREC 2021 Deep Learning Track "baseline" submissions:
+
++ `dseg_bm25` = BM25 (default), `k1=0.9`, `b=0.4`
++ `dseg_bm25rm3` = BM25 (default) + RM3, `k1=0.9`, `b=0.4`
diff --git a/docs/regressions-dl21-doc-segmented-unicoil-noexp-0shot.md b/docs/regressions-dl21-doc-segmented-unicoil-noexp-0shot.md
@@ -8,7 +8,7 @@ You must download the qrels from NIST's "active participants" password-protected
 The qrels will be added to Anserini when they are publicly released in Spring 2022.
 
 Note that the NIST relevance judgments provide far more relevant documents per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast).
-For additional instructions on working with MS MARCO document collection, refer to [this page](experiments-msmarco-v2.md).
+For additional instructions on working with MS MARCO V2 document collection, refer to [this page](experiments-msmarco-v2.md).
 
 The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl21-doc-segmented-unicoil-noexp-0shot.yaml).
 Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/dl21-doc-segmented-unicoil-noexp-0shot.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.

diff --git a/docs/regressions-dl21-doc-segmented.md b/docs/regressions-dl21-doc-segmented.md
@@ -7,7 +7,7 @@ You must download the qrels from NIST's "active participants" password-protected
 The qrels will be added to Anserini when they are publicly released in Spring 2022.
 
 Note that the NIST relevance judgments provide far more relevant documents per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast).
-For additional instructions on working with MS MARCO document collection, refer to [this page](experiments-msmarco-v2.md).
+For additional instructions on working with MS MARCO V2 document collection, refer to [this page](experiments-msmarco-v2.md).
 
 Note that there are four different regression conditions for this task, and this page describes the following:
 

diff --git a/docs/regressions-dl21-doc.md b/docs/regressions-dl21-doc.md
@@ -7,7 +7,7 @@ You must download the qrels from NIST's "active participants" password-protected
 The qrels will be added to Anserini when they are publicly released in Spring 2022.
 
 Note that the NIST relevance judgments provide far more relevant documents per topic, unlike the "sparse" judgments provided by Microsoft (these are sometimes called "dense" judgments to emphasize this contrast).
-For additional instructions on working with MS MARCO document collection, refer to [this page](experiments-msmarco-v2.md).
+For additional instructions on working with MS MARCO V2 document collection, refer to [this page](experiments-msmarco-v2.md).
 
 Note that there are four different regression conditions for this task, and this page describes the following: