-
Notifications
You must be signed in to change notification settings - Fork 478
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update regressions for DeepImpact and uniCOIL (#1657)
- Loading branch information
Showing
12 changed files
with
268 additions
and
37 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
91 changes: 91 additions & 0 deletions
91
docs/regressions-msmarco-passage-unicoil-tilde-expansion.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # Anserini: Regressions for uniCOIL w/ TILDE on [MS MARCO Passage](https://github.com/microsoft/MSMARCO-Passage-Ranking) | ||
|
|
||
| This page documents regression experiments for uniCOIL w/ TILDE document expansion on the MS MARCO Passage Ranking Task, which is integrated into Anserini's regression testing framework. | ||
| The model is described in the following paper: | ||
|
|
||
| > Shengyao Zhuang and Guido Zuccon. [Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion.](https://arxiv.org/pdf/2108.08513) _arXiv:2108.08513_. | ||
| For more complete instructions on how to run end-to-end experiments, refer to [this page](experiments-msmarco-passage-unicoil-tilde-expansion.md). | ||
|
|
||
| The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/msmarco-passage-unicoil-tilde-expansion.yaml). | ||
| Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/msmarco-passage-unicoil-tilde-expansion.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead. | ||
|
|
||
| ## Indexing | ||
|
|
||
| Typical indexing command: | ||
|
|
||
| ``` | ||
| nohup sh target/appassembler/bin/IndexCollection -collection JsonVectorCollection \ | ||
| -input /path/to/msmarco-passage-unicoil-tilde-expansion \ | ||
| -index indexes/lucene-index.msmarco-passage-unicoil-tilde-expansion \ | ||
| -generator DefaultLuceneDocumentGenerator \ | ||
| -threads 16 -impact -pretokenized \ | ||
| >& logs/log.msmarco-passage-unicoil-tilde-expansion & | ||
| ``` | ||
|
|
||
| The directory `/path/to/msmarco-passage-unicoil-tilde-expansion/` should be a directory containing the compressed `jsonl` files that comprise the corpus. | ||
| See [this page](experiments-msmarco-passage-unicoil-tilde-expansion.md) for additional details. | ||
|
|
||
| For additional details, see explanation of [common indexing options](common-indexing-options.md). | ||
|
|
||
| ## Retrieval | ||
|
|
||
| Topics and qrels are stored in [`src/main/resources/topics-and-qrels/`](../src/main/resources/topics-and-qrels/). | ||
| The regression experiments here evaluate on the 6980 dev set questions; see [this page](experiments-msmarco-passage.md) for more details. | ||
|
|
||
| After indexing has completed, you should be able to perform retrieval as follows: | ||
|
|
||
| ``` | ||
| nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.msmarco-passage-unicoil-tilde-expansion \ | ||
| -topicreader TsvInt -topics src/main/resources/topics-and-qrels/topics.msmarco-passage.dev-subset.unicoil-tilde-expansion.tsv.gz \ | ||
| -output runs/run.msmarco-passage-unicoil-tilde-expansion.unicoil.topics.msmarco-passage.dev-subset.unicoil-tilde-expansion.tsv.gz \ | ||
| -impact -pretokenized & | ||
| ``` | ||
|
|
||
| Evaluation can be performed using `trec_eval`: | ||
|
|
||
| ``` | ||
| tools/eval/trec_eval.9.0.4/trec_eval -m map -c -m recip_rank -c -m recall.1000 -c src/main/resources/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-unicoil-tilde-expansion.unicoil.topics.msmarco-passage.dev-subset.unicoil-tilde-expansion.tsv.gz | ||
| ``` | ||
|
|
||
| ## Effectiveness | ||
|
|
||
| With the above commands, you should be able to reproduce the following results: | ||
|
|
||
| MAP | uniCOIL | | ||
| :---------------------------------------|-----------| | ||
| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking)| 0.3560 | | ||
|
|
||
|
|
||
| MRR | uniCOIL | | ||
| :---------------------------------------|-----------| | ||
| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking)| 0.3606 | | ||
|
|
||
|
|
||
| R@1000 | uniCOIL | | ||
| :---------------------------------------|-----------| | ||
| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking)| 0.9646 | | ||
|
|
||
| The above runs are in TREC output format and evaluated with `trec_eval`. | ||
| In order to reproduce results reported in the paper, we need to convert to MS MARCO output format and then evaluate: | ||
|
|
||
| ```bash | ||
| python tools/scripts/msmarco/convert_trec_to_msmarco_run.py \ | ||
| --input runs/run.msmarco-passage-unicoil-tilde-expansion.unicoil.topics.msmarco-passage.dev-subset.unicoil-tilde-expansion.tsv.gz \ | ||
| --output runs/run.msmarco-passage-unicoil-tilde-expansion.unicoil.topics.msmarco-passage.dev-subset.unicoil-tilde-expansion.tsv.gz.msmarco --quiet | ||
|
|
||
| python tools/scripts/msmarco/msmarco_passage_eval.py \ | ||
| tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt \ | ||
| runs/run.msmarco-passage-unicoil-tilde-expansion.unicoil.topics.msmarco-passage.dev-subset.unicoil-tilde-expansion.tsv.gz.msmarco | ||
| ``` | ||
|
|
||
| The results should be as follows: | ||
|
|
||
| ``` | ||
| ##################### | ||
| MRR @10: 0.34957184927457136 | ||
| QueriesRanked: 6980 | ||
| ##################### | ||
| ``` | ||
|
|
||
| This corresponds to the effectiveness reported in the paper. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
71 changes: 71 additions & 0 deletions
71
src/main/resources/docgen/templates/msmarco-passage-unicoil-tilde-expansion.template
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # Anserini: Regressions for uniCOIL w/ TILDE on [MS MARCO Passage](https://github.com/microsoft/MSMARCO-Passage-Ranking) | ||
|
|
||
| This page documents regression experiments for uniCOIL w/ TILDE document expansion on the MS MARCO Passage Ranking Task, which is integrated into Anserini's regression testing framework. | ||
| The model is described in the following paper: | ||
|
|
||
| > Shengyao Zhuang and Guido Zuccon. [Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion.](https://arxiv.org/pdf/2108.08513) _arXiv:2108.08513_. | ||
|
|
||
| For more complete instructions on how to run end-to-end experiments, refer to [this page](experiments-msmarco-passage-unicoil-tilde-expansion.md). | ||
|
|
||
| The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/msmarco-passage-unicoil-tilde-expansion.yaml). | ||
| Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/msmarco-passage-unicoil-tilde-expansion.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead. | ||
|
|
||
| ## Indexing | ||
|
|
||
| Typical indexing command: | ||
|
|
||
| ``` | ||
| ${index_cmds} | ||
| ``` | ||
|
|
||
| The directory `/path/to/msmarco-passage-unicoil-tilde-expansion/` should be a directory containing the compressed `jsonl` files that comprise the corpus. | ||
| See [this page](experiments-msmarco-passage-unicoil-tilde-expansion.md) for additional details. | ||
|
|
||
| For additional details, see explanation of [common indexing options](common-indexing-options.md). | ||
|
|
||
| ## Retrieval | ||
|
|
||
| Topics and qrels are stored in [`src/main/resources/topics-and-qrels/`](../src/main/resources/topics-and-qrels/). | ||
| The regression experiments here evaluate on the 6980 dev set questions; see [this page](experiments-msmarco-passage.md) for more details. | ||
|
|
||
| After indexing has completed, you should be able to perform retrieval as follows: | ||
|
|
||
| ``` | ||
| ${ranking_cmds} | ||
| ``` | ||
|
|
||
| Evaluation can be performed using `trec_eval`: | ||
|
|
||
| ``` | ||
| ${eval_cmds} | ||
| ``` | ||
|
|
||
| ## Effectiveness | ||
|
|
||
| With the above commands, you should be able to reproduce the following results: | ||
|
|
||
| ${effectiveness} | ||
|
|
||
| The above runs are in TREC output format and evaluated with `trec_eval`. | ||
| In order to reproduce results reported in the paper, we need to convert to MS MARCO output format and then evaluate: | ||
|
|
||
| ```bash | ||
| python tools/scripts/msmarco/convert_trec_to_msmarco_run.py \ | ||
| --input runs/run.msmarco-passage-unicoil-tilde-expansion.unicoil.topics.msmarco-passage.dev-subset.unicoil-tilde-expansion.tsv.gz \ | ||
| --output runs/run.msmarco-passage-unicoil-tilde-expansion.unicoil.topics.msmarco-passage.dev-subset.unicoil-tilde-expansion.tsv.gz.msmarco --quiet | ||
|
|
||
| python tools/scripts/msmarco/msmarco_passage_eval.py \ | ||
| tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt \ | ||
| runs/run.msmarco-passage-unicoil-tilde-expansion.unicoil.topics.msmarco-passage.dev-subset.unicoil-tilde-expansion.tsv.gz.msmarco | ||
| ``` | ||
|
|
||
| The results should be as follows: | ||
|
|
||
| ``` | ||
| ##################### | ||
| MRR @10: 0.34957184927457136 | ||
| QueriesRanked: 6980 | ||
| ##################### | ||
| ``` | ||
|
|
||
| This corresponds to the effectiveness reported in the paper. |
Oops, something went wrong.