Skip to content

Commit

Permalink
Rename beir-dataset-wp to beir-dataset-flat-wp, update README.md (#1886)
Browse files Browse the repository at this point in the history 
1. Rename beir-dataset-wp to beir-dataset-flat-wp
2. Add missing docs for some datasets
3. Add beir-flat-wp docs to README.md
4. Update beir table format in README.md

Co-authored-by: Jimmy Lin <jimmylin@uwaterloo.ca>
  • Loading branch information
@amyxie361 @lintool
amyxie361 and lintool authored May 25, 2022
1 parent 060685a commit 30c9974
Show file tree
Hide file tree
Showing 88 changed files with 1,036 additions and 356 deletions.
62 changes: 31 additions & 31 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -120,37 +120,37 @@ See individual pages for details!
+ MF = "multfield" baseline
+ SPLADE = SPLADE-distill CoCodenser-medium

| Corpus | Conditions |
|--------|------------|
| TREC-COVID | [[F](docs/regressions-beir-v1.0.0-trec-covid-flat.md)] [[MF](docs/regressions-beir-v1.0.0-trec-covid-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-trec-covid-splade-distil-cocodenser-medium.md)] |
| BioASQ | [[F](docs/regressions-beir-v1.0.0-bioasq-flat.md)] [[MF](docs/regressions-beir-v1.0.0-bioasq-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-bioasq-splade-distil-cocodenser-medium.md)] |
| NFCorpus | [[F](docs/regressions-beir-v1.0.0-nfcorpus-flat.md)] [[MF](docs/regressions-beir-v1.0.0-nfcorpus-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-nfcorpus-splade-distil-cocodenser-medium.md)] |
| NQ | [[F](docs/regressions-beir-v1.0.0-nq-flat.md)] [[MF](docs/regressions-beir-v1.0.0-nq-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-nq-splade-distil-cocodenser-medium.md)] |
| HotpotQA | [[F](docs/regressions-beir-v1.0.0-hotpotqa-flat.md)] [[MF](docs/regressions-beir-v1.0.0-hotpotqa-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-hotpotqa-splade-distil-cocodenser-medium.md)] |
| FiQA-2018 | [[F](docs/regressions-beir-v1.0.0-fiqa-flat.md)] [[MF](docs/regressions-beir-v1.0.0-fiqa-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-fiqa-splade-distil-cocodenser-medium.md)] |
| Signal-1M(RT) | [[F](docs/regressions-beir-v1.0.0-signal1m-flat.md)] [[MF](docs/regressions-beir-v1.0.0-signal1m-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-signal1m-splade-distil-cocodenser-medium.md)] |
| TREC-NEWS | [[F](docs/regressions-beir-v1.0.0-trec-news-flat.md)] [[MF](docs/regressions-beir-v1.0.0-trec-news-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-trec-news-splade-distil-cocodenser-medium.md)] |
| Robust04 | [[F](docs/regressions-beir-v1.0.0-robust04-flat.md)] [[MF](docs/regressions-beir-v1.0.0-robust04-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-robust04-splade-distil-cocodenser-medium.md)] |
| ArguAna | [[F](docs/regressions-beir-v1.0.0-arguana-flat.md)] [[MF](docs/regressions-beir-v1.0.0-arguana-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-arguana-splade-distil-cocodenser-medium.md)] |
| Touche2020 | [[F](docs/regressions-beir-v1.0.0-webis-touche2020-flat.md)] [[MF](docs/regressions-beir-v1.0.0-arguana-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-webis-touche2020-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Android | [[F](docs/regressions-beir-v1.0.0-cqadupstack-android-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-android-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-android-splade-distil-cocodenser-medium.md)] |
| CQADupStack-English | [[F](docs/regressions-beir-v1.0.0-cqadupstack-english-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-english-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-english-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Gaming | [[F](docs/regressions-beir-v1.0.0-cqadupstack-gaming-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-gaming-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-gaming-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Gis | [[F](docs/regressions-beir-v1.0.0-cqadupstack-gis-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-gis-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-gis-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Mathematica | [[F](docs/regressions-beir-v1.0.0-cqadupstack-mathematica-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-mathematica-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-mathematica-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Physics | [[F](docs/regressions-beir-v1.0.0-cqadupstack-physics-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-physics-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-physics-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Programmers | [[F](docs/regressions-beir-v1.0.0-cqadupstack-programmers-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-programmers-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-programmers-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Stats | [[F](docs/regressions-beir-v1.0.0-cqadupstack-stats-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-stats-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-stats-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Tex | [[F](docs/regressions-beir-v1.0.0-cqadupstack-tex-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-tex-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-tex-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Unix | [[F](docs/regressions-beir-v1.0.0-cqadupstack-unix-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-unix-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-unix-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Webmasters | [[F](docs/regressions-beir-v1.0.0-cqadupstack-webmasters-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-webmasters-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-webmasters-splade-distil-cocodenser-medium.md)] |
| CQADupStack-Wordpress | [[F](docs/regressions-beir-v1.0.0-cqadupstack-wordpress-flat.md)] [[MF](docs/regressions-beir-v1.0.0-cqadupstack-wordpress-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-cqadupstack-wordpress-splade-distil-cocodenser-medium.md)] |
| Quora | [[F](docs/regressions-beir-v1.0.0-quora-flat.md)] [[MF](docs/regressions-beir-v1.0.0-quora-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-quora-splade-distil-cocodenser-medium.md)] |
| DBPedia | [[F](docs/regressions-beir-v1.0.0-dbpedia-entity-flat.md)] [[MF](docs/regressions-beir-v1.0.0-dbpedia-entity-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-dbpedia-entity-splade-distil-cocodenser-medium.md)] |
| SCIDOCS | [[F](docs/regressions-beir-v1.0.0-scidocs-flat.md)] [[MF](docs/regressions-beir-v1.0.0-multifield-flat.md)] [[SPLADE](docs/regressions-beir-v1.0.0-scidocs-splade-distil-cocodenser-medium.md)] |
| FEVER | [[F](docs/regressions-beir-v1.0.0-fever-flat.md)] [[MF](docs/regressions-beir-v1.0.0-fever-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-fever-splade-distil-cocodenser-medium.md)] |
| Climate-FEVER | [[F](docs/regressions-beir-v1.0.0-climate-fever-flat.md)] [[MF](docs/regressions-beir-v1.0.0-climate-fever-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-climate-fever-splade-distil-cocodenser-medium.md)] |
| SciFact | [[F](docs/regressions-beir-v1.0.0-scifact-flat.md)] [[MF](docs/regressions-beir-v1.0.0-scifact-multifield.md)] [[SPLADE](docs/regressions-beir-v1.0.0-scifact-splade-distil-cocodenser-medium.md)] |
| Corpus | flat | multifield | flat-wp | SPLADE |
|--------|:----:|:----------:|:-------:|:------:|
| TREC-COVID | [+](docs/regressions-beir-v1.0.0-trec-covid-flat.md) | [+](docs/regressions-beir-v1.0.0-trec-covid-multifield.md) | [+](docs/regressions-beir-v1.0.0-trec-covid-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-trec-covid-splade-distil-cocodenser-medium.md) |
| BioASQ | [+](docs/regressions-beir-v1.0.0-bioasq-flat.md) | [+](docs/regressions-beir-v1.0.0-bioasq-multifield.md) | [+](docs/regressions-beir-v1.0.0-bioasq-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-bioasq-splade-distil-cocodenser-medium.md) |
| NFCorpus | [+](docs/regressions-beir-v1.0.0-nfcorpus-flat.md) | [+](docs/regressions-beir-v1.0.0-nfcorpus-multifield.md) | [+](docs/regressions-beir-v1.0.0-nfcorpus-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-nfcorpus-splade-distil-cocodenser-medium.md) |
| NQ | [+](docs/regressions-beir-v1.0.0-nq-flat.md) | [+](docs/regressions-beir-v1.0.0-nq-multifield.md) | [+](docs/regressions-beir-v1.0.0-nq-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-nq-splade-distil-cocodenser-medium.md) |
| HotpotQA | [+](docs/regressions-beir-v1.0.0-hotpotqa-flat.md) | [+](docs/regressions-beir-v1.0.0-hotpotqa-multifield.md) | [+](docs/regressions-beir-v1.0.0-hotpotqa-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-hotpotqa-splade-distil-cocodenser-medium.md) |
| FiQA-2018 | [+](docs/regressions-beir-v1.0.0-fiqa-flat.md) | [+](docs/regressions-beir-v1.0.0-fiqa-multifield.md) | [+](docs/regressions-beir-v1.0.0-fiqa-flat-wp.md) |[+](docs/regressions-beir-v1.0.0-fiqa-splade-distil-cocodenser-medium.md) |
| Signal-1M(RT) | [+](docs/regressions-beir-v1.0.0-signal1m-flat.md) | [+](docs/regressions-beir-v1.0.0-signal1m-multifield.md) | [+](docs/regressions-beir-v1.0.0-signal1m-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-signal1m-splade-distil-cocodenser-medium.md) |
| TREC-NEWS | [+](docs/regressions-beir-v1.0.0-trec-news-flat.md) | [+](docs/regressions-beir-v1.0.0-trec-news-multifield.md) | [+](docs/regressions-beir-v1.0.0-trec-news-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-trec-news-splade-distil-cocodenser-medium.md) |
| Robust04 | [+](docs/regressions-beir-v1.0.0-robust04-flat.md) | [+](docs/regressions-beir-v1.0.0-robust04-multifield.md) | [+](docs/regressions-beir-v1.0.0-robust04-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-robust04-splade-distil-cocodenser-medium.md) |
| ArguAna | [+](docs/regressions-beir-v1.0.0-arguana-flat.md) | [+](docs/regressions-beir-v1.0.0-arguana-multifield.md) | [+](docs/regressions-beir-v1.0.0-arguana-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-arguana-splade-distil-cocodenser-medium.md) |
| Touche2020 | [+](docs/regressions-beir-v1.0.0-webis-touche2020-flat.md) | [+](docs/regressions-beir-v1.0.0-arguana-multifield.md) | [+](docs/regressions-beir-v1.0.0-webis-touche2020-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-webis-touche2020-splade-distil-cocodenser-medium.md) |
| CQADupStack-Android | [+](docs/regressions-beir-v1.0.0-cqadupstack-android-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-android-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-android-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-android-splade-distil-cocodenser-medium.md) |
| CQADupStack-English | [+](docs/regressions-beir-v1.0.0-cqadupstack-english-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-english-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-english-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-english-splade-distil-cocodenser-medium.md) |
| CQADupStack-Gaming | [+](docs/regressions-beir-v1.0.0-cqadupstack-gaming-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-gaming-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-gaming-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-gaming-splade-distil-cocodenser-medium.md) |
| CQADupStack-Gis | [+](docs/regressions-beir-v1.0.0-cqadupstack-gis-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-gis-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-gis-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-gis-splade-distil-cocodenser-medium.md) |
| CQADupStack-Mathematica | [+](docs/regressions-beir-v1.0.0-cqadupstack-mathematica-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-mathematica-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-mathematica-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-mathematica-splade-distil-cocodenser-medium.md) |
| CQADupStack-Physics | [+](docs/regressions-beir-v1.0.0-cqadupstack-physics-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-physics-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-physics-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-physics-splade-distil-cocodenser-medium.md) |
| CQADupStack-Programmers | [+](docs/regressions-beir-v1.0.0-cqadupstack-programmers-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-programmers-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-programmers-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-programmers-splade-distil-cocodenser-medium.md) |
| CQADupStack-Stats | [+](docs/regressions-beir-v1.0.0-cqadupstack-stats-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-stats-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-stats-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-stats-splade-distil-cocodenser-medium.md) |
| CQADupStack-Tex | [+](docs/regressions-beir-v1.0.0-cqadupstack-tex-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-tex-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-tex-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-tex-splade-distil-cocodenser-medium.md) |
| CQADupStack-Unix | [+](docs/regressions-beir-v1.0.0-cqadupstack-unix-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-unix-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-unix-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-unix-splade-distil-cocodenser-medium.md) |
| CQADupStack-Webmasters | [+](docs/regressions-beir-v1.0.0-cqadupstack-webmasters-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-webmasters-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-webmasters-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-webmasters-splade-distil-cocodenser-medium.md) |
| CQADupStack-Wordpress | [+](docs/regressions-beir-v1.0.0-cqadupstack-wordpress-flat.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-wordpress-multifield.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-wordpress-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-cqadupstack-wordpress-splade-distil-cocodenser-medium.md) |
| Quora | [+](docs/regressions-beir-v1.0.0-quora-flat.md) | [+](docs/regressions-beir-v1.0.0-quora-multifield.md) | [+](docs/regressions-beir-v1.0.0-quora-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-quora-splade-distil-cocodenser-medium.md) |
| DBPedia | [+](docs/regressions-beir-v1.0.0-dbpedia-entity-flat.md) | [+](docs/regressions-beir-v1.0.0-dbpedia-entity-multifield.md) | [+](docs/regressions-beir-v1.0.0-dbpedia-entity-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-dbpedia-entity-splade-distil-cocodenser-medium.md) |
| SCIDOCS | [+](docs/regressions-beir-v1.0.0-scidocs-flat.md) | [+](docs/regressions-beir-v1.0.0-multifield-flat.md) | [+](docs/regressions-beir-v1.0.0-scidocs-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-scidocs-splade-distil-cocodenser-medium.md) |
| FEVER | [+](docs/regressions-beir-v1.0.0-fever-flat.md) | [+](docs/regressions-beir-v1.0.0-fever-multifield.md) | [+](docs/regressions-beir-v1.0.0-fever-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-fever-splade-distil-cocodenser-medium.md) |
| Climate-FEVER | [+](docs/regressions-beir-v1.0.0-climate-fever-flat.md) | [+](docs/regressions-beir-v1.0.0-climate-fever-multifield.md) | [+](docs/regressions-beir-v1.0.0-climate-fever-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-climate-fever-splade-distil-cocodenser-medium.md) |
| SciFact | [+](docs/regressions-beir-v1.0.0-scifact-flat.md) | [+](docs/regressions-beir-v1.0.0-scifact-multifield.md) | [+](docs/regressions-beir-v1.0.0-scifact-flat-wp.md) | [+](docs/regressions-beir-v1.0.0-scifact-splade-distil-cocodenser-medium.md) |

### Other Regressions

Expand Down
22 changes: 11 additions & 11 deletions docs/regressions-beir-v1.0.0-arguana-wp.md → ...egressions-beir-v1.0.0-arguana-flat-wp.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ This page documents BM25 regression experiments for [BEIR (v1.0.0) &mdash; ArguA
These experiments index the corpus in a "flat" manner, by concatenating the "title" and "text" into the "contents" field.
All the documents and queries are pre-tokenized with `bert-base-uncased` tokenizer.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/beir-v1.0.0-arguana-wp.yaml).
Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/beir-v1.0.0-arguana-wp.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.
The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/beir-v1.0.0-arguana-flat-wp.yaml).
Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/beir-v1.0.0-arguana-flat-wp.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.

From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:

```
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana-wp
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana-flat-wp
```

## Indexing
Expand All @@ -20,11 +20,11 @@ Typical indexing command:
```
target/appassembler/bin/IndexCollection \
-collection BeirFlatCollection \
-input /path/to/beir-v1.0.0-arguana-wp \
-index indexes/lucene-index.beir-v1.0.0-arguana-wp/ \
-input /path/to/beir-v1.0.0-arguana-flat-wp \
-index indexes/lucene-index.beir-v1.0.0-arguana-flat-wp/ \
-generator DefaultLuceneDocumentGenerator \
-threads 1 -storePositions -storeDocvectors -storeRaw -pretokenized \
>& logs/log.beir-v1.0.0-arguana-wp &
>& logs/log.beir-v1.0.0-arguana-flat-wp &
```

For additional details, see explanation of [common indexing options](common-indexing-options.md).
Expand All @@ -35,19 +35,19 @@ After indexing has completed, you should be able to perform retrieval as follows

```
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.beir-v1.0.0-arguana-wp/ \
-index indexes/lucene-index.beir-v1.0.0-arguana-flat-wp/ \
-topics src/main/resources/topics-and-qrels/topics.beir-v1.0.0-arguana.test.wp.tsv.gz \
-topicreader TsvString \
-output runs/run.beir-v1.0.0-arguana-wp.bm25.topics.beir-v1.0.0-arguana.test.wp.txt \
-output runs/run.beir-v1.0.0-arguana-flat-wp.bm25.topics.beir-v1.0.0-arguana.test.wp.txt \
-bm25 -removeQuery -pretokenized &
```

Evaluation can be performed using `trec_eval`:

```
tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana-wp.bm25.topics.beir-v1.0.0-arguana.test.wp.txt
tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana-wp.bm25.topics.beir-v1.0.0-arguana.test.wp.txt
tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana-wp.bm25.topics.beir-v1.0.0-arguana.test.wp.txt
tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana-flat-wp.bm25.topics.beir-v1.0.0-arguana.test.wp.txt
tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana-flat-wp.bm25.topics.beir-v1.0.0-arguana.test.wp.txt
tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-arguana.test.txt runs/run.beir-v1.0.0-arguana-flat-wp.bm25.topics.beir-v1.0.0-arguana.test.wp.txt
```

## Effectiveness
Expand Down
Loading

0 comments on commit 30c9974

Please sign in to comment.