Skip to content

Commit

Permalink
Adding Robust04 and TREC-NEWS multifield regressions in Anserini (#1844)
Browse files Browse the repository at this point in the history
* add: fiqa multifield regressions from beir benchmark

* Adding Robust04 multifield regression to Anserini

* Adding TREC-NEWS multifield regressions in Anserini

* Fixes robust04 to Robust04 (with caps)

* Fixes trec-news to TREC-NEWS (with caps)
  • Loading branch information
thakur-nandan authored Apr 22, 2022
1 parent 6ca065f commit 35d7801
Show file tree
Hide file tree
Showing 7 changed files with 342 additions and 2 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,8 @@ See individual pages for details!
+ HotpotQA: ["flat" baseline](docs/regressions-beir-v1.0.0-hotpotqa-flat.md), [SPLADE-distill CoCodenser-medium](docs/regressions-beir-v1.0.0-hotpotqa-splade-distil-cocodenser-medium.md)
+ FiQA-2018: ["flat" baseline](docs/regressions-beir-v1.0.0-fiqa-flat.md), ["multifield" baseline](docs/regressions-beir-v1.0.0-fiqa-multifield.md), [SPLADE-distill CoCodenser-medium](docs/regressions-beir-v1.0.0-fiqa-splade-distil-cocodenser-medium.md)
+ Signal-1M(RT): ["flat" baseline](docs/regressions-beir-v1.0.0-signal1m-flat.md)
+ TREC-NEWS: ["flat" baseline](docs/regressions-beir-v1.0.0-trec-news-flat.md)
+ Robust04: ["flat" baseline](docs/regressions-beir-v1.0.0-robust04-flat.md)
+ TREC-NEWS: ["flat" baseline](docs/regressions-beir-v1.0.0-trec-news-flat.md), ["multifield" baseline](docs/regressions-beir-v1.0.0-trec-news-multifield.md)
+ Robust04: ["flat" baseline](docs/regressions-beir-v1.0.0-robust04-flat.md), ["multifield" baseline](docs/regressions-beir-v1.0.0-robust04-multifield.md)
+ ArguAna: ["flat" baseline](docs/regressions-beir-v1.0.0-arguana-flat.md), ["multifield" baseline](docs/regressions-beir-v1.0.0-arguana-multifield.md), [SPLADE-distill CoCodenser-medium](docs/regressions-beir-v1.0.0-arguana-splade-distil-cocodenser-medium.md)
+ Touche2020: ["flat" baseline](docs/regressions-beir-v1.0.0-webis-touche2020-flat.md), [SPLADE-distill CoCodenser-medium](docs/regressions-beir-v1.0.0-webis-touche2020-splade-distil-cocodenser-medium.md)
+ CQADupStack-Android: ["flat" baseline](docs/regressions-beir-v1.0.0-cqadupstack-android-flat.md)
Expand Down
69 changes: 69 additions & 0 deletions docs/regressions-beir-v1.0.0-robust04-multifield.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Anserini Regressions: BEIR (v1.0.0) — Robust04

This page documents BM25 regression experiments for [BEIR (v1.0.0) — Robust04](http://beir.ai/).
These experiments index the "title" and "text" fields in corpus separately.
At retrieval time, a query is issued across both fields (equally weighted).

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/beir-v1.0.0-robust04-multifield.yaml).
Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/beir-v1.0.0-robust04-multifield.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.

From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:

```
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-robust04-multifield
```

## Indexing

Typical indexing command:

```
target/appassembler/bin/IndexCollection \
-collection BeirMultifieldCollection \
-input /path/to/beir-v1.0.0-robust04-multifield \
-index indexes/lucene-index.beir-v1.0.0-robust04-multifield/ \
-generator DefaultLuceneDocumentGenerator \
-threads 1 -storePositions -storeDocvectors -storeRaw -fields title \
>& logs/log.beir-v1.0.0-robust04-multifield &
```

For additional details, see explanation of [common indexing options](common-indexing-options.md).

## Retrieval

After indexing has completed, you should be able to perform retrieval as follows:

```
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.beir-v1.0.0-robust04-multifield/ \
-topics src/main/resources/topics-and-qrels/topics.beir-v1.0.0-robust04.test.tsv.gz \
-topicreader TsvString \
-output runs/run.beir-v1.0.0-robust04-multifield.bm25.topics.beir-v1.0.0-robust04.test.txt \
-bm25 -removeQuery -hits 1000 -fields contents=1.0 title=1.0 &
```

Evaluation can be performed using `trec_eval`:

```
tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-robust04.test.txt runs/run.beir-v1.0.0-robust04-multifield.bm25.topics.beir-v1.0.0-robust04.test.txt
tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-robust04.test.txt runs/run.beir-v1.0.0-robust04-multifield.bm25.topics.beir-v1.0.0-robust04.test.txt
tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-robust04.test.txt runs/run.beir-v1.0.0-robust04-multifield.bm25.topics.beir-v1.0.0-robust04.test.txt
```

## Effectiveness

With the above commands, you should be able to reproduce the following results:

| nDCG@10 | BM25 |
|:-------------------------------------------------------------------------------------------------------------|-----------|
| BEIR (v1.0.0): robust04 | 0.4070 |


| R@100 | BM25 |
|:-------------------------------------------------------------------------------------------------------------|-----------|
| BEIR (v1.0.0): robust04 | 0.3746 |


| R@1000 | BM25 |
|:-------------------------------------------------------------------------------------------------------------|-----------|
| BEIR (v1.0.0): robust04 | 0.6345 |
69 changes: 69 additions & 0 deletions docs/regressions-beir-v1.0.0-trec-news-multifield.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Anserini Regressions: BEIR (v1.0.0) — TREC-NEWS

This page documents BM25 regression experiments for [BEIR (v1.0.0) — TREC-NEWS](http://beir.ai/).
These experiments index the "title" and "text" fields in corpus separately.
At retrieval time, a query is issued across both fields (equally weighted).

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/beir-v1.0.0-trec-news-multifield.yaml).
Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/beir-v1.0.0-trec-news-multifield.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.

From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:

```
python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-trec-news-multifield
```

## Indexing

Typical indexing command:

```
target/appassembler/bin/IndexCollection \
-collection BeirMultifieldCollection \
-input /path/to/beir-v1.0.0-trec-news-multifield \
-index indexes/lucene-index.beir-v1.0.0-trec-news-multifield/ \
-generator DefaultLuceneDocumentGenerator \
-threads 1 -storePositions -storeDocvectors -storeRaw -fields title \
>& logs/log.beir-v1.0.0-trec-news-multifield &
```

For additional details, see explanation of [common indexing options](common-indexing-options.md).

## Retrieval

After indexing has completed, you should be able to perform retrieval as follows:

```
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.beir-v1.0.0-trec-news-multifield/ \
-topics src/main/resources/topics-and-qrels/topics.beir-v1.0.0-trec-news.test.tsv.gz \
-topicreader TsvString \
-output runs/run.beir-v1.0.0-trec-news-multifield.bm25.topics.beir-v1.0.0-trec-news.test.txt \
-bm25 -removeQuery -hits 1000 -fields contents=1.0 title=1.0 &
```

Evaluation can be performed using `trec_eval`:

```
tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-trec-news.test.txt runs/run.beir-v1.0.0-trec-news-multifield.bm25.topics.beir-v1.0.0-trec-news.test.txt
tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-trec-news.test.txt runs/run.beir-v1.0.0-trec-news-multifield.bm25.topics.beir-v1.0.0-trec-news.test.txt
tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.beir-v1.0.0-trec-news.test.txt runs/run.beir-v1.0.0-trec-news-multifield.bm25.topics.beir-v1.0.0-trec-news.test.txt
```

## Effectiveness

With the above commands, you should be able to reproduce the following results:

| nDCG@10 | BM25 |
|:-------------------------------------------------------------------------------------------------------------|-----------|
| BEIR (v1.0.0): trec-news | 0.3977 |


| R@100 | BM25 |
|:-------------------------------------------------------------------------------------------------------------|-----------|
| BEIR (v1.0.0): trec-news | 0.4216 |


| R@1000 | BM25 |
|:-------------------------------------------------------------------------------------------------------------|-----------|
| BEIR (v1.0.0): trec-news | 0.6993 |
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Anserini Regressions: BEIR (v1.0.0) — Robust04

This page documents BM25 regression experiments for [BEIR (v1.0.0) — Robust04](http://beir.ai/).
These experiments index the "title" and "text" fields in corpus separately.
At retrieval time, a query is issued across both fields (equally weighted).

The exact configurations for these regressions are stored in [this YAML file](${yaml}).
Note that this page is automatically generated from [this template](${template}) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.

From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:

```
python src/main/python/run_regression.py --index --verify --search --regression ${test_name}
```

## Indexing

Typical indexing command:

```
${index_cmds}
```

For additional details, see explanation of [common indexing options](common-indexing-options.md).

## Retrieval

After indexing has completed, you should be able to perform retrieval as follows:

```
${ranking_cmds}
```

Evaluation can be performed using `trec_eval`:

```
${eval_cmds}
```

## Effectiveness

With the above commands, you should be able to reproduce the following results:

${effectiveness}
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Anserini Regressions: BEIR (v1.0.0) — TREC-NEWS

This page documents BM25 regression experiments for [BEIR (v1.0.0) — TREC-NEWS](http://beir.ai/).
These experiments index the "title" and "text" fields in corpus separately.
At retrieval time, a query is issued across both fields (equally weighted).

The exact configurations for these regressions are stored in [this YAML file](${yaml}).
Note that this page is automatically generated from [this template](${template}) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.

From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:

```
python src/main/python/run_regression.py --index --verify --search --regression ${test_name}
```

## Indexing

Typical indexing command:

```
${index_cmds}
```

For additional details, see explanation of [common indexing options](common-indexing-options.md).

## Retrieval

After indexing has completed, you should be able to perform retrieval as follows:

```
${ranking_cmds}
```

Evaluation can be performed using `trec_eval`:

```
${eval_cmds}
```

## Effectiveness

With the above commands, you should be able to reproduce the following results:

${effectiveness}
57 changes: 57 additions & 0 deletions src/main/resources/regression/beir-v1.0.0-robust04-multifield.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
corpus: beir-v1.0.0-robust04-multifield
corpus_path: collections/beir-v1.0.0/corpus/robust04/

index_path: indexes/lucene-index.beir-v1.0.0-robust04-multifield/
collection_class: BeirMultifieldCollection
generator_class: DefaultLuceneDocumentGenerator
index_threads: 1
index_options: -storePositions -storeDocvectors -storeRaw -fields title
index_stats:
documents: 528036
documents (non-empty): 528036
total terms: 174384263

metrics:
- metric: nDCG@10
command: tools/eval/trec_eval.9.0.4/trec_eval
params: -c -m ndcg_cut.10
separator: "\t"
parse_index: 2
metric_precision: 4
can_combine: false
- metric: R@100
command: tools/eval/trec_eval.9.0.4/trec_eval
params: -c -m recall.100
separator: "\t"
parse_index: 2
metric_precision: 4
can_combine: false
- metric: R@1000
command: tools/eval/trec_eval.9.0.4/trec_eval
params: -c -m recall.1000
separator: "\t"
parse_index: 2
metric_precision: 4
can_combine: false

topic_reader: TsvString
topic_root: src/main/resources/topics-and-qrels/
qrels_root: src/main/resources/topics-and-qrels/
topics:
- name: "BEIR (v1.0.0): robust04"
id: test
path: topics.beir-v1.0.0-robust04.test.tsv.gz
qrel: qrels.beir-v1.0.0-robust04.test.txt

models:
- name: bm25
display: BM25
params: -bm25 -removeQuery -hits 1000 -fields contents=1.0 title=1.0
results:
nDCG@10:
- 0.4070
R@100:
- 0.3746
R@1000:
- 0.6345
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
corpus: beir-v1.0.0-trec-news-multifield
corpus_path: collections/beir-v1.0.0/corpus/trec-news/

index_path: indexes/lucene-index.beir-v1.0.0-trec-news-multifield/
collection_class: BeirMultifieldCollection
generator_class: DefaultLuceneDocumentGenerator
index_threads: 1
index_options: -storePositions -storeDocvectors -storeRaw -fields title
index_stats:
documents: 578605
documents (non-empty): 578572
total terms: 270886723

metrics:
- metric: nDCG@10
command: tools/eval/trec_eval.9.0.4/trec_eval
params: -c -m ndcg_cut.10
separator: "\t"
parse_index: 2
metric_precision: 4
can_combine: false
- metric: R@100
command: tools/eval/trec_eval.9.0.4/trec_eval
params: -c -m recall.100
separator: "\t"
parse_index: 2
metric_precision: 4
can_combine: false
- metric: R@1000
command: tools/eval/trec_eval.9.0.4/trec_eval
params: -c -m recall.1000
separator: "\t"
parse_index: 2
metric_precision: 4
can_combine: false

topic_reader: TsvString
topic_root: src/main/resources/topics-and-qrels/
qrels_root: src/main/resources/topics-and-qrels/
topics:
- name: "BEIR (v1.0.0): trec-news"
id: test
path: topics.beir-v1.0.0-trec-news.test.tsv.gz
qrel: qrels.beir-v1.0.0-trec-news.test.txt

models:
- name: bm25
display: BM25
params: -bm25 -removeQuery -hits 1000 -fields contents=1.0 title=1.0
results:
nDCG@10:
- 0.3977
R@100:
- 0.4216
R@1000:
- 0.6993

0 comments on commit 35d7801

Please sign in to comment.