Skip to content

Commit

Permalink
Combine robust04 regression into disk45 regression (#1693)
Browse files Browse the repository at this point in the history
  • Loading branch information
@lintool
lintool authored Dec 13, 2021
1 parent 07b6b42 commit 64f4d1e
Show file tree
Hide file tree
Showing 8 changed files with 75 additions and 265 deletions.
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Anserini is designed to support experiments on various standard IR test collecti
The following experiments are backed by [rigorous end-to-end regression tests](docs/regressions.md) with [`run_regression.py`](src/main/python/run_regression.py) and [the Anserini reproducibility promise](docs/regressions.md).
For the most part, these runs are based on [_default_ parameter settings](https://github.com/castorini/Anserini/blob/master/src/main/java/io/anserini/search/SearchArgs.java).

+ Regressions for [Disks 1 & 2 (TREC 1-3)](docs/regressions-disk12.md), [Disks 4 & 5 (TREC 7-8)](docs/regressions-disk45.md), [Robust04](docs/regressions-robust04.md), [AQUAINT (Robust05)](docs/regressions-robust05.md)
+ Regressions for [Disks 1 & 2 (TREC 1-3)](docs/regressions-disk12.md), [Disks 4 & 5 (TREC 7-8, Robust04)](docs/regressions-disk45.md), [AQUAINT (Robust05)](docs/regressions-robust05.md)
+ Regressions for [the New York Times Corpus (Core17)](docs/regressions-core17.md), [the Washington Post Corpus (Core18)](docs/regressions-core18.md)
+ Regressions for [Wt10g](docs/regressions-wt10g.md), [Gov2](docs/regressions-gov2.md)
+ Regressions for [ClueWeb09 (Category B)](docs/regressions-cw09b.md), [ClueWeb12-B13](docs/regressions-cw12b13.md), [ClueWeb12](docs/regressions-cw12.md)
Expand Down
45 changes: 45 additions & 0 deletions docs/regressions-disk45.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,10 @@ nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk4
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt \
-output runs/run.disk45.bm25.topics.adhoc.401-450.txt \
-bm25 &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.robust04.txt \
-output runs/run.disk45.bm25.topics.robust04.txt \
-bm25 &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt \
Expand All @@ -51,6 +55,10 @@ nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk4
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt \
-output runs/run.disk45.bm25+rm3.topics.adhoc.401-450.txt \
-bm25 -rm3 &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.robust04.txt \
-output runs/run.disk45.bm25+rm3.topics.robust04.txt \
-bm25 -rm3 &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt \
Expand All @@ -60,6 +68,10 @@ nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk4
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt \
-output runs/run.disk45.bm25+ax.topics.adhoc.401-450.txt \
-bm25 -axiom -axiom.deterministic -rerankCutoff 20 &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.robust04.txt \
-output runs/run.disk45.bm25+ax.topics.robust04.txt \
-bm25 -axiom -axiom.deterministic -rerankCutoff 20 &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt \
Expand All @@ -69,6 +81,10 @@ nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk4
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt \
-output runs/run.disk45.ql.topics.adhoc.401-450.txt \
-qld &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.robust04.txt \
-output runs/run.disk45.ql.topics.robust04.txt \
-qld &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt \
Expand All @@ -78,6 +94,10 @@ nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk4
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt \
-output runs/run.disk45.ql+rm3.topics.adhoc.401-450.txt \
-qld -rm3 &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.robust04.txt \
-output runs/run.disk45.ql+rm3.topics.robust04.txt \
-qld -rm3 &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt \
Expand All @@ -87,28 +107,38 @@ nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk4
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt \
-output runs/run.disk45.ql+ax.topics.adhoc.401-450.txt \
-qld -axiom -axiom.deterministic -rerankCutoff 20 &
nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.disk45.pos+docvectors+raw \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.robust04.txt \
-output runs/run.disk45.ql+ax.topics.robust04.txt \
-qld -axiom -axiom.deterministic -rerankCutoff 20 &
```

Evaluation can be performed using `trec_eval`:

```
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.351-400.txt runs/run.disk45.bm25.topics.adhoc.351-400.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.bm25.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.bm25.topics.robust04.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.351-400.txt runs/run.disk45.bm25+rm3.topics.adhoc.351-400.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.bm25+rm3.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.bm25+rm3.topics.robust04.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.351-400.txt runs/run.disk45.bm25+ax.topics.adhoc.351-400.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.bm25+ax.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.bm25+ax.topics.robust04.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.351-400.txt runs/run.disk45.ql.topics.adhoc.351-400.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.ql.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.ql.topics.robust04.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.351-400.txt runs/run.disk45.ql+rm3.topics.adhoc.351-400.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.ql+rm3.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.ql+rm3.topics.robust04.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.351-400.txt runs/run.disk45.ql+ax.topics.adhoc.351-400.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.ql+ax.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.ql+ax.topics.robust04.txt
```

## Effectiveness
Expand All @@ -119,9 +149,24 @@ MAP | BM25 | +RM3 | +Ax | QL
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC-7 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt)| 0.1862 | 0.2354 | 0.2431 | 0.1843 | 0.2168 | 0.2298 |
[TREC-8 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt)| 0.2515 | 0.2750 | 0.2812 | 0.2460 | 0.2702 | 0.2647 |
[TREC 2004 Robust Track Topics](../src/main/resources/topics-and-qrels/topics.robust04.txt)| 0.2531 | 0.2903 | 0.2896 | 0.2467 | 0.2747 | 0.2774 |


P30 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC-7 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt)| 0.3093 | 0.3447 | 0.3287 | 0.3073 | 0.3307 | 0.3193 |
[TREC-8 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt)| 0.3560 | 0.3760 | 0.3753 | 0.3480 | 0.3680 | 0.3500 |
[TREC 2004 Robust Track Topics](../src/main/resources/topics-and-qrels/topics.robust04.txt)| 0.3102 | 0.3365 | 0.3333 | 0.3079 | 0.3232 | 0.3229 |

## Reproduction Log[*](reproducibility.md)

(Prior to the addition of TREC 7/8 topics)

+ Results reproduced by [@chriskamphuis](https://github.com/chriskamphuis) on 2018-12-18 (commit [`a15235`](https://github.com/castorini/Anserini/commit/a152359435ac6ae694b39f561343bba5eed8fdc9))
+ Results reproduced by [@kelvin-jiang](https://github.com/kelvin-jiang) on 2019-09-08 (commit [`a1892ae`](https://github.com/castorini/anserini/commit/a1892aec726efe55111a7bc501ab0914afab3a30))
+ Results reproduced by [@JMMackenzie](https://github.com/JMMackenzie) on 2020-01-21 (commit [`f63cd22`](https://github.com/castorini/anserini/commit/f63cd2275fa5a9d4da2d17e5f983a3308e8b50ce))
+ Results reproduced by [@nikhilro](https://github.com/nikhilro) on 2020-01-26 (commit [`d5ee069`](https://github.com/castorini/anserini/commit/d5ee069399e6a306d7685bda756c1f19db721156))
+ Results reproduced by [@edwinzhng](https://github.com/edwinzhng) on 2020-01-26 (commit [`7b76dfb`](https://github.com/castorini/anserini/commit/7b76dfbea7e0c01a3a5dc13e74f54852c780ec9b))
+ Results reproduced by [@yuki617](https://github.com/yuki617) on 2020-05-17 (commit [`cee4463`](https://github.com/castorini/anserini/commit/cee446338137415899436f0b2f2d738769745cde))
+ Results reproduced by [@x65han](https://github.com/x65han) on 2020-05-19 (commit [`33b0684`](https://github.com/castorini/anserini/commit/33b068437c4582067486e5fe79dfbecb8d4a145c))
+ Results reproduced by [@yxzhu16](https://github.com/yxzhu16) on 2020-07-17 (commit [`fad12be`](https://github.com/castorini/anserini/commit/fad12be2e37a075100707c3a674eb67bc0aa57ef))
104 changes: 0 additions & 104 deletions docs/regressions-robust04.md
View file

This file was deleted.

1 change: 0 additions & 1 deletion docs/regressions.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ Copy and paste the following lines into console to run the regressions from the
```
nohup python src/main/python/run_regression.py --index --collection disk12 >& logs/log.disk12 &
nohup python src/main/python/run_regression.py --index --collection disk45 >& logs/log.disk45 &
nohup python src/main/python/run_regression.py --index --collection robust04 >& logs/log.robust04 &
nohup python src/main/python/run_regression.py --index --collection robust05 >& logs/log.robust05 &
nohup python src/main/python/run_regression.py --index --collection core17 >& logs/log.core17 &
nohup python src/main/python/run_regression.py --index --collection core18 >& logs/log.core18 &
Expand Down
Loading

0 comments on commit 64f4d1e

Please sign in to comment.