diff --git a/README.md b/README.md index e1fc7c8022..707d455851 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,11 @@ Note that these regressions capture the "out of the box" experience, based on [_ + [Regressions for the MS MARCO Passage Task with Doc2query expansion](docs/regressions-msmarco-passage-doc2query.md) + [Regressions for the MS MARCO Document Task](docs/regressions-msmarco-doc.md) + [Regressions for NTCIR-8 ACLIA (IR4QA subtask, Chinese monolingual)](docs/regressions-ntcir8-zh.md) ++ [Regressions for CLEF2006 Monolingual French](docs/regressions-clef06-fr.md) ++ [Regressions for TREC2002 Monolingual Arabic](docs/regressions-trec02-ar.md) ++ [Regressions for FIRE 2012 Monolingual Bengali](docs/regressions-fire12-bn.md) ++ [Regressions for FIRE 2012 Monolingual Hindi](docs/regressions-fire12-hi.md) ++ [Regressions for FIRE 2012 Monolingual English](docs/regressions-fire12-en.md) Other experiments: diff --git a/docs/regressions-clef06-fr.md b/docs/regressions-clef06-fr.md index 116c89c682..9e99943147 100644 --- a/docs/regressions-clef06-fr.md +++ b/docs/regressions-clef06-fr.md @@ -11,10 +11,9 @@ Note that this page is automatically generated from [this template](../src/main/ Typical indexing command: ``` -nohup sh target/appassembler/bin/IndexCollection -collection JsonCollection \ --generator LuceneDocumentGenerator -threads 16 -input /path/to/clef06-fr -index \ -lucene-index.clef06-fr.pos+docvectors+rawdocs -storePositions -storeDocvectors \ --storeRawDocs -language fr >& log.clef06-fr.pos+docvectors+rawdocs & +nohup sh target/appassembler/bin/IndexCollection -collection JsonCollection -input /path/to/clef06-fr \ + -index lucene-index.clef06-fr.pos+docvectors+rawdocs -generator LuceneDocumentGenerator -threads 16 \ + -storePositions -storeDocvectors -storeRawDocs -language fr >& log.clef06-fr.pos+docvectors+rawdocs & ``` The directory `/path/to/clef06-fr/` should be a directory containing the collection (the format is jsonline format). @@ -29,7 +28,9 @@ The regression experiments here evaluate on the 49 questions. After indexing has completed, you should be able to perform retrieval as follows: ``` -nohup target/appassembler/bin/SearchCollection -topicreader TsvString -index lucene-index.clef06-fr.pos+docvectors+rawdocs -topics src/main/resources/topics-and-qrels/topics.clef06fr.mono.fr.txt -output run.clef06-fr.bm25.topics.clef06fr.mono.fr.txt -language fr -bm25 & +nohup target/appassembler/bin/SearchCollection -index lucene-index.clef06-fr.pos+docvectors+rawdocs \ + -topicreader TsvString -topics src/main/resources/topics-and-qrels/topics.clef06fr.mono.fr.txt \ + -language fr -bm25 -output run.clef06-fr.bm25.topics.clef06fr.mono.fr.txt & ``` diff --git a/docs/regressions-fire12-bn.md b/docs/regressions-fire12-bn.md index e736c501a2..5c25968fa3 100644 --- a/docs/regressions-fire12-bn.md +++ b/docs/regressions-fire12-bn.md @@ -3,7 +3,7 @@ This page documents regression experiments for [FIRE 2012 Ad-hoc retrieval (Monolingual Bengali topic)](http://isical.ac.in/~fire/2012/adhoc.html). The document collection can be found in [FIRE 2012 data page](http://fire.irsi.res.in/fire/static/data). -The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire-bn.yaml). +The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire12-bn.yaml). Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/fire12-bn.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead. ## Indexing @@ -11,10 +11,9 @@ Note that this page is automatically generated from [this template](../src/main/ Typical indexing command: ``` -nohup sh target/appassembler/bin/IndexCollection -collection TrecCollection \ --generator LuceneDocumentGenerator -threads 16 -input /path/to/fire12-bn -index \ -lucene-index.fire12-hi.pos+docvectors+rawdocs -storePositions -storeDocvectors \ --storeRawDocs -language bn >& log.fire12-bn.pos+docvectors+rawdocs & +nohup sh target/appassembler/bin/IndexCollection -collection TrecCollection -input /path/to/fire12-bn \ + -index lucene-index.fire12-bn.pos+docvectors+rawdocs -generator LuceneDocumentGenerator -threads 16 \ + -storePositions -storeDocvectors -storeRawDocs -language bn >& log.fire12-bn.pos+docvectors+rawdocs & ``` The directory `/path/to/fire12-bn/` should be a directory containing the collection, containing `bn_ABP` and `bn_BDNews24` directories. @@ -29,14 +28,16 @@ The regression experiments here evaluate on the 50 questions. After indexing has completed, you should be able to perform retrieval as follows: ``` -nohup target/appassembler/bin/SearchCollection -topicreader TsvString -index lucene-index.fire12-bn.pos+docvectors+rawdocs -topics src/main/resources/topics-and-qrels/topics.fire12bn.176-225.txt -output run.fire12-bn.bm25.topics.fire12bn.176-225.txt -language bn -bm25 & +nohup target/appassembler/bin/SearchCollection -index lucene-index.fire12-bn.pos+docvectors+rawdocs \ + -topicreader TsvString -topics src/main/resources/topics-and-qrels/topics.fire12bn.176-225.txt \ + -language bn -bm25 -output run.fire12-bn.bm25.topics.fire12bn.176-225.txt & ``` Evaluation can be performed using `trec_eval`: ``` -eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.fire12bn.176-225.txt run.fire12-bn.bm25.topics.fire12bn.176-225.txt +eval/trec_eval.9.0.4/trec_eval -m map -m P.20 -m ndcg_cut.20 src/main/resources/topics-and-qrels/qrels.fire12bn.176-225.txt run.fire12-bn.bm25.topics.fire12bn.176-225.txt ``` @@ -46,11 +47,16 @@ With the above commands, you should be able to replicate the following results: MAP | BM25 | :---------------------------------------|-----------| -[FIRE2012 (Bengali monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.2881 | +[FIRE 2012 (Bengali monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.2881 | -P30 | BM25 | +P20 | BM25 | :---------------------------------------|-----------| -[FIRE2012 (Bengali monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.3360 | +[FIRE 2012 (Bengali monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.3740 | + + +NDCG20 | BM25 | +:---------------------------------------|-----------| +[FIRE 2012 (Bengali monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.4261 | diff --git a/docs/regressions-fire12-en.md b/docs/regressions-fire12-en.md index 46c70e9516..fe5857bd2f 100644 --- a/docs/regressions-fire12-en.md +++ b/docs/regressions-fire12-en.md @@ -3,7 +3,7 @@ This page documents regression experiments for [FIRE 2012 Ad-hoc retrieval (Monolingual English topic)](http://isical.ac.in/~fire/2012/adhoc.html). The document collection can be found in [FIRE 2012 data page](http://fire.irsi.res.in/fire/static/data). -The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire-en.yaml). +The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire12-en.yaml). Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/fire12-en.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead. ## Indexing @@ -11,10 +11,9 @@ Note that this page is automatically generated from [this template](../src/main/ Typical indexing command: ``` -nohup sh target/appassembler/bin/IndexCollection -collection TrecCollection \ --generator LuceneDocumentGenerator -threads 16 -input /path/to/fire12-en -index \ -lucene-index.fire12-en.pos+docvectors+rawdocs -storePositions -storeDocvectors \ --storeRawDocs -language en >& log.fire12-en.pos+docvectors+rawdocs & +nohup sh target/appassembler/bin/IndexCollection -collection TrecCollection -input /path/to/fire12-en \ + -index lucene-index.fire12-en.pos+docvectors+rawdocs -generator LuceneDocumentGenerator -threads 16 \ + -storePositions -storeDocvectors -storeRawDocs -language en >& log.fire12-en.pos+docvectors+rawdocs & ``` The directory `/path/to/fire12-en/` should be a directory containing the collection, containing `en_BDNews24` and `en_TheTelegraph_2001-2010` directories. @@ -29,14 +28,16 @@ The regression experiments here evaluate on the 50 questions. After indexing has completed, you should be able to perform retrieval as follows: ``` -nohup target/appassembler/bin/SearchCollection -topicreader TsvString -index lucene-index.fire12-en.pos+docvectors+rawdocs -topics src/main/resources/topics-and-qrels/topics.fire12en.176-225.txt -output run.fire12-en.bm25.topics.fire12en.176-225.txt -language en -bm25 & +nohup target/appassembler/bin/SearchCollection -index lucene-index.fire12-en.pos+docvectors+rawdocs \ + -topicreader TsvString -topics src/main/resources/topics-and-qrels/topics.fire12en.176-225.txt \ + -language en -bm25 -output run.fire12-en.bm25.topics.fire12en.176-225.txt & ``` Evaluation can be performed using `trec_eval`: ``` -eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.fire12en.176-225.txt run.fire12-en.bm25.topics.fire12en.176-225.txt +eval/trec_eval.9.0.4/trec_eval -m map -m P.20 -m ndcg_cut.20 src/main/resources/topics-and-qrels/qrels.fire12en.176-225.txt run.fire12-en.bm25.topics.fire12en.176-225.txt ``` @@ -46,11 +47,16 @@ With the above commands, you should be able to replicate the following results: MAP | BM25 | :---------------------------------------|-----------| -[FIRE2012 (English monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.3867 | +[FIRE 2012 (English monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.3713 | -P30 | BM25 | +P20 | BM25 | :---------------------------------------|-----------| -[FIRE2012 (English monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.3920 | +[FIRE 2012 (English monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.4970 | + + +NDCG20 | BM25 | +:---------------------------------------|-----------| +[FIRE 2012 (English monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.5420 | diff --git a/docs/regressions-fire12-hi.md b/docs/regressions-fire12-hi.md index a19b2e4d77..d300bef4ab 100644 --- a/docs/regressions-fire12-hi.md +++ b/docs/regressions-fire12-hi.md @@ -3,7 +3,7 @@ This page documents regression experiments for [FIRE 2012 Ad-hoc retrieval (Monolingual Hindi topic)](http://isical.ac.in/~fire/2012/adhoc.html). The document collection can be found in [FIRE 2012 data page](http://fire.irsi.res.in/fire/static/data). -The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire-hi.yaml). +The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire12-hi.yaml). Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/fire12-hi.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead. ## Indexing @@ -11,10 +11,9 @@ Note that this page is automatically generated from [this template](../src/main/ Typical indexing command: ``` -nohup sh target/appassembler/bin/IndexCollection -collection TrecCollection \ --generator LuceneDocumentGenerator -threads 16 -input /path/to/fire12-hi -index \ -lucene-index.fire12-hi.pos+docvectors+rawdocs -storePositions -storeDocvectors \ --storeRawDocs -language hi >& log.fire12-hi.pos+docvectors+rawdocs & +nohup sh target/appassembler/bin/IndexCollection -collection TrecCollection -input /path/to/fire12-hi \ + -index lucene-index.fire12-hi.pos+docvectors+rawdocs -generator LuceneDocumentGenerator -threads 16 \ + -storePositions -storeDocvectors -storeRawDocs -language hi >& log.fire12-hi.pos+docvectors+rawdocs & ``` The directory `/path/to/fire12-hi/` should be a directory containing the collection, containing `hi_AmarUjala` and `hi_NavbharatTimes` directories. @@ -29,14 +28,16 @@ The regression experiments here evaluate on the 50 questions. After indexing has completed, you should be able to perform retrieval as follows: ``` -nohup target/appassembler/bin/SearchCollection -topicreader TsvString -index lucene-index.fire12-hi.pos+docvectors+rawdocs -topics src/main/resources/topics-and-qrels/topics.fire12hi.176-225.txt -output run.fire12-hi.bm25.topics.fire12hi.176-225.txt -language hi -bm25 & +nohup target/appassembler/bin/SearchCollection -index lucene-index.fire12-hi.pos+docvectors+rawdocs \ + -topicreader TsvString -topics src/main/resources/topics-and-qrels/topics.fire12hi.176-225.txt \ + -language hi -bm25 -output run.fire12-hi.bm25.topics.fire12hi.176-225.txt & ``` Evaluation can be performed using `trec_eval`: ``` -eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.fire12hi.176-225.txt run.fire12-hi.bm25.topics.fire12hi.176-225.txt +eval/trec_eval.9.0.4/trec_eval -m map -m P.20 -m ndcg_cut.20 src/main/resources/topics-and-qrels/qrels.fire12hi.176-225.txt run.fire12-hi.bm25.topics.fire12hi.176-225.txt ``` @@ -46,11 +47,16 @@ With the above commands, you should be able to replicate the following results: MAP | BM25 | :---------------------------------------|-----------| -[FIRE2012 (Hindi monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.3867 | +[FIRE 2012 (Hindi monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.3867 | -P30 | BM25 | +P20 | BM25 | :---------------------------------------|-----------| -[FIRE2012 (Hindi monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.3920 | +[FIRE 2012 (Hindi monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.4470 | + + +NDCG20 | BM25 | +:---------------------------------------|-----------| +[FIRE 2012 (Hindi monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)| 0.5310 | diff --git a/docs/regressions-trec02-ar.md b/docs/regressions-trec02-ar.md index 07c84e38d6..a770fdd5fc 100644 --- a/docs/regressions-trec02-ar.md +++ b/docs/regressions-trec02-ar.md @@ -11,10 +11,9 @@ Note that this page is automatically generated from [this template](../src/main/ Typical indexing command: ``` -nohup sh target/appassembler/bin/IndexCollection -collection JsonCollection \ --generator LuceneDocumentGenerator -threads 16 -input /path/to/trec02-ar -index \ -lucene-index.trec02-ar.pos+docvectors+rawdocs -storePositions -storeDocvectors \ --storeRawDocs -language ar >& log.trec02-ar.pos+docvectors+rawdocs & +nohup sh target/appassembler/bin/IndexCollection -collection JsonCollection -input /path/to/trec02-ar \ + -index lucene-index.trec02-ar.pos+docvectors+rawdocs -generator LuceneDocumentGenerator -threads 16 \ + -storePositions -storeDocvectors -storeRawDocs -language ar >& log.trec02-ar.pos+docvectors+rawdocs & ``` The directory `/path/to/trec02-ar/` should be a directory containing the collection, 2337 gzipped files from LDC2007T38. @@ -29,7 +28,9 @@ The regression experiments here evaluate on the 50 questions. After indexing has completed, you should be able to perform retrieval as follows: ``` -nohup target/appassembler/bin/SearchCollection -topicreader TsvString -index lucene-index.trec02-ar.pos+docvectors+rawdocs -topics src/main/resources/topics-and-qrels/topics.trec02ar.mono.ar.txt -output run.trec02-ar.bm25.topics.trec02ar.mono.ar.txt -language ar -bm25 & +nohup target/appassembler/bin/SearchCollection -index lucene-index.trec02-ar.pos+docvectors+rawdocs \ + -topicreader TsvString -topics src/main/resources/topics-and-qrels/topics.trec02ar.mono.ar.txt \ + -language ar -bm25 -output run.trec02-ar.bm25.topics.trec02ar.mono.ar.txt & ``` diff --git a/src/main/resources/docgen/templates/fire12-bn.template b/src/main/resources/docgen/templates/fire12-bn.template index 65efc582e9..1366d9274b 100644 --- a/src/main/resources/docgen/templates/fire12-bn.template +++ b/src/main/resources/docgen/templates/fire12-bn.template @@ -3,7 +3,7 @@ This page documents regression experiments for [FIRE 2012 Ad-hoc retrieval (Monolingual Bengali topic)](http://isical.ac.in/~fire/2012/adhoc.html). The document collection can be found in [FIRE 2012 data page](http://fire.irsi.res.in/fire/static/data). -The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire-bn.yaml). +The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire12-bn.yaml). Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/fire12-bn.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead. ## Indexing diff --git a/src/main/resources/docgen/templates/fire12-en.template b/src/main/resources/docgen/templates/fire12-en.template index 32260d19de..fafce9c653 100644 --- a/src/main/resources/docgen/templates/fire12-en.template +++ b/src/main/resources/docgen/templates/fire12-en.template @@ -3,7 +3,7 @@ This page documents regression experiments for [FIRE 2012 Ad-hoc retrieval (Monolingual English topic)](http://isical.ac.in/~fire/2012/adhoc.html). The document collection can be found in [FIRE 2012 data page](http://fire.irsi.res.in/fire/static/data). -The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire-en.yaml). +The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire12-en.yaml). Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/fire12-en.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead. ## Indexing diff --git a/src/main/resources/docgen/templates/fire12-hi.template b/src/main/resources/docgen/templates/fire12-hi.template index 89fcbd0876..9343974491 100644 --- a/src/main/resources/docgen/templates/fire12-hi.template +++ b/src/main/resources/docgen/templates/fire12-hi.template @@ -3,7 +3,7 @@ This page documents regression experiments for [FIRE 2012 Ad-hoc retrieval (Monolingual Hindi topic)](http://isical.ac.in/~fire/2012/adhoc.html). The document collection can be found in [FIRE 2012 data page](http://fire.irsi.res.in/fire/static/data). -The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire-hi.yaml). +The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/fire12-hi.yaml). Note that this page is automatically generated from [this template](../src/main/resources/docgen/templates/fire12-hi.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead. ## Indexing diff --git a/src/main/resources/regression/fire12-bn.yaml b/src/main/resources/regression/fire12-bn.yaml index 17a66bb4d1..3689566578 100644 --- a/src/main/resources/regression/fire12-bn.yaml +++ b/src/main/resources/regression/fire12-bn.yaml @@ -1,5 +1,5 @@ --- -name: fire12-hi +name: fire12-bn index_command: target/appassembler/bin/IndexCollection index_utils_command: target/appassembler/bin/IndexUtils search_command: target/appassembler/bin/SearchCollection @@ -29,10 +29,18 @@ evals: can_combine: true - command: eval/trec_eval.9.0.4/trec_eval params: - - -m P.30 + - -m P.20 separator: "\t" parse_index: 2 - metric: p30 + metric: p20 + metric_precision: 4 + can_combine: true + - command: eval/trec_eval.9.0.4/trec_eval + params: + - -m ndcg_cut.20 + separator: "\t" + parse_index: 2 + metric: ndcg20 metric_precision: 4 can_combine: true input_roots: @@ -46,7 +54,7 @@ index_stats: documents (non-empty): 500122 total terms: 143972612 topics: - - name: "[FIRE2012 (Bengali monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)" + - name: "[FIRE 2012 (Bengali monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)" path: topics.fire12bn.176-225.txt qrel: qrels.fire12bn.176-225.txt models: @@ -57,5 +65,7 @@ models: results: map: - 0.2881 - p30: - - 0.3360 + p20: + - 0.3740 + ndcg20: + - 0.4261 diff --git a/src/main/resources/regression/fire12-en.yaml b/src/main/resources/regression/fire12-en.yaml index 94da4a6452..10d7328191 100644 --- a/src/main/resources/regression/fire12-en.yaml +++ b/src/main/resources/regression/fire12-en.yaml @@ -29,24 +29,32 @@ evals: can_combine: true - command: eval/trec_eval.9.0.4/trec_eval params: - - -m P.30 + - -m P.20 separator: "\t" parse_index: 2 - metric: p30 + metric: p20 + metric_precision: 4 + can_combine: true + - command: eval/trec_eval.9.0.4/trec_eval + params: + - -m ndcg_cut.20 + separator: "\t" + parse_index: 2 + metric: ndcg20 metric_precision: 4 can_combine: true input_roots: - /tuna1/ # on tuna - /store/ # on orca - /scratch2/ # on damiano -input: collections/fire/hindi/en.docs.2011 +input: collections/fire/english/en.docs.2011 index_path: indexes/lucene-index.fire12-en.pos+docvectors+rawdocs index_stats: documents: 392577 documents (non-empty): 392577 total terms: 115311163 topics: - - name: "[FIRE2012 (English monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)" + - name: "[FIRE 2012 (English monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)" path: topics.fire12en.176-225.txt qrel: qrels.fire12en.176-225.txt models: @@ -57,5 +65,7 @@ models: results: map: - 0.3713 - p30: - - 0.4560 + p20: + - 0.4970 + ndcg20: + - 0.5420 diff --git a/src/main/resources/regression/fire12-hi.yaml b/src/main/resources/regression/fire12-hi.yaml index 43beb77747..52beaeec9d 100644 --- a/src/main/resources/regression/fire12-hi.yaml +++ b/src/main/resources/regression/fire12-hi.yaml @@ -29,10 +29,18 @@ evals: can_combine: true - command: eval/trec_eval.9.0.4/trec_eval params: - - -m P.30 + - -m P.20 separator: "\t" parse_index: 2 - metric: p30 + metric: p20 + metric_precision: 4 + can_combine: true + - command: eval/trec_eval.9.0.4/trec_eval + params: + - -m ndcg_cut.20 + separator: "\t" + parse_index: 2 + metric: ndcg20 metric_precision: 4 can_combine: true input_roots: @@ -46,7 +54,7 @@ index_stats: documents (non-empty): 331599 total terms: 57038417 topics: - - name: "[FIRE2012 (Hindi monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)" + - name: "[FIRE 2012 (Hindi monolingual)](http://isical.ac.in/~fire/2012/adhoc.html)" path: topics.fire12hi.176-225.txt qrel: qrels.fire12hi.176-225.txt models: @@ -57,5 +65,7 @@ models: results: map: - 0.3867 - p30: - - 0.3920 + p20: + - 0.4470 + ndcg20: + - 0.5310 diff --git a/src/main/resources/regression/trec02-ar.yaml b/src/main/resources/regression/trec02-ar.yaml index f8276e9a9b..0c325e1bf8 100644 --- a/src/main/resources/regression/trec02-ar.yaml +++ b/src/main/resources/regression/trec02-ar.yaml @@ -7,7 +7,7 @@ topic_root: src/main/resources/topics-and-qrels/ qrels_root: src/main/resources/topics-and-qrels/ index_root: ranking_root: -collection: JsonCollection +collection: TrecCollection generator: LuceneDocumentGenerator threads: 16 index_options: