Skip to content

Commit

Permalink
fixed bug in trec_eval params
Browse files Browse the repository at this point in the history
  • Loading branch information
kelvin-jiang committed May 30, 2021
1 parent 62c4d4a commit 308f587
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 8 deletions.
10 changes: 5 additions & 5 deletions docs/experiments-fever.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,10 +119,10 @@ This run produces the following results:
|:----|----------------:|----------------:|
| 1 | 0.3887 | 0.5925 |
| 5 | 0.6517 | 0.7678 |
| 10 | 0.7349 | 0.8233 |
| 25 | 0.8117 | 0.8745 |
| 50 | 0.8570 | 0.9047 |
| 100 | 0.8900 | 0.9267 |
| 10 | 0.7349 | 0.8233 |
| 25 | 0.8117 | 0.8745 |
| 50 | 0.8570 | 0.9047 |
| 100 | 0.8900 | 0.9267 |

Note that this outperforms the TF-IDF baseline in the FEVER paper at every value of k.

Expand Down Expand Up @@ -158,7 +158,7 @@ python src/main/python/fever/tune_bm25.py \
--runs_folder runs/fever-bm25 \
--index_folder indexes/fever/lucene-index-fever-paragraph \
--queries_file collections/fever/queries.paragraph.train-subset.tsv \
--qrels_file collections/fever/qrels.paragraph.train-subset.tsv
--qrels_file collections/fever/qrels.paragraph.train-subset.txt
```

From the grid search, we observe that the parameters `k1=0.9`, `b=0.1` perform fairly well. If we retrieve on the dev set with these parameters:
Expand Down
6 changes: 3 additions & 3 deletions src/main/python/fever/tune_bm25.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,10 @@ def evaluate_runs(args):
run_file = os.path.join(args.runs_folder, file)
# evaluate with trec_eval
results = subprocess.check_output(['tools/eval/trec_eval.9.0.4/trec_eval',
'-mrecall.100',
'-mmap',
args.qrels_file,
run_file,
'-m recall.100',
'-m map'])
run_file])
# regex match trec_eval output to get metrics
match = re.search('map +\tall\t([0-9.]+)', results.decode('utf-8'))
map = float(match.group(1))
Expand Down

0 comments on commit 308f587

Please sign in to comment.