Skip to content

Commit

Permalink
Fix error results in Vector-PRF reproduce instruction (castorini#840)
Browse files Browse the repository at this point in the history
  • Loading branch information
@hanglics
hanglics authored Oct 27, 2021
1 parent 70d79b4 commit 3e4c283
Showing 1 changed file with 29 additions and 29 deletions.
58 changes: 29 additions & 29 deletions docs/experiments-vector-prf.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,15 @@ Here's how our results stack up against all available models and datasets in Pys
| TCT-ColBERT V1 | Original | 0.3906 | 0.5730 | 0.7916 |
| TCT-ColBERT V1 | Average PRF 3 | 0.4336 | 0.6119 | 0.8230 |
| TCT-ColBERT V1 | Rocchio PRF 5 A0.4 B0.6 | 0.4463 | 0.6143 | 0.8393 |
| TCT-ColBERT V2 | Original | 0.4269 | 0.6129 | 0.8342 |
| TCT-ColBERT V2 | Average PRF 3 | 0.4766 | 0.6487 | 0.8574 |
| TCT-ColBERT V2 | Rocchio PRF 5 A0.4 B0.6 | 0.4709 | 0.6435 | 0.8496 |
| DistillBERT KD | Original | 0.3759 | 0.5765 | 0.6853 |
| DistillBERT KD | Average PRF 3 | 0.4362 | 0.6217 | 0.7180 |
| DistillBERT KD | Rocchio PRF 5 A0.4 B0.6 | 0.4378 | 0.6189 | 0.7291 |
| DistillBERT Balanced | Original | 0.4761 | 0.6360 | 0.7826 |
| DistillBERT Balanced | Average PRF 3 | 0.5057 | 0.6526 | 0.8054 |
| DistillBERT Balanced | Rocchio PRF 5 A0.4 B0.6 | 0.5249 | 0.6684 | 0.8352 |
| TCT-ColBERT V2 HN+ | Original | 0.4469 | 0.6318 | 0.8261 |
| TCT-ColBERT V2 HN+ | Average PRF 3 | 0.4879 | 0.6719 | 0.8586 |
| TCT-ColBERT V2 HN+ | Rocchio PRF 5 A0.4 B0.6 | 0.4883 | 0.6684 | 0.8694 |
| DistillBERT KD | Original | 0.4053 | 0.5765 | 0.7653 |
| DistillBERT KD | Average PRF 3 | 0.4575 | 0.6217 | 0.7939 |
| DistillBERT KD | Rocchio PRF 5 A0.4 B0.6 | 0.4548 | 0.6189 | 0.8049 |
| DistillBERT Balanced | Original | 0.4590 | 0.6360 | 0.8406 |
| DistillBERT Balanced | Average PRF 3 | 0.4856 | 0.6526 | 0.8515 |
| DistillBERT Balanced | Rocchio PRF 5 A0.4 B0.6 | 0.4974 | 0.6684 | 0.8775 |
| SBERT | Original | 0.4060 | 0.5985 | 0.7872 |
| SBERT | Average PRF 3 | 0.4354 | 0.6149 | 0.7937 |
| SBERT | Rocchio PRF 5 A0.4 B0.6 | 0.4371 | 0.6149 | 0.7941 |
Expand All @@ -55,15 +55,15 @@ Here's how our results stack up against all available models and datasets in Pys
| TCT-ColBERT V1 | Original | 0.4290 | 0.5826 | 0.8181 |
| TCT-ColBERT V1 | Average PRF 3 | 0.4725 | 0.6101 | 0.8667 |
| TCT-ColBERT V1 | Rocchio PRF 5 A0.4 B0.6 | 0.4625 | 0.6056 | 0.8576 |
| TCT-ColBERT V2 | Original | 0.4717 | 0.6200 | 0.8407 |
| TCT-ColBERT V2 | Average PRF 3 | 0.4701 | 0.6209 | 0.8739 |
| TCT-ColBERT V2 | Rocchio PRF 5 A0.4 B0.6 | 0.4819 | 0.6324 | 0.8760 |
| DistillBERT KD | Original | 0.3909 | 0.5728 | 0.6893 |
| DistillBERT KD | Average PRF 3 | 0.3955 | 0.5755 | 0.7279 |
| DistillBERT KD | Rocchio PRF 5 A0.4 B0.6 | 0.3990 | 0.5760 | 0.7222 |
| DistillBERT Balanced | Original | 0.4755 | 0.6346 | 0.8009 |
| DistillBERT Balanced | Average PRF 3 | 0.4873 | 0.6449 | 0.8392 |
| DistillBERT Balanced | Rocchio PRF 5 A0.4 B0.6 | 0.4846 | 0.6470 | 0.8262 |
| TCT-ColBERT V2 HN+ | Original | 0.4754 | 0.6206 | 0.8429 |
| TCT-ColBERT V2 HN+ | Average PRF 3 | 0.4811 | 0.6228 | 0.8579 |
| TCT-ColBERT V2 HN+ | Rocchio PRF 5 A0.4 B0.6 | 0.4860 | 0.6254 | 0.8518 |
| DistillBERT KD | Original | 0.4159 | 0.5728 | 0.7953 |
| DistillBERT KD | Average PRF 3 | 0.4214 | 0.5755 | 0.8403 |
| DistillBERT KD | Rocchio PRF 5 A0.4 B0.6 | 0.4145 | 0.5760 | 0.8433 |
| DistillBERT Balanced | Original | 0.4698 | 0.6346 | 0.8727 |
| DistillBERT Balanced | Average PRF 3 | 0.4887 | 0.6449 | 0.9030 |
| DistillBERT Balanced | Rocchio PRF 5 A0.4 B0.6 | 0.4879 | 0.6470 | 0.8926 |
| SBERT | Original | 0.4124 | 0.5734 | 0.7937 |
| SBERT | Average PRF 3 | 0.4258 | 0.5781 | 0.8169 |
| SBERT | Rocchio PRF 5 A0.4 B0.6 | 0.4342 | 0.5851 | 0.8226 |
Expand All @@ -77,20 +77,20 @@ The PRF does not perform well with sparse judgements like in MS MARCO, the resul

| Model | Method | MAP | nDCG@100 | Recall@1000 |
|:---------------------|:------------------------|:------:|:--------:|:-----------:|
| ANCE | Original | 0.3363 | 0.4457 | 0.9584 |
| ANCE | Average PRF 3 | 0.3132 | 0.4246 | 0.9490 |
| ANCE | Rocchio PRF 5 A0.4 B0.6 | 0.3116 | 0.4250 | 0.9547 |
| ANCE | Original | 0.3362 | 0.4457 | 0.9587 |
| ANCE | Average PRF 3 | 0.3133 | 0.4247 | 0.9490 |
| ANCE | Rocchio PRF 5 A0.4 B0.6 | 0.3115 | 0.4250 | 0.9545 |
| TCT-ColBERT V1 | Original | 0.3416 | 0.4514 | 0.9640 |
| TCT-ColBERT V1 | Average PRF 3 | 0.2882 | 0.4014 | 0.9452 |
| TCT-ColBERT V1 | Rocchio PRF 5 A0.4 B0.6 | 0.2809 | 0.3988 | 0.9543 |
| TCT-ColBERT V2 | Original | 0.3504 | 0.4625 | 0.9679 |
| TCT-ColBERT V2 | Average PRF 3 | 0.3055 | 0.4189 | 0.9547 |
| TCT-ColBERT V2 | Rocchio PRF 5 A0.4 B0.6 | 0.3002 | 0.4190 | 0.9627 |
| TCT-ColBERT V2 HN+ | Original | 0.3644 | 0.4750 | 0.9695 |
| TCT-ColBERT V2 HN+ | Average PRF 3 | 0.3183 | 0.4325 | 0.9585 |
| TCT-ColBERT V2 HN+ | Rocchio PRF 5 A0.4 B0.6 | 0.3190 | 0.4360 | 0.9659 |
| DistillBERT KD | Original | 0.3309 | 0.4391 | 0.9553 |
| DistillBERT KD | Average PRF 3 | 0.2830 | 0.3940 | 0.9325 |
| DistillBERT KD | Rocchio PRF 5 A0.4 B0.6 | 0.2787 | 0.3937 | 0.9432 |
| DistillBERT Balanced | Original | 0.3515 | 0.4651 | 0.9771 |
| DistillBERT Balanced | Average PRF 3 | 0.2978 | 0.4150 | 0.9613 |
| DistillBERT Balanced | Average PRF 3 | 0.2979 | 0.4151 | 0.9613 |
| DistillBERT Balanced | Rocchio PRF 5 A0.4 B0.6 | 0.2969 | 0.4178 | 0.9702 |
| SBERT | Original | 0.3373 | 0.4453 | 0.9558 |
| SBERT | Average PRF 3 | 0.3094 | 0.4183 | 0.9446 |
Expand Down Expand Up @@ -138,7 +138,7 @@ different values to run on different datasets.
`--index`: <br />
&nbsp;&nbsp;&nbsp;&nbsp;ANCE index with MS MARCO V1 passage collection: `msmarco-passage-ance-bf` <br />
&nbsp;&nbsp;&nbsp;&nbsp;TCT-ColBERT V1 index with MS MARCO V1 passage collection: `msmarco-passage-tct_colbert-bf` <br />
&nbsp;&nbsp;&nbsp;&nbsp;TCT-ColBERT V2 index with MS MARCO V1 passage collection: `msmarco-passage-tct_colbert-v2-bf` <br />
&nbsp;&nbsp;&nbsp;&nbsp;TCT-ColBERT V2 HN+ index with MS MARCO V1 passage collection: `msmarco-passage-tct_colbert-v2-hnp-bf` <br />
&nbsp;&nbsp;&nbsp;&nbsp;DistillBERT KD index with MS MARCO V1 passage collection: `msmarco-passage-distilbert-dot-margin_mse-T2-bf` <br />
&nbsp;&nbsp;&nbsp;&nbsp;DistillBERT Balanced index with MS MARCO V1 passage collection: `msmarco-passage-distilbert-dot-tas_b-b256-bf` <br />
&nbsp;&nbsp;&nbsp;&nbsp;SBERT index with MS MARCO V1 passage collection: `msmarco-passage-sbert-bf` <br />
Expand All @@ -148,7 +148,7 @@ _Note: TREC DL 2019, TREC DL 2020, and MS MARCO Passage V1 use the same passage
`--encoder`: <br />
&nbsp;&nbsp;&nbsp;&nbsp;ANCE: `castorini/ance-msmarco-passage` <br />
&nbsp;&nbsp;&nbsp;&nbsp;TCT-ColBERT V1: `castorini/tct_colbert-msmarco` <br />
&nbsp;&nbsp;&nbsp;&nbsp;TCT-ColBERT V2: `castorini/tct_colbert-v2-hnp-msmarco` <br />
&nbsp;&nbsp;&nbsp;&nbsp;TCT-ColBERT V2 HN+: `castorini/tct_colbert-v2-hnp-msmarco` <br />
&nbsp;&nbsp;&nbsp;&nbsp;DistillBERT KD: `sebastian-hofstaetter/distilbert-dot-margin_mse-T2-msmarco` <br />
&nbsp;&nbsp;&nbsp;&nbsp;DistillBERT Balanced: `sebastian-hofstaetter/distilbert-dot-tas_b-b256-msmarco` <br />
&nbsp;&nbsp;&nbsp;&nbsp;SBERT: `sentence-transformers/msmarco-distilbert-base-v3` <br />
Expand Down Expand Up @@ -206,8 +206,8 @@ Qrels file also available in Pyserini, just replace the `runs/run.ance.dl20-pass
For MS MARCO Passage V1, no need to use `-l 2` option:
```
$ python -m pyserini.eval.trec_eval -c -m map -m ndcg_cut.100 -m recall.1000 msmarco-passage-dev-subset runs/run.ance.msmarco-passage.average_prf3.trec
map all 0.3132
ndcg_cut_100 all 0.4246
map all 0.3133
ndcg_cut_100 all 0.4247
recall_1000 all 0.9490
```
Qrels file already available, replace the `runs/run.ance.msmarco-passage.average_prf3.trec` with your own run file path to test your reproduced results.
Expand Down

0 comments on commit 3e4c283

Please sign in to comment.