Skip to content

Commit

Permalink
Fix regression figures for Lucene 9 (#1960)
Browse files Browse the repository at this point in the history 
Everything passed (total of 267 regressions) except for wikipedia-dpr-100w-bm25,
which is pending a Pyserini upgrade because of a circular dependency.
  • Loading branch information
@lintool
lintool authored Aug 15, 2022
1 parent 0fc578a commit cc6337c
Show file tree
Hide file tree
Showing 75 changed files with 190 additions and 190 deletions.
4 changes: 2 additions & 2 deletions docs/regressions-clef06-fr.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ With the above commands, you should be able to reproduce the following results:

| **MAP** | **BM25** |
|:-------------------------------------------------------------------------------------------------------------|-----------|
| [CLEF 2006 (Monolingual French)](../src/main/resources/topics-and-qrels/topics.clef06fr.mono.fr.txt) | 0.3111 |
| [CLEF 2006 (Monolingual French)](../src/main/resources/topics-and-qrels/topics.clef06fr.mono.fr.txt) | 0.3115 |
| **P20** | **BM25** |
| [CLEF 2006 (Monolingual French)](../src/main/resources/topics-and-qrels/topics.clef06fr.mono.fr.txt) | 0.3184 |
| **nDCG@20** | **BM25** |
| [CLEF 2006 (Monolingual French)](../src/main/resources/topics-and-qrels/topics.clef06fr.mono.fr.txt) | 0.4458 |
| [CLEF 2006 (Monolingual French)](../src/main/resources/topics-and-qrels/topics.clef06fr.mono.fr.txt) | 0.4457 |
22 changes: 11 additions & 11 deletions docs/regressions-hc4-neuclir22-ru.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -156,21 +156,21 @@ With the above commands, you should be able to reproduce the following results:

| **MAP** | **BM25 (default)**| **+RM3** | **+Rocchio**|
|:-------------------------------------------------------------------------------------------------------------|-----------|-----------|-----------|
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.0964 | 0.0811 | 0.1245 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.0926 | 0.0605 | 0.1064 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.1113 | 0.0771 | 0.1341 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.1040 | 0.0841 | 0.1231 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.0963 | 0.0640 | 0.0964 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.1264 | 0.0825 | 0.1314 |
| **nDCG@20** | **BM25 (default)**| **+RM3** | **+Rocchio**|
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.1380 | 0.1257 | 0.1668 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.1459 | 0.0963 | 0.1643 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.1640 | 0.1318 | 0.1899 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.1445 | 0.1283 | 0.1655 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.1495 | 0.1037 | 0.1569 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.1762 | 0.1411 | 0.1875 |
| **J@20** | **BM25 (default)**| **+RM3** | **+Rocchio**|
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.0860 | 0.0730 | 0.0940 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.0790 | 0.0610 | 0.0890 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.0900 | 0.0750 | 0.0980 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.0860 | 0.0720 | 0.0930 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.0790 | 0.0620 | 0.0890 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.0900 | 0.0760 | 0.0980 |
| **Recall@1000** | **BM25 (default)**| **+RM3** | **+Rocchio**|
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.6319 | 0.6154 | 0.6887 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.6319 | 0.6154 | 0.6982 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.6640 | 0.5408 | 0.6407 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.6667 | 0.6221 | 0.6743 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.6667 | 0.6254 | 0.6810 |

## Reproduction Log[*](reproducibility.md)

Expand Down
30 changes: 15 additions & 15 deletions docs/regressions-hc4-v1.0-ru.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -246,32 +246,32 @@ With the above commands, you should be able to reproduce the following results:
| **MAP** | **BM25 (default)**| **+RM3** | **+Rocchio**|
|:-------------------------------------------------------------------------------------------------------------|-----------|-----------|-----------|
| [HC4 (Russian): dev-topic title](https://github.com/hltcoe/HC4) | 0.2937 | 0.2390 | 0.3995 |
| [HC4 (Russian): dev-topic description](https://github.com/hltcoe/HC4) | 0.2374 | 0.0844 | 0.2817 |
| [HC4 (Russian): dev-topic description+title](https://github.com/hltcoe/HC4) | 0.3209 | 0.2150 | 0.3565 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.2186 | 0.2369 | 0.2592 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.1880 | 0.1874 | 0.2252 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.2267 | 0.2290 | 0.2703 |
| [HC4 (Russian): dev-topic description](https://github.com/hltcoe/HC4) | 0.2373 | 0.0844 | 0.2817 |
| [HC4 (Russian): dev-topic description+title](https://github.com/hltcoe/HC4) | 0.3186 | 0.2116 | 0.3564 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.2186 | 0.2371 | 0.2641 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.1883 | 0.1868 | 0.2250 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.2265 | 0.2302 | 0.2732 |
| **nDCG@20** | **BM25 (default)**| **+RM3** | **+Rocchio**|
| [HC4 (Russian): dev-topic title](https://github.com/hltcoe/HC4) | 0.3942 | 0.3376 | 0.4719 |
| [HC4 (Russian): dev-topic description](https://github.com/hltcoe/HC4) | 0.2580 | 0.1838 | 0.3168 |
| [HC4 (Russian): dev-topic description+title](https://github.com/hltcoe/HC4) | 0.3993 | 0.3412 | 0.4400 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.2954 | 0.3200 | 0.3108 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.2446 | 0.2402 | 0.2759 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.2983 | 0.2955 | 0.3234 |
| [HC4 (Russian): dev-topic description+title](https://github.com/hltcoe/HC4) | 0.3972 | 0.3367 | 0.4400 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.2944 | 0.3201 | 0.3163 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.2456 | 0.2396 | 0.2767 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.2989 | 0.2994 | 0.3265 |
| **J@20** | **BM25 (default)**| **+RM3** | **+Rocchio**|
| [HC4 (Russian): dev-topic title](https://github.com/hltcoe/HC4) | 0.4375 | 0.4500 | 0.5125 |
| [HC4 (Russian): dev-topic description](https://github.com/hltcoe/HC4) | 0.5125 | 0.3625 | 0.5500 |
| [HC4 (Russian): dev-topic description+title](https://github.com/hltcoe/HC4) | 0.5000 | 0.4625 | 0.5875 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.3480 | 0.3620 | 0.3950 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.3180 | 0.2960 | 0.3510 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.3650 | 0.3520 | 0.3960 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.3470 | 0.3620 | 0.3930 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.3180 | 0.2970 | 0.3510 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.3670 | 0.3520 | 0.3990 |
| **Recall@1000** | **BM25 (default)**| **+RM3** | **+Rocchio**|
| [HC4 (Russian): dev-topic title](https://github.com/hltcoe/HC4) | 0.8432 | 0.7598 | 0.8710 |
| [HC4 (Russian): dev-topic description](https://github.com/hltcoe/HC4) | 0.5942 | 0.3886 | 0.6171 |
| [HC4 (Russian): dev-topic description+title](https://github.com/hltcoe/HC4) | 0.7619 | 0.6428 | 0.7639 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.7182 | 0.7223 | 0.7713 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.7355 | 0.6475 | 0.7669 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.7721 | 0.7273 | 0.8230 |
| [HC4 (Russian): test-topic title](https://github.com/hltcoe/HC4) | 0.7182 | 0.7223 | 0.7728 |
| [HC4 (Russian): test-topic description](https://github.com/hltcoe/HC4) | 0.7355 | 0.6480 | 0.7680 |
| [HC4 (Russian): test-topic description+title](https://github.com/hltcoe/HC4) | 0.7721 | 0.7273 | 0.8271 |

## Reproduction Log[*](reproducibility.md)

Expand Down
4 changes: 2 additions & 2 deletions docs/regressions-mrtydi-v1.1-fi.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,9 @@ With the above commands, you should be able to reproduce the following results:
| **MRR@100** | **BM25** |
|:-------------------------------------------------------------------------------------------------------------|-----------|
| [Mr. TyDi (Finnish): train](https://github.com/castorini/mr.tydi) | 0.4101 |
| [Mr. TyDi (Finnish): dev](https://github.com/castorini/mr.tydi) | 0.4133 |
| [Mr. TyDi (Finnish): dev](https://github.com/castorini/mr.tydi) | 0.4136 |
| [Mr. TyDi (Finnish): test](https://github.com/castorini/mr.tydi) | 0.2836 |
| **R@100** | **BM25** |
| [Mr. TyDi (Finnish): train](https://github.com/castorini/mr.tydi) | 0.8198 |
| [Mr. TyDi (Finnish): dev](https://github.com/castorini/mr.tydi) | 0.8285 |
| [Mr. TyDi (Finnish): test](https://github.com/castorini/mr.tydi) | 0.7193 |
| [Mr. TyDi (Finnish): test](https://github.com/castorini/mr.tydi) | 0.7196 |
12 changes: 6 additions & 6 deletions docs/regressions-mrtydi-v1.1-ja.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,10 @@ With the above commands, you should be able to reproduce the following results:

| **MRR@100** | **BM25** |
|:-------------------------------------------------------------------------------------------------------------|-----------|
| [Mr. TyDi (Japanese): train](https://github.com/castorini/mr.tydi) | 0.2236 |
| [Mr. TyDi (Japanese): dev](https://github.com/castorini/mr.tydi) | 0.2241 |
| [Mr. TyDi (Japanese): test](https://github.com/castorini/mr.tydi) | 0.2112 |
| [Mr. TyDi (Japanese): train](https://github.com/castorini/mr.tydi) | 0.2262 |
| [Mr. TyDi (Japanese): dev](https://github.com/castorini/mr.tydi) | 0.2250 |
| [Mr. TyDi (Japanese): test](https://github.com/castorini/mr.tydi) | 0.2125 |
| **R@100** | **BM25** |
| [Mr. TyDi (Japanese): train](https://github.com/castorini/mr.tydi) | 0.7282 |
| [Mr. TyDi (Japanese): dev](https://github.com/castorini/mr.tydi) | 0.7274 |
| [Mr. TyDi (Japanese): test](https://github.com/castorini/mr.tydi) | 0.6451 |
| [Mr. TyDi (Japanese): train](https://github.com/castorini/mr.tydi) | 0.7290 |
| [Mr. TyDi (Japanese): dev](https://github.com/castorini/mr.tydi) | 0.7252 |
| [Mr. TyDi (Japanese): test](https://github.com/castorini/mr.tydi) | 0.6431 |
12 changes: 6 additions & 6 deletions docs/regressions-mrtydi-v1.1-ru.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,10 @@ With the above commands, you should be able to reproduce the following results:

| **MRR@100** | **BM25** |
|:-------------------------------------------------------------------------------------------------------------|-----------|
| [Mr. TyDi (Russian): train](https://github.com/castorini/mr.tydi) | 0.2205 |
| [Mr. TyDi (Russian): dev](https://github.com/castorini/mr.tydi) | 0.2152 |
| [Mr. TyDi (Russian): test](https://github.com/castorini/mr.tydi) | 0.3129 |
| [Mr. TyDi (Russian): train](https://github.com/castorini/mr.tydi) | 0.2229 |
| [Mr. TyDi (Russian): dev](https://github.com/castorini/mr.tydi) | 0.2202 |
| [Mr. TyDi (Russian): test](https://github.com/castorini/mr.tydi) | 0.3163 |
| **R@100** | **BM25** |
| [Mr. TyDi (Russian): train](https://github.com/castorini/mr.tydi) | 0.5706 |
| [Mr. TyDi (Russian): dev](https://github.com/castorini/mr.tydi) | 0.5673 |
| [Mr. TyDi (Russian): test](https://github.com/castorini/mr.tydi) | 0.6482 |
| [Mr. TyDi (Russian): train](https://github.com/castorini/mr.tydi) | 0.5779 |
| [Mr. TyDi (Russian): dev](https://github.com/castorini/mr.tydi) | 0.5760 |
| [Mr. TyDi (Russian): test](https://github.com/castorini/mr.tydi) | 0.6541 |
Loading

0 comments on commit cc6337c

Please sign in to comment.