Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TREC COVID regression scripts are broken #1669

Closed
lintool opened this issue Oct 31, 2021 · 1 comment
Closed

TREC COVID regression scripts are broken #1669

lintool opened this issue Oct 31, 2021 · 1 comment
Assignees
@lintool

Comments

@lintool
Copy link
Member

lintool commented Oct 31, 2021

The TREC regression scripts are broken:

python src/main/python/trec-covid/generate_round5_baselines.py
python src/main/python/trec-covid/generate_round4_baselines.py
python src/main/python/trec-covid/generate_round3_baselines.py
python src/main/python/trec-covid/generate_round5_doc2query_baselines.py
python src/main/python/trec-covid/generate_round4_doc2query_baselines.py

Reason is because of #1658 - the bug fix causes very slight differences in rounding.

Since the above scripts check for MD5, they break. Comparing MD5 is overly brittle... the scripts need to be made more robust.
@lintool lintool self-assigned this Oct 31, 2021
@lintool lintool mentioned this issue Nov 1, 2021
Merged
lintool added a commit that referenced this issue Nov 1, 2021
@lintool
Refactor TREC-COVID regression scripts (#1671)
48af9ef 
+ Fix issue described in #1669
+ Add checks in r3, r4, r5 and r4/r5 + d2q
+ Add new scripts for r1 and r2
@lintool
Copy link
Member Author

lintool commented Nov 1, 2021

Closed by #1671

@lintool lintool closed this as completed Nov 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lintool lintool

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@lintool