-
Notifications
You must be signed in to change notification settings - Fork 402
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix tie breaking in ndcg metric #2031
Conversation
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #2031 +/- ##
========================================
- Coverage 87% 36% -51%
========================================
Files 283 285 +2
Lines 15934 15993 +59
========================================
- Hits 13832 5691 -8141
- Misses 2102 10302 +8200 |
@lucadiliello could you please review as you are the expert on retrieval? |
I think this issue may also affect other retrieval metrics. The problem with other retrieval metrics is that definitions are not really standardized between Wikipedia and slides of university courses. I mean that the general definition is the same, but the behavior in corner cases like this one is not well defined. I'm currently on vacation, I will try to review the PR once I'm in front of my pc. |
cool, @lucadiliello let us know when you ll check and and ideally if all fine approve this PR :) |
@lucadiliello pls mind checking it ^^ |
LGTM |
* fix implementation * add tests * chlog --------- Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit 1caaf28)
* fix implementation * add tests * chlog --------- Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit 1caaf28)
* fix implementation * add tests * chlog --------- Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
What does this PR do?
Fixes #2022
In the issue it was identified that the overall problem is that our implementation of ndcg currently cannot deal with tie breaks of scores. PR fixes this.
Before submitting
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃
📚 Documentation preview 📚: https://torchmetrics--2031.org.readthedocs.build/en/2031/