Batch NDCG #342
Merged
Batch NDCG #342
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
donglihe-hub
requested review from
cjlin1,
Eleven1Liu,
henryyang42,
JamesLYC88 and
Gordon119
as code owners
donglihe-hub
force-pushed
the
OptimizeNDCG
branch
from
c41e976
to
a26302c
Compare
donglihe-hub
force-pushed
the
OptimizeNDCG
branch
from
a26302c
to
198a58e
Compare
Sinacam
reviewed
Jan 3, 2024
A doubt that has haunted me for years is that why it is preds and target, rather than preds and targets or pred and target. Therefore, I prefer using singular form for all variables except for those concepts that have been accepted by the general public. |
donglihe-hub
force-pushed
the
OptimizeNDCG
branch
from
e8e4b67
to
c96657c
Compare
Sinacam
reviewed
Jan 5, 2024
donglihe-hub
force-pushed
the
OptimizeNDCG
branch
from
979b865
to
30701c7
Compare
cjlin1
reviewed
Jan 6, 2024
libmultilabel/nn/metrics.py
Outdated
| @@ -45,6 +45,7 @@ class NDCG(Metric): | |||
| https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ndcg_score.html | |||
| Please find the formal definition here: | |||
| https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html | |||
| The target has to be a binary tensor. | |||
There was a problem hiding this comment.
add
We do not use NDCG in ?? because of ??
There was a problem hiding this comment.
I'm not pretty sure what should be filled in the ??. As Li-Chung only mention this in function-level comments, I will rewrite it and put it under _idcg() to align with Li-Chung's changes and to not confuse anyone.
donglihe-hub
force-pushed
the
OptimizeNDCG
branch
from
30701c7
to
b08dd9b
Compare
donglihe-hub
force-pushed
the
OptimizeNDCG
branch
from
30c9d54
to
a6b124d
Compare
donglihe-hub
force-pushed
the
OptimizeNDCG
branch
from
8c36c4d
to
fb2a71c
Compare
cjlin1
approved these changes
Jan 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The original NDCG metric calculates scores for one instance at a time, which is inefficient. The new NDCG metric calculate scores in batch.
Performance Test Settings:
Number of labels = 100
Batch size = 40
Number of batchs = 100
Effective number of validation samples = 4000
Results:
Test CLI & API (
bash tests/autotest.sh)Test APIs used by main.py.
Check API Document
If any new APIs are added, please check if the description of the APIs is added to API document.
Test quickstart & API (
bash tests/docs/test_changed_document.sh)If any APIs in quickstarts or tutorials are modified, please run this test to check if the current examples can run correctly after the modified APIs are released.