-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Carla metrics bug fix #1883
Carla metrics bug fix #1883
Conversation
The present change simply returns both mean and per-image. @lcadalzo Please weigh in, if you'd like, on what the behavior of this should be. |
I believe the only OD metrics that need be The cleanest solution, unless there's something here I'm missing, would be as follows: treat the non-AP metrics like we do other "normal" ones (e.g. for image classification), i.e. decorate them as |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is working as expected for me now
Mike's description of the error from slack:
This is essentially because the
GlobalMeter
class used for@populationwise
metric functions assumes that the functions already return the mean over all samples. This is true forAP_per_class
, but not the others.Easy fix: just have the metric functions return the mean over all samples. This seems to be what the writer of this code (David) expected.
But that leaves questions about why the carla OD functions are marked @populationwise in the first place and whether a user would ever want to see per-image results. In that case, should this be controllable from the config, or should we just return both?