ValueError: 0.5/0.75 is not in list - when passing a custom set of IoU thresholds to MeanAveragePrecision

[Chris-hughes10]

🐛 Bug

When I try to use MeanAveragePrecision to compute the mAP over a custom range of IoU thresholds, a value error is thrown if 0.5 and 0.75 are not included in the list.

This is due to these values being hardcoded here.

To Reproduce

Steps to reproduce the behavior...

Traceback (most recent call last):
 line 27, in <module>
    print(map.compute())
  File "C:\Users\hughesc\Anaconda3\envs\accelerated-dev\lib\site-packages\torchmetrics\metric.py", line 440, in wrapped_func
    value = compute(*args, **kwargs)
  File "C:\Users\hughesc\Anaconda3\envs\accelerated-dev\lib\site-packages\torchmetrics\detection\mean_ap.py", line 766, in compute
    map_val, mar_val = self._summarize_results(precisions, recalls)
  File "C:\Users\hughesc\Anaconda3\envs\accelerated-dev\lib\site-packages\torchmetrics\detection\mean_ap.py", line 656, in _summarize_results
    map_metrics.map_50 = self._summarize(results, True, iou_threshold=0.5, max_dets=last_max_det_thr)
  File "C:\Users\hughesc\Anaconda3\envs\accelerated-dev\lib\site-packages\torchmetrics\detection\mean_ap.py", line 570, in _summarize
    thr = self.iou_thresholds.index(iou_threshold)
ValueError: 0.5 is not in list

Code sample

import torch
from torchmetrics.detection import MeanAveragePrecision

if __name__ == '__main__':

    device = torch.device('cuda:0')

    preds = [
        dict( boxes=torch.tensor([[1.5, 2., 3., 4.]], device=device),
        scores=torch.tensor([0.6], device=device),
        labels=torch.tensor([1], device=device),)
    ]

    targets = [
        dict(boxes=torch.tensor([[1., 2., 3., 4.]], device=device),
             scores=torch.tensor([0.8], device=device),
             labels=torch.tensor([1], device=device), )
    ]

    map = MeanAveragePrecision(class_metrics=True,
                               iou_thresholds=[0.1, 0.2],
                               ).to(device)

    map.update(preds, targets)

    print(map.compute())

Expected behavior

Ideally, I would like to be able to calculate mAP using any custom range of IoU values, or even a single value if it is passed as a list (e.g. [0.4]). Although this is a departure from the pycoctools implementation, this would make this metric well suited to use cases outside of the coco competition.

At a minimum, this restriction should be documented, and the error should be handled more gracefully.

Environment

• OS (e.g., Linux): Windows, Ubuntu
• Python & PyTorch Version (e.g., 1.0): Python 3.8 & 3.9, PyTorch 1.10
• How you installed PyTorch (conda, pip, build command if you used source): conda
• Any other relevant information:

[bepuca]

I am facing the same issue when passing a single threshold, which I need to do some error analysis.

The easiest solution could be to return -1 for either of these thresholds if they are not present in the specified thresholds.

A more neat approach could be to have an optional argument at the beginning set by default at [0.5, 0.75] to indicate the desired thresholds to be reported? Then would be possible to iterate over the asked thresholds and report them.

I can try and implement the solution and submit a PR. Ideally I would do the former (with the -1) as I am a bit budgeted in time but if you think the later is what should be there I can try to give it a go too.

[SkafteNicki]

Hi @Chris-hughes10 and @bepuca,
I created PR #995 to address this. We will simply check for the 0.5 and 0.75 in the list of thresholds and if they are not present we will set them to -1. Hope this works for your both :]

[bepuca]

Thank you very much for such a quick response @SkafteNicki !

[ekmungi]

I know the issue is closed, but I am wondering, how can I get the mAP for a custom IOU threshold = 0.25 for example? The resulting dict from map from the fix still gives the following format. Is the mAP calculated for a custom IOU (for example = 0.25) internally?

{'map': , 'map_50': , 'map_75': , 'map_large': , 'map_medium': , 'map_per_class': , 'map_small': , 'mar_1': , 'mar_10': , 'mar_100': , 'mar_100_per_class': , 'mar_large': , 'mar_medium': , 'mar_small': }