Provide a way to Retrieve Precision, Recall, and Confusion Matrix scores from MeanAveragePrecision metric

[devforfu]

🚀 Feature

The torchmetrics.detection.mean_ap.MeanAveragePrecision metric returns a dictionary with standard COCO object detection quality metrics. However, it doesn't provide a way to retrieve non-aggregated metrics computed internally, like Precisions, Recalls, IoU scores or confusion matrix counters (TP/FP/TN/FN).

This feature request proposes to extend the metric to return those values in addition to aggregated scores.

Motivation

Our team works on building object detection models. In addition to computing mAP and mAR scores, we would also like to inspect PR curves and IoU scores, plus derive other metrics (like F1-score or mIoU) from them. The MeanAveragePrecision metric already computes these values internally, but doesn't provide a way to access them.

Pitch

Extend __init__ parameters of the MeanAveragePrecision class that to allow users to request additional reported metrics:

• Precision
• Recall
• IoU
• Confusion matrices

Currently, we "patched" the metric to access precisions and recalls tensors, plus updated the _compute_iou method via inheritance to collect the computed IoU scores. (We did it for the older implementation that now lives in _mean_ap.py and doesn't use the code from pycocoeval.py)

Here is a diff file showing the changes we made.

14a26
> from dataclasses import dataclass
20d31
<
126a138,146
> @dataclass
> class COCOMetricsExtended:
>     """"Aggregated COCO metrics as in the default implementation plus PR curves."""
>
>     aggregated: dict
>     precisions: dict
>     recalls: dict
>
>
199c219
< class MeanAveragePrecision(Metric):
---
> class PatchedMeanAveragePrecision(Metric):
637c657
<                     m = MeanAveragePrecision._find_best_gt_match(t, gt_matches, idx_iou, gt_ignore, ious, idx_det)
---
>                     m = PatchedMeanAveragePrecision._find_best_gt_match(t, gt_matches, idx_iou, gt_ignore, ious, idx_det)
729c749,752
<                 prec = prec[:, :, area_inds, mdet_inds]
---
>                 # dimension of recall tensor:
>                 # REFERENCE: [TxKxAxM]
>                 # NEW:       [TxRxKxAxM]
>                 prec, _ = prec[:, :, :, area_inds, mdet_inds].max(dim=1)
765c788
<         recall = -torch.ones((nb_iou_thrs, nb_classes, nb_bbox_areas, nb_max_det_thrs))
---
>         recall = -torch.ones((nb_iou_thrs, nb_rec_thrs, nb_classes, nb_bbox_areas, nb_max_det_thrs))
775c798
<                     recall, precision, scores = MeanAveragePrecision.__calculate_recall_precision_scores(
---
>                     recall, precision, scores = PatchedMeanAveragePrecision.__calculate_recall_precision_scores(
873a897
>             recs = torch.zeros((nb_rec_thrs,))
876,877d899
<             recall[idx, idx_cls, idx_bbox_area, idx_max_det_thrs] = rc[-1] if nd else 0
<
889a912,913
>             recs[:num_inds] = rc[inds]
>             recs[-1] = rc[-1] if nd else 0
891a916
>             recall[idx, :, idx_cls, idx_bbox_area, idx_max_det_thrs] = recs
896c921
<     def compute(self) -> dict:
---
>     def compute(self) -> COCOMetricsExtended:
925c950
<         return metrics
\ No newline at end of file
---
>         return COCOMetricsExtended(aggregated=metrics, precisions=precisions, recalls=recalls)

This allows us to access Precision/Recall and, for example, use them to plot PR-curves. We also created a test asserting that both metrics return the same values for aggregated scores. Therefore, our patch didn't break the existing aggregated scores, but only allowed to access internal variables.

Note that we also changed the shape of the precision tensor to return all the scores, and not the highest one only as before. (See the NEW shape comment above.)

Alternatives

We can compute Precision/Recall/IoU/confmat metrics separately, but it would require recomputing the same values multiple times and reimplementing the logic that is already available.

Additional context

If there is a way to compute the metrics we need without changing the MeanAveragePrecision metric's code, please let us know! Currently, we don't see any simple alternative to achieve that without reimplementing large parts of already available functionality, or doing redundant computations with mulitple metric callbacks.

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

Hi @devforfu, thanks for raising this issue.
I created PR #1983 that should solve this issue. It will give out precision, recall and ious as these are automatically computed by pycoco. It does not currently also includes the confusion matrix.

[shanalikhan]

Any way to get P, R by class ID instead?