[Enhancement] Implement Distance-Based Detection Matching in DetectionMetrics #1463

DimaBir · 2023-09-13T11:01:37Z

Pull Request Description

This pull request introduces enhancements and additions related to distance-based metrics in the DetectionMetrics class. The main changes include:

Addition: The DetectionMetricsDistanceBased class has been added to support distance-based metrics for object detection.
Abstract base class DetectionMatching, which defines the interface for matching implementations.
Modifications: The existing DetectionMetrics class has been extended to incorporate distance-based metrics when calculating precision, recall, F1, and mAP.

These changes improve the capabilities of the metrics calculation for object detection tasks, allowing users to analyze detection performance using distance-based criteria. Below are the details of the changes made:

Changes Made

Introduced the DetectionMetricsDistanceBased class, which extends the existing DetectionMetrics class to support distance-based metrics.
Modified the DetectionMetrics class to accept distance metric definitions and thresholds.
Updated the compute method in both classes to calculate metrics based on the results of distance-based matching when using DetectionMetricsDistanceBased.
Introduced an abstract base class, DetectionMatching, which defines the interface for matching implementations. It includes abstract methods for getting thresholds and computing metrics for targets and crowd targets.
Implemented the IoUMatching class to maintain backward compatibility with IoU-based metrics.
Added the DistanceMatching class for computing distance-based matching between predictions and targets.

Proposed Changes

The changes have been tested to ensure the accuracy and reliability of the distance-based metrics calculations.
The new classes and methods maintain compatibility with the existing functionality, ensuring backward compatibility.

Testing

Extensive testing has been performed to validate the accuracy and correctness of the metrics calculation.
The test cases cover various scenarios and edge cases for distance-based metrics.

This PR is ready for review and feedback. I would greatly appreciate your suggestions and suggestions for improvement.

BloodAxe · 2023-09-22T17:25:03Z

Thanks for your PR. It looks like your PR contains commits that are not signed. We required all commits in the PR to be signed. You can use this suggestion to address that: #1465 (comment)

Maybe I'm missing something, but the whole idea of the PR is to allow using some other metric when matching pred vs gt boxes. So I have few questions regarding this:

What are the use cases when one may want to use other metric?
Can you provide any papers / project using custom bbox similarity metric?

I'm just not convinced this feature would be relevant for many users.

DimaBir · 2023-09-22T18:32:41Z

Thanks for your PR. It looks like your PR contains commits that are not signed. We required all commits in the PR to be signed. You can use this suggestion to address that: #1465 (comment)

Maybe I'm missing something, but the whole idea of the PR is to allow using some other metric when matching pred vs gt boxes. So I have few questions regarding this:

What are the use cases when one may want to use other metric?

Can you provide any papers / project using custom bbox similarity metric?

I'm just not convinced this feature would be relevant for many users.

Thank you for your feedback, and I apologize for the oversight regarding the signed commits. I will address this as soon as possible by signing all the commits in the PR according to the repository's guidelines. I will also review the suggestions in #1465 (comment) to ensure compliance.

Regarding your queries about the use of different metrics for bounding box similarity, here are my thoughts:

1. Use Cases for Different Metrics:

Distance-Based Metric:
The task was to implement the distance-based metric to enhance existing Detection Metric to allow metrics based on distances from the bboxes' centers, particularly for small bboxes less than 5 pixels.
Custom Bbox Similarity Metrics in Research:
The implementation of custom bbox similarity metrics is crucial, especially in the context of small object detection. Standard metrics like Intersection over Union (IoU) may not be suitable due to the specific challenges posed by small objects, necessitating the development of more tailored approaches.

2. References to Papers/Projects:

As an example of custom metrics we can look at article titled "A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit" by Rafael Padilla, Wesley L. Passos, Thadeu L. B. Dias, Sergio L. Netto, and Eduardo A. B. da Silva, provides a comprehensive overview of the most relevant evaluation methods used in object detection competitions.

7.1. Spatio-Temporal Tube Average Precision:
The article proposes a new metric for evaluating video object detection, called the Spatio-Temporal Tube Average Precision. This metric is based on the spatio-temporal overlap between the ground truth and detected bounding boxes, emphasizing the importance of considering both spatial and temporal dimensions in object detection tasks, especially in video analysis.

3. Relevance of the Feature:

While the relevance of this feature might take time to become apparent, providing the flexibility to use different metrics can cater to a wider range of use cases and user needs.

…ics (Deci-AI#1463) - Introduced a new subclass, DetectionMetricsDistanceBased, to implement distance-based definitions of true positives, enhancing the existing DetectionMetrics class. - Developed distance-based detection matching using Euclidean and Manhattan distance metrics, allowing for more versatile and accurate object detection evaluations. - Added unit tests to ensure the reliability and accuracy of the new distance-based detection matching methodologies. - Maintained backward compatibility by implementing IoUMatching and introduced DistanceMatching for new distance-based matching strategies. - Refactored and cleaned up the code for better readability and compliance with coding standards, including applying Black for formatting.

BloodAxe · 2023-11-29T13:03:44Z

@DimaBir I would like to proceed with this PR and get it merged. Can you please fix the merge conflict?

DimaBir · 2023-11-29T13:08:14Z

@DimaBir I would like to proceed with this PR and get it merged. Can you please fix the merge conflict?

Hi @BloodAxe, sure!
I will fix the conflicts ASAP until EoD.

BloodAxe · 2023-11-29T13:12:00Z

Awesome, thanks!

src/super_gradients/training/utils/detection_utils.py

DimaBir · 2023-11-30T09:06:06Z

Hi @BloodAxe, thanks for pointing this out. I will do np, tomorrow is it okay?

I have a few meetings to attend today 🫤

tests/unit_tests/detection_metrics_distance_based_test.py

BloodAxe · 2023-11-30T09:08:16Z

@DimaBir thanks for updating PR. Assuming build passes the overall PR looks good to me except a few inline comments I left. Nothing serious, mostly docs improvement.
However there is one blocker thing that needs to be addressed - this PR still contains unsigned commits and for that reason CI will not allow merging it. Fortunately the fix is pretty simple: https://github.com/Deci-AI/super-gradients/blob/master/CONTRIBUTING.md#i-have-created-pr-with-unsigned-commits-do-i-have-to-start-over

BloodAxe · 2023-11-30T09:10:00Z

Hi @BloodAxe, thanks for pointing this out. I will do np, tomorrow is it okay?

I have a few meetings to attend today 🫤

Feel free to proceed with whatever pace is comfortable for you :) No rush here and thanks for your contribution!

DimaBir · 2023-11-30T09:10:01Z

Sure @BloodAxe I will fix everything and fix commits to signed.

DimaBir · 2023-12-08T10:38:18Z

@DimaBir thanks for updating PR. Assuming build passes the overall PR looks good to me except a few inline comments I left. Nothing serious, mostly docs improvement. However there is one blocker thing that needs to be addressed - this PR still contains unsigned commits and for that reason CI will not allow merging it. Fortunately the fix is pretty simple: https://github.com/Deci-AI/super-gradients/blob/master/CONTRIBUTING.md#i-have-created-pr-with-unsigned-commits-do-i-have-to-start-over

Done, force pushed branch with signed commits

- Updated `compute_detection_matching` to conditionally require `iou_thresholds` only for IoU-based strategies.

Returned check if matching_strategy object is passed

- Updated `compute_img_detection_matching` to conditionally get thresholds using matching_strategy get_thresholds() method. Note: Since thresholds passed via matching_strategy class, consider to remove iou_thresholds param

BloodAxe

LGTM

DimaBir requested review from shaydeci, ofrimasad, BloodAxe and Louis-Dupont as code owners September 13, 2023 11:01

DimaBir force-pushed the distance-based-metrics-refactored branch from 79e2d5c to 1487382 Compare September 23, 2023 08:07

BloodAxe reviewed Nov 30, 2023

View reviewed changes

src/super_gradients/training/utils/detection_utils.py Show resolved Hide resolved

BloodAxe reviewed Nov 30, 2023

View reviewed changes

src/super_gradients/training/utils/detection_utils.py Outdated Show resolved Hide resolved

BloodAxe reviewed Nov 30, 2023

View reviewed changes

tests/unit_tests/detection_metrics_distance_based_test.py Show resolved Hide resolved

Signing unsigned commits in new branch

3ed7359

DimaBir force-pushed the distance-based-metrics-refactored branch from 31ad623 to 3ed7359 Compare December 8, 2023 10:26

Merge branch 'master' into distance-based-metrics-refactored

187b3ed

DimaBir added 3 commits December 8, 2023 13:45

Fix Missing iou_thresholds Argument for Distance-Based Metrics

14445e2

- Updated `compute_detection_matching` to conditionally require `iou_thresholds` only for IoU-based strategies.

Check for matching_strategy object

e99863e

Returned check if matching_strategy object is passed

Fix thresholds len for Distance-Based and IoU Metrics

1ca0b3b

- Updated `compute_img_detection_matching` to conditionally get thresholds using matching_strategy get_thresholds() method. Note: Since thresholds passed via matching_strategy class, consider to remove iou_thresholds param

BloodAxe approved these changes Dec 8, 2023

View reviewed changes

BloodAxe merged commit 953a671 into Deci-AI:master Dec 8, 2023

DimaBir deleted the distance-based-metrics-refactored branch December 22, 2023 11:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Implement Distance-Based Detection Matching in DetectionMetrics #1463

[Enhancement] Implement Distance-Based Detection Matching in DetectionMetrics #1463

DimaBir commented Sep 13, 2023 •

edited

Loading

BloodAxe commented Sep 22, 2023

DimaBir commented Sep 22, 2023 •

edited

Loading

BloodAxe commented Nov 29, 2023

DimaBir commented Nov 29, 2023 •

edited

Loading

BloodAxe commented Nov 29, 2023

DimaBir commented Nov 30, 2023

BloodAxe commented Nov 30, 2023

BloodAxe commented Nov 30, 2023

DimaBir commented Nov 30, 2023

DimaBir commented Dec 8, 2023

BloodAxe left a comment

[Enhancement] Implement Distance-Based Detection Matching in DetectionMetrics #1463

[Enhancement] Implement Distance-Based Detection Matching in DetectionMetrics #1463

Conversation

DimaBir commented Sep 13, 2023 • edited Loading

Pull Request Description

Changes Made

Proposed Changes

Testing

BloodAxe commented Sep 22, 2023

DimaBir commented Sep 22, 2023 • edited Loading

1. Use Cases for Different Metrics:

2. References to Papers/Projects:

3. Relevance of the Feature:

BloodAxe commented Nov 29, 2023

DimaBir commented Nov 29, 2023 • edited Loading

BloodAxe commented Nov 29, 2023

DimaBir commented Nov 30, 2023

BloodAxe commented Nov 30, 2023

BloodAxe commented Nov 30, 2023

DimaBir commented Nov 30, 2023

DimaBir commented Dec 8, 2023

BloodAxe left a comment

Choose a reason for hiding this comment

DimaBir commented Sep 13, 2023 •

edited

Loading

DimaBir commented Sep 22, 2023 •

edited

Loading

DimaBir commented Nov 29, 2023 •

edited

Loading