Fix F1 auto-threshold to choose best largest confidence

CHANGELOG.md

@@ -8,31 +8,32 @@ All notable changes to this project will be documented in this file.
 
 - Support encrypted dataset training (<https://github.com/openvinotoolkit/training_extensions/pull/2209>)
 - Add custom max iou assigner to prevent CPU OOM when large annotations are used (<https://github.com/openvinotoolkit/training_extensions/pull/2228>)
-- Auto train type detection for Semi-SL, Self-SL and Incremental: "--train-type" now is optional (https://github.com/openvinotoolkit/training_extensions/pull/2195)
-- Add per-class XAI saliency maps for Mask R-CNN model (https://github.com/openvinotoolkit/training_extensions/pull/2227)
+- Auto train type detection for Semi-SL, Self-SL and Incremental: "--train-type" now is optional (<https://github.com/openvinotoolkit/training_extensions/pull/2195>)
+- Add per-class XAI saliency maps for Mask R-CNN model (<https://github.com/openvinotoolkit/training_extensions/pull/2227>)
 - Add new object detector Deformable DETR (<https://github.com/openvinotoolkit/training_extensions/pull/2249>)
 - Add new object detector DINO(<https://github.com/openvinotoolkit/training_extensions/pull/2266>)
-- Add new visual prompting task: train/eval (https://github.com/openvinotoolkit/training_extensions/pull/2203)
-- Add new visual prompting task: export (https://github.com/openvinotoolkit/training_extensions/pull/2274)
-- Add new visual prompting task: deploy (https://github.com/openvinotoolkit/training_extensions/pull/2311)
-- Add new visual prompting task: documentation (https://github.com/openvinotoolkit/training_extensions/pull/2354)
-- Add new visual prompting task: optimize (PTQ) (https://github.com/openvinotoolkit/training_extensions/pull/2318)
+- Add new visual prompting task: train/eval (<https://github.com/openvinotoolkit/training_extensions/pull/2203>)
+- Add new visual prompting task: export (<https://github.com/openvinotoolkit/training_extensions/pull/2274>)
+- Add new visual prompting task: deploy (<https://github.com/openvinotoolkit/training_extensions/pull/2311>)
+- Add new visual prompting task: documentation (<https://github.com/openvinotoolkit/training_extensions/pull/2354>)
+- Add new visual prompting task: optimize (PTQ) (<https://github.com/openvinotoolkit/training_extensions/pull/2318>)
 - Add new object detector ResNeXt101-ATSS (<https://github.com/openvinotoolkit/training_extensions/pull/2309>)
 
 ### Enhancements
 
 - Introduce channel_last parameter to improve the performance (<https://github.com/openvinotoolkit/training_extensions/pull/2205>)
 - Decrease a time for making a workspace (<https://github.com/openvinotoolkit/training_extensions/pull/2223>)
 - Set persistent_workers and pin_memory as True in detection task (<https://github.com/openvinotoolkit/training_extensions/pull/2224>)
-- New algorithm for Semi-SL semantic segmentation based on metric lerning via class prototypes (https://github.com/openvinotoolkit/training_extensions/pull/2156)
-- Self-SL for classification now can recieve just folder with any images to start contrastive pretraining (https://github.com/openvinotoolkit/training_extensions/pull/2219)
+- New algorithm for Semi-SL semantic segmentation based on metric lerning via class prototypes (<https://github.com/openvinotoolkit/training_extensions/pull/2156>)
+- Self-SL for classification now can recieve just folder with any images to start contrastive pretraining (<https://github.com/openvinotoolkit/training_extensions/pull/2219>)
 - Update OpenVINO version to 2023.0, and NNCF verion to 2.5 (<https://github.com/openvinotoolkit/training_extensions/pull/2090>)
-- Improve XAI saliency map generation for tiling detection and tiling instance segmentation (https://github.com/openvinotoolkit/training_extensions/pull/2240)
+- Improve XAI saliency map generation for tiling detection and tiling instance segmentation (<https://github.com/openvinotoolkit/training_extensions/pull/2240>)
 
 ### Bug fixes
 
 - Fix the bug that auto adapt batch size is unavailable with IterBasedRunner (<https://github.com/openvinotoolkit/training_extensions/pull/2182>)
 - Fix the bug that learning rate isn't scaled when multi-GPU trianing is enabled(<https://github.com/openvinotoolkit/training_extensions/pull/2254>)
+- Fix F1 auto-threshold to choose best largest confidence (<https://github.com/openvinotoolkit/training_extensions/pull/2367>)
 
 ### Known issues
 

src/otx/api/usecases/evaluation/f_measure.py

@@ -1,10 +1,8 @@
 """This module contains the f-measure performance provider class."""
-
-# Copyright (C) 2021-2022 Intel Corporation
+# Copyright (C) 2021-2023 Intel Corporation
 # SPDX-License-Identifier: Apache-2.0
 #
 
-
 import logging
 from typing import Dict, List, Optional, Tuple
 
@@ -363,7 +361,7 @@ def get_results_per_confidence(
                 result.f_measure_curve[class_name].append(result_point[class_name].f_measure)
                 result.precision_curve[class_name].append(result_point[class_name].precision)
                 result.recall_curve[class_name].append(result_point[class_name].recall)
-            if all_classes_f_measure > result.best_f_measure:
+            if all_classes_f_measure > 0.0 and all_classes_f_measure >= result.best_f_measure:
                 result.best_f_measure = all_classes_f_measure
                 result.best_threshold = confidence_threshold
         return result
@@ -417,7 +415,7 @@ def get_results_per_nms(
                 result.precision_curve[class_name].append(result_point[class_name].precision)
                 result.recall_curve[class_name].append(result_point[class_name].recall)
 
-            if all_classes_f_measure >= result.best_f_measure:
+            if all_classes_f_measure > 0.0 and all_classes_f_measure >= result.best_f_measure:
                 result.best_f_measure = all_classes_f_measure
                 result.best_threshold = nms_threshold
         return result

tests/unit/api/usecases/evaluation/test_f_measure.py

@@ -1,16 +1,6 @@
-# Copyright (C) 2020-2021 Intel Corporation
+# Copyright (C) 2020-2023 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
 #
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions
-# and limitations under the License.
 
 import datetime
 from typing import cast
@@ -962,7 +952,7 @@ def test_f_measure_calculator_get_results_per_confidence(self):
         # Check "_AggregatedResults" object returned by "get_results_per_confidence" when All Classes f-measure is more
         # than best f-measure in results_per_confidence
         expected_results_per_confidence = _AggregatedResults(["class_1", "class_2"])
-        for confidence_threshold in np.arange(*[0.6, 0.9]):
+        for confidence_threshold in np.arange(*[0.6, 0.9, 0.1]):
             result_point = f_measure_calculator.evaluate_classes(
                 classes=["class_1", "class_2"],
                 iou_threshold=0.7,
@@ -978,7 +968,7 @@ def test_f_measure_calculator_get_results_per_confidence(self):
 
         actual_results_per_confidence = f_measure_calculator.get_results_per_confidence(
             classes=["class_1", "class_2"],
-            confidence_range=[0.6, 0.9],
+            confidence_range=[0.6, 0.9, 0.1],  # arrange(0.6, 0.9, 0.1)
             iou_threshold=0.7,
         )
         assert actual_results_per_confidence.all_classes_f_measure_curve == (
@@ -987,7 +977,9 @@ def test_f_measure_calculator_get_results_per_confidence(self):
         assert actual_results_per_confidence.f_measure_curve == expected_results_per_confidence.f_measure_curve
         assert actual_results_per_confidence.recall_curve == expected_results_per_confidence.recall_curve
         assert actual_results_per_confidence.best_f_measure == 0.5454545454545453
-        assert actual_results_per_confidence.best_threshold == 0.6
+        # 0.6 -> 0.54, 0.7 -> 0.54, 0.8 -> 0.54, 0.9 -> 0.44
+        # Best ""LARGEST" trehshold should be 0.8 (considering numerical error)
+        assert abs(actual_results_per_confidence.best_threshold - 0.8) < 0.001
         # Check "_AggregatedResults" object returned by "get_results_per_confidence" when All Classes f-measure is less
         # than best f-measure in results_per_confidence
         actual_results_per_confidence = f_measure_calculator.get_results_per_confidence(