Introduce additional active learning sampling strategies

[PawelPeczek-Roboflow]

Description

On top of previously added Active Learning capabilities, we are adding 3 new sampling strategies for Active Learning:

Close to threshold sampling

Description

Sampling method to be used when one wants to sample datapoints causing certain predictions confidences for certain classes. Works for both detection and classification models - although the behaviour slightly differ.

Degrees of freedom in configuration

• selected_class_names - class names to be taken into consideration while sampling - optional (if not given - all classes can be sampled)
• threshold and epsilon - represent centre and radius for confidence range that can trigger sampling - for instance - if one is interested to get datapoints that classifier is highly confident (0.8, 1.0) -> threshold=0.9, epsilon=0.1 - this is however limited to the outcomes of model post-processing (and threshold filtering)
• probability - fraction of datapoints that matches sampling criterions that will be persisted
• minimum_objects_close_to_threshold - (for detection predictions only) - specify how many detected objects from selected classes must be close to threshold to accept datapoint
• only_top_classes - (for classification predictions only) - flag to decide if only top, or predicted_classes (for multi-class / multi-label cases respectively) classes should be taken into consideration (to avoid sampling based on non-leading classes in predictions)

Example configuration

{
            "name": "hard_examples",
            "type": "close_to_threshold",
            "selected_class_names": ["a", "b"],
            "threshold": 0.25,
            "epsilon": 0.1,
            "probability": 0.5,
            "tags": ["b"],
            "limits": [
                {"type": "minutely", "value": 10},
                {"type": "hourly", "value": 100},
                {"type": "daily", "value": 1000},
            ],
        },

Classes based sampling (for classification)

Description

Sampling method to be used when one wants to sample specific classes from classifier predictions

Degrees of freedom in configuration

• selected_class_names - class names to be taken into consideration while sampling - required
• probability - fraction of datapoints that matches sampling criterions that will be persisted

Example configuration

 {
                "name": "underrepresented_classes",
                "type": "classes_based",
                "selected_class_names": ["cat"],
                "probability": 1.0,
                "tags": ["hard-classes"],
                "limits": [
                    {"type": "minutely", "value": 10},
                    {"type": "hourly", "value": 100},
                    {"type": "daily", "value": 1000},
                ],
            },

Detection number based sampling (for detection)

Description

Sampling method to be used when one wants to sample specific detections (based on count and classes)

Degrees of freedom in configuration

• selected_class_names - class names to be taken into consideration while sampling - optional (if not given - all classes can be sampled)
• probability - fraction of datapoints that matches sampling criterions that will be persisted
• more_than - minimal number of detected objects (optional - if not given - lower limit is not applied)
• less_than - maximum number of detected objects (optional - if not given - upper limit is not applied)
• one of more_than, less_than must be given

Example configuration

{
                "name": "multiple_detections",
                "type": "detections_number_based",
                "probability": 0.2,
                "more_than": 3,
                "tags": ["crowded"],
                "limits": [
                    {"type": "minutely", "value": 10},
                    {"type": "hourly", "value": 100},
                    {"type": "daily", "value": 1000},
                ],
            },

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update (deferred waiting for AL intreoduction)

How has this change been tested, please provide a testcase or example of how you tested the change?

• automated tests added
• tested e2e locally

Any specific deployment considerations

For example, documentation changes, usability, usage/costs, secrets, etc.

Docs

• Docs updated? What were the changes:

[paulguerrie]

Looks great! This is going to be really powerful for users!!