Question about "Unknown" classification

[Plasma]

Hello,

When using a multiclass classifier, eg StochasticDualCoordinateAscentClassifier, the scores are all relative to 1.

What is the best way to detect when there are no suitable matches, rather than the classifier leaning towards one of them?

thanks

[codemzs]

Hi @Plasma ,

For a given feature array every label is scored and the one with highest score is selected as "PredictedLabel". Every label is a match or no match up to a certain magnitude. Does that answer your question?

[Plasma]

Hey @codemzs ,

I think it confirms my understanding, but to give a more concrete example:

Say I have a multi-class classifier that has been trained to match against Red, Green and Blue colors.

If I then present a Yellow color, something it has never seen before, in my testing I will still get back scores that add up to 1.0, eg 0.1 red, 0.7 green, 0.2 blue (0.1 + 0.7 + 0.2 == 1.0) and thus are all just a ratio between each other. If I were to have an internal cutoff of at least 70% score for a predicted label to be correct, I'd incorrectly pick Green here, instead of the score being too low to match.

I'd ideally expect, having not seen Yellow before, the scores returned are more likely say 0.1, 0.1, 0.2 -- or something that is very low score for all known classifications. But it seems all scores returned always add up to 1.0, so for an unknown class being classified, it can still strongly appear as a known class.

If my understanding is correct, that all scores from the classifier add up to 1.0 and are just ratios between each other, then I can't easily tell when the classifier has no idea what the presented color was.

[TomFinley]

The trouble though is, typical machine learning methods don't attempt to model unknown classes -- machine learning methods are trained on labeled data, they simultaneously make an IID assumption about their test data being similarly distributed. With typical loss functions for classification being softmax, they aren't even trying to model the problem of encountering data that looks different. It just isn't part of what they're trying to do, at all.

That said, the problem you've described is important, but the solution is actually to introduce a different method. Sometimes called one-class methods or more frequently nowadays anomaly detection models. The one we've migrated so-far is PcaAnomalyDetector (not in ML.NET v0.1, but surfaced AFAIK in #221 so will be in v0.2).

[Plasma]

Thank you @TomFinley for the explanation. Looking forward to seeing more developments in ML.NET

[sfilipi]

Closing since the question seems answered.