Metrics

[ToucheSir]

Metrics. Everybody needs them, nobody knows which ones to use! Here is a necessarily non-exhaustive list of packages that have them (in alphabetical order):

• https://github.com/rizalzaf/AdversarialPrediction.jl
• https://github.com/dfdx/Avalon.jl
• https://github.com/sylvaticus/BetaML.jl
• https://github.com/PharmCat/ClinicalTrialUtilities.jl
• https://github.com/JuliaStats/Distances.jl
• https://github.com/FluxML/FastAI.jl/
• https://github.com/VaclavMacha/EvalMetrics.jl
• https://github.com/sadit/KernelMethods.jl
• https://github.com/KnetML/KnetMetrics.jl
• https://github.com/beacon-biosignals/Lighthouse.jl
• https://github.com/JuliaML/LossFunctions.jl
• https://github.com/zhmz90/Metrics.jl
• https://github.com/JuliaStats/MLBase.jl
• https://github.com/JuliaAI/MLJBase.jl https://github.com/JuliaAI/StatisticalMeasures.jl
• https://github.com/JuliaML/MLMetrics.jl
• https://github.com/milanflach/MultivariateAnomalies.jl
• https://github.com/bcbi/PredictMD.jl
• https://github.com/davidavdav/ROCAnalysis.jl
• https://github.com/cossio/ROCCurves.jl

Getting lost among packages is certainly not a theoretical concern, as evidenced by frequent questions such as https://discourse.julialang.org/t/r2-in-mlj-machine-learning/66310.

Prior work has looked at API coverage for certain packages, but I don't believe there's been much if any communication between packages or package ecosystems. At the risk of invoking XKCD #927, I think there are opportunities here for deduplication. Thoughts and ideas very much welcome.

[ablaom]

List of metrics provided by JuliaAI/MLJBase.

[zjpi]

A centralized repository of well-implemented algorithms would be highly appreciated.

But a lot of work needs to go into the design and community feedback. Otherwise, we just end up with one more package.

[ToucheSir]

You'll notice I invoked https://xkcd.com/927/ in the top post ;)

Ideally we'd have a similar community building process as happened for the fancy array discussions. The good news is that this seems like an easier problem to make progress on than fancy arrays. The hard part is that there is less visibility and not all of the package maintainers are as engaged (if not completely inactive on GH) compared to that ecosystem.

[ablaom]

One of the issues that likely led to the fragmentation is competing use-cases. Consider, for example, the ubiquitous and seemingly innocuous log loss (aka cross entropy, or negative log score). What might the arguments in a call log_loss(yhat, y) be?

• If you only care about binary classification and you want maximum performance, then maybe y is Bool and yhat is some Real.
• For vectors (representing multiple observations) or arrays (representing batches of observations) you're probably happy with broadcasting this method.
• If you want also multi-class classification (n > 2) you might argue for y an Integer and yhat an (n-1)-tuple of Real for consistency with n=2) or an n-tuple (because that you don't want to insist the "probabilities" are normalised).
• On the other hand, if you're dealing with labelled data (eg, classes "yes", "no", "maybe") and you want to mitigate ambiguity about how these are ordered, then y is a CategoricalArray (possibly unordered) and yhat a finite distribution with labelled sample space (what we call UnivariateFinite in MLJ). (In this case broadcasting a scalar method is horrible, because of the overheads of wrapping probability vectors as distrubutions, for example, and you need to implement an abstract array type for your distributions, and so forth.) Such an implementation is not going to be used to train neural networks!
• If you are coming from the world of probabilistic programming, you do not even have a probability mass function for your distribution yhat - all you have is a "sampleable" distribution. In that case, you'll need extra parameters to control the accuracy of your estimate (eg, sample size). That is, you'll need an interface point for parameters no-one else cares about.

The point I want to make is that no single package should implement all this functionality for log loss. This is not the only issue facing us, but I suggest it's the one that should be addressed first.

What would be useful, from my point-of-view, is a MetricsBase package that:

1. Defines the parameterised types of all metrics ("metrics" loosely understood here)
2. Gives a wiki-link or paper as minimum doc-string (or could be a trait).
3. Implements zero functionality
4. Possibly organises the metrics in a type hierarchy, or with traits, or with a combination of the two.

1-3 are doable. I suggest getting consensus around 4 will be challenging (I've been there before!). Which may be an argument for keeping the type heirarchy very basic, because traits can be strapped on later, or by client packages.

Thoughts?

[ablaom]

This package would also function as a "TODO" list for client packages that implement functionality. Next time I get a request to implement such-and-such, I can just say "add the type to MetricsBase" and we'll get around to it sometime 😄

[ToucheSir]

I have a similar trepidation about trying to do 4). However, I think we can get around that by being a little less ambitious:

• Start by grouping together redundant packages that have unambiguously related metrics. For example, we do not need 5 versions of AUROC. However, trying to unify metrics derived from the confusion matrix with distance based ones would be out of scope for at least the near future.
• When defining common base packages, favour the lowest denominator. That means simple, low-level functions, few if any types and as AbstractArray-friendly as possible. Dependent packages can always define their own optimized versions for custom array types, but having that fallback is important to ensure a base level of coverage.

So for the log loss example, each of point 1-3 would be a separate function/overload. Yes MLJ or a PPL library could implement a method for fancier types, but if they don't I should still be able to use it, albeit less efficiently or not idiomatically. Common libraries need not be ergonomic, nor do they need to give optimal performance.

[ablaom]

That means simple, low-level functions, few if any types and as AbstractArray-friendly as possible.

You are right. Having pure functions makes sense. However, types are inevitable at some level (users don't want to do metric = (yhat, y) -> lploss(yhat, y, p=3) and anonymous functions don't display nicely) which begets the question, Where do these type live? if not in a base package?

[ablaom]

MLJBase.jl will shortly remove it's built-in metrics, which are to be provided instead by a new external package StatisticalMeasures.jl. Also new is StatisticalMeausuresBase.jl which specifies an API for metrics and wrappers for adding functionality.

@ToucheSir Perhaps you could please update your list above accordingly.

[ToucheSir]

Congratulations and done :)

[ablaom]

Thanks!

[KronosTheLate]

This is a great resource. Could copy the list for a page on juliapackagecomparisons.github.io?

[ToucheSir]

Go for it.

[KronosTheLate]

Great! It should be live here in a few moments. If you know any more information that users would benefit from when they are tying to choose, we would love a PR where you write a little about the options <3