Implement `precision` and `recall` metrics for classification evaluation #49671

przemekwitek · 2019-11-28T06:38:45Z

This PR implements precision and recall metrics for classification evaluation.

Additionally, it:

registers evaluation metrics in XContent and Writeable registries under qualified names in order to avoid name clashes
changes the interface of EvaluationMetric interface by allowing pipeline aggregations to be requested by aggs method
moves actualIsTrueQuery method from interface to implementation as this method is implementation-specific

Relates #48759

elasticmachine · 2019-11-28T09:14:19Z

Pinging @elastic/ml-core (:ml)

przemekwitek · 2019-12-04T08:55:17Z

run elasticsearch-ci/1
run elasticsearch-ci/packaging-sample-matrix

dimitris-athanasiou

Leaving a few comments. I'll revisit when recall has been adjusted.

...ain/java/org/elasticsearch/client/ml/dataframe/evaluation/classification/AccuracyMetric.java

...in/java/org/elasticsearch/client/ml/dataframe/evaluation/classification/PrecisionMetric.java

dimitris-athanasiou

Some more comments as I go along the way.

dimitris-athanasiou · 2019-12-17T11:27:21Z

...main/java/org/elasticsearch/xpack/core/ml/dataframe/evaluation/classification/Precision.java

+            return Tuple.tuple(
+                List.of(
+                    AggregationBuilders.terms(ACTUAL_CLASSES_NAMES_AGG_NAME)
+                        .field(actualField)


Do we also need a size parameter in all these like in the multiclass confusion matrix?

I've just added size parameter to Precision and Recall metrics.
Also, I've added other_class_count parameter to Precision.Result and Recall.Result so that the user can tell if the result is complete.

I've just added size parameter to Precision and Recall metrics.
Also, I've added other_class_count parameter to Precision.Result and Recall.Result so that the user can tell if the result is complete.

I reverted this change and implemented max cardinality enforcement as discussed. PTAL

dimitris-athanasiou · 2019-12-17T11:31:08Z

...main/java/org/elasticsearch/xpack/core/ml/dataframe/evaluation/classification/Precision.java

+                String className = bucket.getKeyAsString();
+                NumericMetricsAggregation.SingleValue precisionAgg = bucket.getAggregations().get(PER_PREDICTED_CLASS_PRECISION_AGG_NAME);
+                double precision = precisionAgg.value();
+                if (Double.isFinite(precision)) {


Should we be checking this? If for some reason precision is not finite, we'll end up reporting zero instead of NaN or infinity.

We will not report zero but rather we will not report this particular class.
My reasoning behind this condition is that if we have an actual class that is never predicted (e.g: "cat"), precision for "cats" cannot be calculated so there is no point in reporting a precision entry with NaN value.

przemekwitek · 2019-12-19T08:17:33Z

run elasticsearch-ci/default-distro
run elasticsearch-ci/bwc

przemekwitek · 2019-12-19T08:25:41Z

run elasticsearch-ci/bwc

…tionMetric, RegressionMetric)

…ficationIT

This reverts commit 81be647d6465e62971ac763605be4a080161cbdf.

…l metrics

dimitris-athanasiou

LGTM

…valuation (#49671) (#50378)

…ion (elastic#49671)

przemekwitek added the WIP label Nov 28, 2019

cbuescher added the :ml Machine learning label Nov 28, 2019

przemekwitek force-pushed the precision_and_recall branch 12 times, most recently from 91d83e0 to 3edeb6b Compare December 3, 2019 14:45

przemekwitek removed the WIP label Dec 3, 2019

przemekwitek marked this pull request as ready for review December 3, 2019 15:00

przemekwitek added >feature v7.6.0 v8.0.0 labels Dec 3, 2019

dimitris-athanasiou self-requested a review December 4, 2019 11:36

przemekwitek force-pushed the precision_and_recall branch 4 times, most recently from 2b97424 to 93fc929 Compare December 11, 2019 09:41

przemekwitek mentioned this pull request Dec 11, 2019

Implement accuracy, precision and recall metrics for multiclass classification evaluation. #48759

Closed

3 tasks

przemekwitek force-pushed the precision_and_recall branch from 93fc929 to 726c822 Compare December 12, 2019 07:54

dimitris-athanasiou reviewed Dec 12, 2019

View reviewed changes

przemekwitek force-pushed the precision_and_recall branch from 5249855 to 86dcaf3 Compare December 13, 2019 12:49

dimitris-athanasiou reviewed Dec 17, 2019

View reviewed changes

przemekwitek force-pushed the precision_and_recall branch 2 times, most recently from 6c6406d to 6f254d8 Compare December 18, 2019 13:30

przemekwitek added 13 commits December 19, 2019 13:33

Implement precision and recall metrics for classification evaluation

97bea88

Fix compile errors

1863ef5

Remove markup interface classes (SoftClassificationMetric, Classifica…

8365ad3

…tionMetric, RegressionMetric)

Apply review comments

ccf5c30

Revert changes related to accuracy

26bb9bf

Add Precision and Recall metrics to evaluation verification in Classi…

935a26c

…ficationIT

Add size parameter to Precision and Recall metrics

fc080be

Fix MlClientDocumentationIT

8376bb6

Relax assertEvaluation method in ClassificationIT

28be815

Revert "Add size parameter to Precision and Recall metrics"

db9c4f6

This reverts commit 81be647d6465e62971ac763605be4a080161cbdf.

Enforce max cardinality of actual_class field for precision and recal…

452dfc8

…l metrics

Fix MlClientDocumentationIT

7f571ae

Simplify handling actualField in process() method

dd74939

przemekwitek force-pushed the precision_and_recall branch from d820384 to dd74939 Compare December 19, 2019 12:53

dimitris-athanasiou reviewed Dec 19, 2019

View reviewed changes

przemekwitek merged commit 786ead6 into elastic:master Dec 19, 2019

przemekwitek deleted the precision_and_recall branch December 19, 2019 15:07

przemekwitek mentioned this pull request Dec 19, 2019

[7.x] Implement precision and recall metrics for classification evaluation (#49671) #50378

Merged

przemekwitek added a commit that referenced this pull request Dec 19, 2019

[7.x] Implement precision and recall metrics for classification e…

cc4bc79

…valuation (#49671) (#50378)

przemekwitek mentioned this pull request Dec 20, 2019

Get rid of maxClassesCardinality internal parameter #50418

Merged

SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this pull request Jan 23, 2020

Implement precision and recall metrics for classification evaluat…

d1b4000

…ion (elastic#49671)

This was referenced Feb 3, 2020

[meta] 7.6 release elastic/elasticsearch-net#4340

Closed

[meta] 7.6 release elastic/elasticsearch-net#4341

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement `precision` and `recall` metrics for classification evaluation #49671

Implement `precision` and `recall` metrics for classification evaluation #49671

przemekwitek commented Nov 28, 2019 •

edited

Loading

elasticmachine commented Nov 28, 2019

przemekwitek commented Dec 4, 2019

dimitris-athanasiou left a comment

dimitris-athanasiou left a comment

dimitris-athanasiou Dec 17, 2019

przemekwitek Dec 18, 2019

przemekwitek Dec 18, 2019

dimitris-athanasiou Dec 17, 2019

przemekwitek Dec 17, 2019

przemekwitek commented Dec 19, 2019

przemekwitek commented Dec 19, 2019

dimitris-athanasiou left a comment

Implement precision and recall metrics for classification evaluation #49671

Implement precision and recall metrics for classification evaluation #49671

Conversation

przemekwitek commented Nov 28, 2019 • edited Loading

elasticmachine commented Nov 28, 2019

przemekwitek commented Dec 4, 2019

dimitris-athanasiou left a comment

Choose a reason for hiding this comment

dimitris-athanasiou left a comment

Choose a reason for hiding this comment

dimitris-athanasiou Dec 17, 2019

Choose a reason for hiding this comment

przemekwitek Dec 18, 2019

Choose a reason for hiding this comment

przemekwitek Dec 18, 2019

Choose a reason for hiding this comment

dimitris-athanasiou Dec 17, 2019

Choose a reason for hiding this comment

przemekwitek Dec 17, 2019

Choose a reason for hiding this comment

przemekwitek commented Dec 19, 2019

przemekwitek commented Dec 19, 2019

dimitris-athanasiou left a comment

Choose a reason for hiding this comment

Implement `precision` and `recall` metrics for classification evaluation #49671

Implement `precision` and `recall` metrics for classification evaluation #49671

przemekwitek commented Nov 28, 2019 •

edited

Loading