Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Implement AucRoc metric for classification #60502

Merged
merged 21 commits into from
Sep 30, 2020

Conversation

przemekwitek
Copy link
Contributor

@przemekwitek przemekwitek commented Jul 31, 2020

This PR implements AucRoc metric for evaluating classification.
It achieves it the following way:

  • adds new field (top_classes_field) to the Classification evaluation object
  • adds new metric (AucRoc) under classification package
  • adds new method (EvaluationMetric::getRequiredFields) by which each metric may request fields it needs for its calculation

It also introduces a few refactorings:

  • field names interesting to the evaluation flow are now stored in the EvaluationFields object (reused by all the evaluation types)
  • EvaluationMetric::aggs now accepts the new EvaluationFields object instead of the individual field names
  • calculation code is shared between classification.AucRoc and outlierdetection.AucRoc classes using a base class (AbstractAucRoc)

The newly added fields are never indexed (persisted) so they don't require mappings.

Relates #62160

@przemekwitek przemekwitek changed the title [DRAFT] Implement AucRoc metric for classification [DRAFT] [ML] Implement AucRoc metric for classification Aug 24, 2020
@przemekwitek przemekwitek force-pushed the auc_roc branch 7 times, most recently from cd6d20a to 2b1f9f3 Compare August 27, 2020 07:25
@przemekwitek przemekwitek force-pushed the auc_roc branch 6 times, most recently from 3f5f0c9 to 9c70df7 Compare September 9, 2020 11:13
@przemekwitek przemekwitek changed the title [DRAFT] [ML] Implement AucRoc metric for classification [ML] Implement AucRoc metric for classification Sep 9, 2020
@przemekwitek przemekwitek force-pushed the auc_roc branch 2 times, most recently from 95c85aa to 1dc8a0d Compare September 9, 2020 12:48
@przemekwitek przemekwitek added WIP and removed WIP labels Sep 9, 2020
@przemekwitek przemekwitek force-pushed the auc_roc branch 8 times, most recently from f2c19b3 to a366fbb Compare September 10, 2020 08:26
@przemekwitek przemekwitek removed the WIP label Sep 10, 2020
@przemekwitek
Copy link
Contributor Author

run elasticsearch-ci/packaging-sample-windows

@przemekwitek
Copy link
Contributor Author

LGTM The only open question left is how we deal with docs whose actual field is the class in question, the point that Tom raised.

I'll discuss the necessary changes and push them in a separate PR as this one has grown big already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants