Changes to mxnet.metric #18083

acphile · 2020-04-16T09:56:47Z

Description

change based on #18046

make improvements in metric
a. improve Class MAE (and MSE, RMSE)
b. improve Class _BinaryClassification
c. improve Class TopKAccuracy
d. add Class MeanCosineSimilarity
e. add Class MeanPairwiseDistance
move mxnet.metric to mxnet.gluon.metric

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Feature1, tests, (and when applicable, API doc)
Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

…thon/mxnet/

mxnet-bot · 2020-04-16T09:56:51Z

Hey @acphile , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

To trigger all jobs: @mxnet-bot run ci [all]
To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [centos-cpu, sanity, miscellaneous, centos-gpu, windows-cpu, website, unix-gpu, edge, windows-gpu, clang, unix-cpu]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

sxjscience · 2020-04-16T15:41:50Z

python/mxnet/gluon/metric.py

+        for label, pred in zip(labels, preds):
+            self.metrics.update_binary_stats(label, pred)
+
+        if self.average == "macro":


In fact, macro averaging + F1 does not mean to average the F1 of each batch. I think we should revise it to be the same as https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html .

Averaging F1 per batch here previously existed in metric.py before I made changes. This calculation also exists in MAE, MSE, RMSE, and PearsonCorrelation. Should I remove all of them accordingly? For the average "macro" in sklearn, it seems used in calculating F1 score for multiclass/multilabel targets. But currently our F1 only supports binary classification. I think I need to make extensions for F1.

Let's remove it and make it similar to sklearn. This is in fact the reason why I never use the metric class in MXNet.

sxjscience · 2020-04-16T15:43:35Z

python/mxnet/gluon/metric.py

-
-            mae = numpy.abs(label - pred).mean()
+
+            if self.average == "macro":


It's very strange to have macro + MAE. See scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error

leezu

Thank you @acphile! Some comments

@sxjscience suggested compatibility with sklearn's metrics. If so, we should have a mechanism to ensure compatibility / correctness. One way to ensure this is to compare to add tests that compare the output of sklearn to the output of the gluon metric for different inputs. Such test may even include random data to ensure compatibility in edge cases (cf https://en.wikipedia.org/wiki/Fuzzing)
We currently support get() vs. get_global(), reset() vs. reset_local(), but in fact the global functionality is not used anywhere in MXNet and there may not be a good widely used use-case for it. To make our metric API more pythonic and easier to understand, we may remove the global support.
@sxjscience suggests to remove the macro support because it's not correct and not widely used
Your code needs to pass the sanity checks for coding style http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fsanity/detail/PR-18083/1/pipeline

leezu · 2020-04-22T01:36:43Z

python/mxnet/gluon/metric.py

-            return 2 * self.precision * self.recall / (self.precision + self.recall)
-        else:
-            return 0.
+        return (1 + self.beta ** 2) * self.precision * self.recall / numpy.maximum(self.beta ** 2 * self.precision + self.recall, 1e-12)

    @property
    def global_fscore(self):


This method should be removed as you dropped the global states?

This method actually refers to the micro calculation for F1 and it is not related to original global support.

Would it make sense to adjust the name?

I think it is ok to use global_fscore since it is in a private container class.

leezu · 2020-04-22T02:01:54Z

python/mxnet/gluon/metric.py

@@ -24,9 +24,9 @@

 import numpy


Instead of using numpy, we can use mxnet.numpy as it runs asynchronously and has GPU support.
The summary states of a metric should be stored on CPU, but if for example data and label are on GPU and input to the metric, we can calculate the sufficient statistics on GPU

leezu · 2020-04-24T23:43:34Z

python/mxnet/gluon/metric.py

+        label = label.as_np_ndarray().astype('int32')
+        if self.class_type == "binary":
+            self._set(1)
+            if len(numpy.unique(label)) > 2:


This will trigger synchronization (as we need to wait for the result of the np.unique operator). Could we make error checking that triggers synchronization optional?

https://github.com/apache/incubator-mxnet/blob/83b51703ed354f41024423f140de38df2ba22d50/src/imperative/imperative.cc#L123-L127

acphile · 2020-04-28T00:47:23Z

@mxnet-bot run ci [centos-cpu, sanity, centos-gpu, windows-cpu, unix-gpu, windows-gpu, unix-cpu]

mxnet-bot · 2020-04-28T00:47:37Z

Jenkins CI successfully triggered : [centos-gpu, centos-cpu, windows-gpu, unix-cpu, sanity, unix-gpu, windows-cpu]

leezu · 2020-04-30T07:32:21Z

Let's disable it here, because it blocks this PR

leezu · 2020-04-30T07:33:19Z

You can go ahead and try reproduce the issue locally:

The reproducer is available since more than a month at #17886 (comment)

marcoabreu · 2020-04-30T07:34:53Z

No, it's unrelated and should be a separated and isolated PR. Each PR should serve one propose. That way, we can focus discussions, have single purpose commits and also allow reverting

leezu · 2020-04-30T07:40:44Z

Let's disable it in #18204

acphile · 2020-05-01T06:44:34Z

@mxnet-bot run ci [unix-gpu]

mxnet-bot · 2020-05-01T06:44:42Z

Jenkins CI successfully triggered : [unix-gpu]

acphile · 2020-05-07T05:49:35Z

@mxnet-bot run ci [unix-cpu, windows-gpu]

mxnet-bot · 2020-05-07T05:49:41Z

Jenkins CI successfully triggered : [unix-cpu, windows-gpu]

leezu · 2020-05-07T18:41:19Z

tests/python/tensorrt/test_cvnets.py

@@ -29,7 +28,12 @@
 def get_classif_model(model_name, use_tensorrt, ctx=mx.gpu(0), batch_size=128):
    mx.contrib.tensorrt.set_use_fp16(False)
    h, w = 32, 32
-    net = gluoncv.model_zoo.get_model(model_name, pretrained=True)
+    model_url = "https://raw.githubusercontent.com/dmlc/web-data/master/gluoncv/models/"


Please don't hardcode master in the URL here. The repository may change and will then break the CI. Instead, use the commit ID: https://raw.githubusercontent.com/dmlc/web-data/221ce5b7c6d5b0777a1e3471f7f03ff98da90a0a/gluoncv/models

acphile · 2020-05-08T07:53:28Z

@mxnet-bot run ci [windows-gpu]

mxnet-bot · 2020-05-08T07:53:35Z

Jenkins CI successfully triggered : [windows-gpu]

tests/python/train/test_mlp.py

tests/python/unittest/test_metric.py

acphile · 2020-05-09T15:57:30Z

@mxnet-bot run ci [unix-cpu]

mxnet-bot · 2020-05-09T15:57:35Z

Jenkins CI successfully triggered : [unix-cpu]

This reverts commit effbb8b.

mseth10 · 2020-05-14T08:54:31Z

@acphile this PR fails nightly CD while running nightly python unit tests. The following tests fail:
test_mcc, test_multilabel_f1, test_binary_f1
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/restricted-mxnet-cd%2Fmxnet-cd-release-job/detail/mxnet-cd-release-job/1119/pipeline/329

We'll need to revert this PR and fix the failures before re-merging it. Here's the link to revert PR: #18318

This reverts commit effbb8b.

leezu · 2020-05-14T16:28:39Z

The reason it fails is that the CI check in this PR was run too long ago. I should have restarted it before merging the PR. Meantime master changed and caused some additional changes to be necessary. They are in https://github.com/apache/incubator-mxnet/pull/18312/files

* finish 5 changes * move metric.py to gluon, replace mx.metric with mx.gluon.metric in python/mxnet/ * fix importError * replace mx.metric with mx.gluon.metric in tests/python * remove global support * remove macro support * rewrite BinaryAccuracy * extend F1 to multiclass/multilabel * add tests for new F1, remove global tests * use mxnet.numpy instead of numpy * fix sanity * rewrite ce and ppl, improve some details * use mxnet.numpy.float64 * remove sklearn * remove reset_local() and get_global in other files * fix test_mlp * replace mx.metric with mx.gluon.metric in example * fix context difference * Disable -DUSE_TVM_OP on GPU builds * Fix disable tvm op for gpu runs * use label.ctx in metric.py; remove gluoncv dependency in test_cvnets * fix sanity * fix importError * remove nose Co-authored-by: Ubuntu <ubuntu@ip-172-31-12-243.us-east-2.compute.internal> Co-authored-by: Leonard Lausen <lausen@amazon.com>

This reverts commit effbb8b.

Ubuntu and others added 4 commits April 15, 2020 15:02

finish 5 changes

f07d35e

move metric.py to gluon, replace mx.metric with mx.gluon.metric in py…

575f23b

…thon/mxnet/

fix importError

8992995

replace mx.metric with mx.gluon.metric in tests/python

1b8f521

acphile requested a review from szha as a code owner April 16, 2020 09:56

sxjscience reviewed Apr 16, 2020

View reviewed changes

leezu reviewed Apr 16, 2020

View reviewed changes

acphile added 5 commits April 20, 2020 04:10

remove global support

2ff2e38

remove macro support

c06f363

rewrite BinaryAccuracy

6beba21

extend F1 to multiclass/multilabel

b1fc42b

add tests for new F1, remove global tests

4b091b0

leezu reviewed Apr 22, 2020

View reviewed changes

use mxnet.numpy instead of numpy

1dfe0e0

leezu mentioned this pull request Apr 23, 2020

Self-attentive Sentence Embedding Tutorial Undeclared Dependency (sklearn) dmlc/gluon-nlp#886

Closed

Merge remote-tracking branch 'upstream/master'

083e85b

leezu reviewed Apr 24, 2020

View reviewed changes

acphile added 3 commits April 25, 2020 03:24

fix sanity

59d98b3

rewrite ce and ppl, improve some details

40e87e3

use mxnet.numpy.float64

5e153e1

acphile added 2 commits April 28, 2020 14:32

remove sklearn

bf68c6d

remove reset_local() and get_global in other files

56b846e

acphile requested a review from aaronmarkham as a code owner April 29, 2020 03:57

acphile added 2 commits April 29, 2020 06:10

fix test_mlp

8a437e9

replace mx.metric with mx.gluon.metric in example

b7c2b3b

leezu mentioned this pull request Apr 30, 2020

Disable -DUSE_TVM_OP on GPU builds #18204

Merged

resolve conflicts

2a80a0a

acphile added 2 commits May 7, 2020 10:37

use label.ctx in metric.py; remove gluoncv dependency in test_cvnets

8163fbb

fix sanity

d53e6ef

leezu reviewed May 7, 2020

View reviewed changes

leezu and others added 2 commits May 7, 2020 11:53

Merge branch 'master' into master

3adfa5e

fix importError

a2b0ffe

leezu reviewed May 9, 2020

View reviewed changes

tests/python/train/test_mlp.py Show resolved Hide resolved

tests/python/unittest/test_metric.py Outdated Show resolved Hide resolved

remove nose

ef3058a

leezu merged commit effbb8b into apache:master May 14, 2020

leezu added a commit that referenced this pull request May 14, 2020

Revert "Changes to mxnet.metric (#18083)"

c54a903

This reverts commit effbb8b.

mseth10 added a commit to mseth10/incubator-mxnet that referenced this pull request May 14, 2020

Revert "Changes to mxnet.metric (apache#18083)"

326410e

This reverts commit effbb8b.

leezu mentioned this pull request May 27, 2020

Proposal to mxnet.metric #18046

Closed

szha mentioned this pull request Aug 15, 2020

[Development] MXNet 2.0 Update #18931

Open

chinakook added a commit to chinakook/mxnet that referenced this pull request Nov 23, 2020

Revert "Changes to mxnet.metric (apache#18083)"

5f5df74

This reverts commit effbb8b.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes to mxnet.metric #18083

Changes to mxnet.metric #18083

acphile commented Apr 16, 2020

mxnet-bot commented Apr 16, 2020

sxjscience Apr 16, 2020

acphile Apr 17, 2020

sxjscience Apr 17, 2020

sxjscience Apr 16, 2020

leezu left a comment •

edited

Loading

leezu Apr 22, 2020

acphile Apr 22, 2020

leezu Apr 22, 2020

acphile Apr 22, 2020

leezu Apr 22, 2020

leezu Apr 24, 2020

leezu Apr 24, 2020

acphile commented Apr 28, 2020

mxnet-bot commented Apr 28, 2020

leezu commented Apr 30, 2020

leezu commented Apr 30, 2020

marcoabreu commented Apr 30, 2020

leezu commented Apr 30, 2020

acphile commented May 1, 2020

mxnet-bot commented May 1, 2020

acphile commented May 7, 2020

mxnet-bot commented May 7, 2020

leezu May 7, 2020 •

edited

Loading

acphile commented May 8, 2020

mxnet-bot commented May 8, 2020

acphile commented May 9, 2020

mxnet-bot commented May 9, 2020

mseth10 commented May 14, 2020 •

edited

Loading

leezu commented May 14, 2020


		mae = numpy.abs(label - pred).mean()

		if self.average == "macro":

Changes to mxnet.metric #18083

Changes to mxnet.metric #18083

Conversation

acphile commented Apr 16, 2020

Description

Checklist

Essentials

Changes

Comments

mxnet-bot commented Apr 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leezu left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acphile commented Apr 28, 2020

mxnet-bot commented Apr 28, 2020

leezu commented Apr 30, 2020

leezu commented Apr 30, 2020

marcoabreu commented Apr 30, 2020

leezu commented Apr 30, 2020

acphile commented May 1, 2020

mxnet-bot commented May 1, 2020

acphile commented May 7, 2020

mxnet-bot commented May 7, 2020

leezu May 7, 2020 • edited Loading

Choose a reason for hiding this comment

acphile commented May 8, 2020

mxnet-bot commented May 8, 2020

acphile commented May 9, 2020

mxnet-bot commented May 9, 2020

mseth10 commented May 14, 2020 • edited Loading

leezu commented May 14, 2020

leezu left a comment •

edited

Loading

leezu May 7, 2020 •

edited

Loading

mseth10 commented May 14, 2020 •

edited

Loading