Unify the metrics implementation between low-level and high-level API. #26158

qingqing01 · 2020-08-11T10:49:45Z

PR types

Function optimization

PR changes

APIs

Describe

Unify the metrics implementation between low-level and high-level API.
Change function name add_metric_op in Metric to compute.
Return scalar, not [scalar] in update() and accumulate() of Accuracy.
Refine and add comments for sub-function in Accuracy.
- Add comments for compute, update, reset..
- Add standalone usage and usage with Model API for Metrics
Add unit testing in python/paddle/tests

test=develop

paddle-bot-old · 2020-08-11T10:49:51Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

paddle-bot-old · 2020-08-11T10:49:53Z

✅ This PR's description meets the template requirements!
Please wait for other CI results.

… hapi_metrics

heavengate · 2020-08-17T09:37:31Z

python/paddle/metric/metrics.py

+                       |                      ||
+                outputs & labels              ||
+                       |                      ||    tensor data
+             {Metric.compute}                 ||


可以调一下缩进对齐一下~

test=develop

saxon-zh · 2020-08-17T11:14:38Z

python/paddle/incubate/hapi/model.py

        """
        Configures the model before runing.

        Args:
            optimizer (Optimizer|None): Optimizer must be set in training
                and should be a Optimizer instance. It can be None in eval
                and test mode.
-            loss_function (Loss|callable function|None): Loss function can
+            loss (Loss|callable function|None): Loss function can
                be a `fluid.dygraph.Layer` instance or any callable function


fluid.dygraph.Layer => paddle.nn.Layer ，下面用到fluid.dygraph.Layer的地方同样都需要修改一下

saxon-zh · 2020-08-17T11:15:48Z

python/paddle/incubate/hapi/model.py

@@ -1362,15 +1362,15 @@ def evaluate(
            input = hapi.Input('image', [-1, 1, 28, 28], 'float32')
            label = hapi.Input('label', [None, 1], 'int64')
            model = hapi.Model(hapi.vision.LeNet(), input, label)
-            model.prepare(metrics=hapi.metrics.Accuracy())
+            model.prepare(metrics=paddle.metric.Accuracy())

            result = model.evaluate(val_dataset, batch_size=64)
            print(result)

            # imperative mode
            fluid.enable_dygraph()


fluid.enable_dygraph => paddle.disable_static

guoshengCS · 2020-08-17T11:10:09Z

python/paddle/metric/metrics.py

+
+    Args:
+        name (str, optional): String name of the metric instance.
+            Default is `precision`.


这里name默认值是否应是recall呢

guoshengCS · 2020-08-17T11:14:52Z

python/paddle/metric/metrics.py

+
+    Args:
+        name (str, optional): String name of the metric instance. Default
+            is `acc`.


这里name默认值是否应是 auc 呢

guoshengCS · 2020-08-17T11:16:32Z

python/paddle/metric/metrics.py

+        name (str, optional): String name of the metric instance. Default
+            is `acc`.
+        curve (str): Specifies the mode of the curve to be computed,
+            'ROC' or 'PR' for the Precision-Recall-curve. Default is 'ROC'.


Maybe add doc for num_thresholds later

saxon-zh · 2020-08-17T11:17:15Z

python/paddle/metric/metrics.py

+        .. code-block:: python
+            def compute(pred, label):
+                # sort prediction and slice the top-5 scores
+                pred = fluid.layers.argsort(pred, descending=True)[1][:, :5]


argsort用paddle.下面的API，去掉fluid

saxon-zh · 2020-08-17T11:17:27Z

python/paddle/metric/metrics.py

+                pred = fluid.layers.argsort(pred, descending=True)[1][:, :5]
+                # calculate whether the predictions are correct
+                correct = pred == label
+                return fluid.layers.cast(correct, dtype='float32')


去掉fluid

saxon-zh · 2020-08-17T11:18:40Z

python/paddle/metric/metrics.py

+class Accuracy(Metric):
+    """
+    Encapsulates accuracy metric logic.
+    def __init__(self, topk=(1, ), name='acc', *args, **kwargs):


没用的信息吧？

Removed. Thanks!

saxon-zh · 2020-08-17T11:21:38Z

python/paddle/metric/metrics.py

+        import paddle
+
+        paddle.disable_static()
+        x = paddle.to_variable(np.array([


to_variable => to_tensor

saxon-zh · 2020-08-17T11:21:50Z

python/paddle/metric/metrics.py

+            [0.1, 0.4, 0.3, 0.2],
+            [0.1, 0.2, 0.4, 0.3],
+            [0.1, 0.2, 0.3, 0.4]]))
+        y = paddle.to_variable(np.array([[0], [1], [2], [3]]))


saxon-zh · 2020-08-17T11:22:04Z

python/paddle/metric/metrics.py

+        .. code-block:: python
+
+        import paddle
+        import paddle.fluid as fluid


去掉fluid

saxon-zh · 2020-08-17T11:23:17Z

python/paddle/metric/metrics.py

+        train_dataset = hapi.datasets.MNIST(mode='train')
+
+        model = hapi.Model(hapi.vision.LeNet(classifier_activation=None))
+        optim = fluid.optimizer.Adam(


paddle.optimizer.Adam

test=develop

… hapi_metrics

… hapi_metrics test=develop

… hapi_metrics

test=develop

jzhang533 · 2020-08-20T10:25:17Z

python/paddle/metric/__init__.py

-
-
-
-from ..fluid.metrics import Accuracy, Auc, ChunkEvaluator, CompositeMetric, DetectionMAP, EditDistance, \


paddle/fluid/metrics.py 里的API计划怎么处理？
还有ChunkEvaluator, CompositeMetric, DetectionMAP, EditDistance在重构后的paddle.metric下没有对应的API。

@XiaoguangHu01 晓光麻烦看看__init__.py文件的改动。

相关被使用统计在agroup中。 DetectionMAP、EditDistance建议先废弃，相关套件已在模型里重新实现。Precision/Recall当前没有被任何模型使用到，当前fluid下面的实现还太简单。会上讨论，本次PR先以统一位置为主，后续版本还需要增强相关功能，包括增加ChunkEvaluator,以及改进功能。

计划废弃的API都添加一下deprecated decorator吧。

jzhang533 · 2020-08-20T10:25:23Z

python/paddle/metric/__init__.py

-from ..fluid.metrics import Accuracy, Auc, ChunkEvaluator, CompositeMetric, DetectionMAP, EditDistance, \
-        Precision, Recall
+from .metrics import *
+from . import metrics

 from ..fluid.layers.metric_op import accuracy, auc
 from ..fluid.layers.nn import chunk_eval, cos_sim, mean_iou


paddle.metric下的class形式API，跟小写的调用op的API的关系，是否讨论过啊？

class形式: 提供batch累积功能。一般用于高层API。底层API也可以使用。参考内部wiki记录。

function(op形式): 保留了fluid下面的api，一般用于底层API。

Accuracy: 只支持当前batch的统计，需用户做batch累积计算。底层API可以使用。

auc: 可计算batch和累积batch的auc。 PaddleRec中使用。

jzhang533 · 2020-08-20T10:29:22Z

python/paddle/metric/metrics.py

+        model.prepare(
+            optim,
+            loss=nn.BCELoss(),
+            metrics=paddle.metric.Recall())


这里给一个可以同时计算precison和recall的例子？
通常情况下会同时算这两个指标的。

@jzhang533 Precision和Recall的功能会上讨论后续的版本还需增强，见上面回复，本次先以统一为主。所以后续可以增强下。

我看model.prepare的metrics是可以接受list的，这里改一下是否就可以？（先不考虑运行效率问题）

@jzhang533 Done. Thanks!

jzhang533 · 2020-08-20T10:31:00Z

tools/wlist.json

+        "Auc.update",
+        "Auc.accumulate",
+        "Auc.name",
+        "Auc.compute",


同意这次对于白名单的改动。因为在class Auc等的地方已经有了示例代码。

test=develop

XiaoguangHu01

LTGM

jzhang533

lgtm
will have follow up pr to deprecate fluid apis.

saxon-zh

LGTM

… hapi_metrics

test=develop

qingqing01 · 2020-08-24T03:32:29Z

Fix sample code about Adam, since #26288 is merged.

@XiaoguangHu01 @raindrops2sea @jzhang533 @saxon-zh Please help to review again.

saxon-zh

LGTM

XiaoguangHu01

LGTM

jzhang533

lgtm

qingqing01 added 2 commits August 10, 2020 10:14

Refine paddle/metrics

16974da

Move paddle/incubate/hapi/metrics to paddle/metric

43d8b64

test=develop

qingqing01 added 3 commits August 15, 2020 10:19

Add Precision, Recall and Auc metric

ad32568

Update unit testing

20b4936

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

1d7dce5

… hapi_metrics

qingqing01 requested review from XiaoguangHu01, LielinJiang, guoshengCS, heavengate and saxon-zh August 17, 2020 07:55

qingqing01 force-pushed the hapi_metrics branch from 12bab82 to 3659013 Compare August 17, 2020 09:31

LielinJiang previously approved these changes Aug 17, 2020

View reviewed changes

heavengate reviewed Aug 17, 2020

View reviewed changes

Update code to latest develop

6b9e9d7

test=develop

qingqing01 dismissed LielinJiang’s stale review via 6b9e9d7 August 17, 2020 09:45

qingqing01 force-pushed the hapi_metrics branch from 3659013 to 6b9e9d7 Compare August 17, 2020 09:45

Follow comments

a6235a4

test=develop

heavengate previously approved these changes Aug 17, 2020

View reviewed changes

saxon-zh reviewed Aug 17, 2020

View reviewed changes

guoshengCS previously approved these changes Aug 17, 2020

View reviewed changes

saxon-zh reviewed Aug 17, 2020

View reviewed changes

qingqing01 dismissed stale reviews from guoshengCS and heavengate via 362d914 August 18, 2020 04:22

qingqing01 added 10 commits August 18, 2020 04:27

--amend

562bd14

Fix unit testing after updating API

e2dd416

test=develop

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

c30a961

… hapi_metrics

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

4faabe3

… hapi_metrics test=develop

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

6d66305

… hapi_metrics

Update test_metrics

180a95f

Enhance unit testing

08308fd

test=develop

Fix code style

52b9f20

test=develop

Fix sample code

24c3849

test=develop

Merge branch 'develop' into hapi_metrics

15420d8

jzhang533 reviewed Aug 20, 2020

View reviewed changes

Update code sample in metric.Recall

553cb01

test=develop

XiaoguangHu01 previously approved these changes Aug 21, 2020

View reviewed changes

jzhang533 previously approved these changes Aug 22, 2020

View reviewed changes

raindrops2sea previously approved these changes Aug 23, 2020

View reviewed changes

saxon-zh previously approved these changes Aug 23, 2020

View reviewed changes

qingqing01 added 2 commits August 24, 2020 03:18

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

d5c31c0

… hapi_metrics

Fix code sample about Adam after develop code is updated

0fb112c

test=develop

qingqing01 dismissed stale reviews from saxon-zh, raindrops2sea, jzhang533, and XiaoguangHu01 via 0fb112c August 24, 2020 03:29

saxon-zh approved these changes Aug 24, 2020

View reviewed changes

raindrops2sea approved these changes Aug 24, 2020

View reviewed changes

XiaoguangHu01 approved these changes Aug 24, 2020

View reviewed changes

jzhang533 approved these changes Aug 24, 2020

View reviewed changes

qingqing01 merged commit 78ca8cf into PaddlePaddle:develop Aug 24, 2020




		from ..fluid.metrics import Accuracy, Auc, ChunkEvaluator, CompositeMetric, DetectionMAP, EditDistance, \

Unify the metrics implementation between low-level and high-level API. #26158

Unify the metrics implementation between low-level and high-level API. #26158

Conversation

qingqing01 commented Aug 11, 2020 • edited Loading

PR types

PR changes

Describe

paddle-bot-old bot commented Aug 11, 2020

paddle-bot-old bot commented Aug 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

jzhang533 left a comment

Choose a reason for hiding this comment

saxon-zh left a comment

Choose a reason for hiding this comment

qingqing01 commented Aug 24, 2020

saxon-zh left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

jzhang533 left a comment

Choose a reason for hiding this comment

qingqing01 commented Aug 11, 2020 •

edited

Loading

paddle-bot-old bot commented Aug 11, 2020 •

edited

Loading

qingqing01 Aug 20, 2020 •

edited

Loading