remove the AdamOptimizer、SGDOptimizer、MomentumOptimizer、ModelAverage、LookaheadOptimizer、FtrlOptimizer、DecayedAdagradOptimizer、DpsgdOptimizer in fluid and relocate the ExponentialMovingAverage、PipelineOptimizer、GradientMergeOptimizer and change optimizer base for LarsMomentumOptimizer and RecomputeOptimizer #55970

longranger2 · 2023-08-03T14:36:54Z

PR types

Others

PR changes

APIs

Description

进行移除的优化器如下：

remove the ModelAverage in paddle/fluid/optimizer.py and use python/paddle/incubate/optimizer/modelaverage.py to replace it.
remove the LookaheadOptimizer in paddle/fluid/optimizer.py and use the LookAhead in paddle/incubate/optimizer/lookahead.py to replace it.
remove the AdamOptimizer in paddle/fluid/contrib/optimizer.py and use python/paddle/optimizer/adam.py to replace it.
remove the SGDOptimizer in paddle/fluid/contrib/optimizer.py and use python/paddle/optimizer/sgd.py to replace it.
remove the MomentumOptimizer in paddle/fluid/contrib/optimizer.py and use python/paddle/optimizer/momentum.py to replace it.
remove the FtrlOptimizer in paddle/fluid/optimizer.py.
remove the DecayedAdagradOptimizer in paddle/fluid/optimizer.py.
remove the DpsgdOptimizer in paddle/fluid/optimizer.py.

进行迁移的优化器如下：

relocate the ExponentialMovingAverage in paddle/fluid/optimizer.py
relocate the PipelineOptimizer in paddle/fluid/optimizer.py
relocate the GradientMergeOptimizer in paddle/fluid/optimizer.py

进行迁移并更新基类的优化器如下：

relocate the LarsMomentumOptimizer in paddle/fluid/optimizer.py and change optimizer base
relocate the RecomputeOptimizer in paddle/fluid/optimizer.py and change optimizer base

… momentum

… SGDOptimizer

…m_sgd_momentum

paddle-bot · 2023-08-03T14:37:01Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

…into LarsMomentum

…into adam_sgd_momentum

XieYunshen

LGTM
单测删除

zoooo0820 · 2023-08-07T08:53:29Z

python/paddle/optimizer/adam.py

+            found_inf = self._get_auxiliary_var('found_inf')
+
+            if found_inf:
+                inputs['SkipUpdate'] = found_inf


此处是被单测 test_mixed_precision检出. 目前只有 fluid.optimizer.Adam and paddle.optimizer.AdamW添加了检测inf跳过更新的策略（见paddle.static.amp.decorator)。但Adam OP是支持这个输入的。因此在paddle.optimizer.Adam添加该策略支持。

zoooo0820 · 2023-08-07T08:58:42Z

test/collective/fleet/test_fleet_sharding_meta_optimizer.py

@@ -609,12 +609,6 @@ def test_sharding_weight_decay(self):
                'c_reduce_sum',
                'c_reduce_sum',
                'c_sync_comm_stream',
-                'scale',


此处是因为paddle.optimizer.Momemtum会将L2Decay fuse到OP中，因此不存在额外的scale+sum操作

zoooo0820 · 2023-08-07T09:18:10Z

test/legacy_test/test_imperative_optimizer.py

-        return optimizer
-
-    def get_optimizer(self):
-        optimizer = paddle.optimizer.Lamb(


这里删除部分2.0优化器paddle.optimizer.xxx的单测，是因为在单测v2版本中存在吗，这两个单测文件间的关系是？

test_imperative_optimizer.py是用来测试旧版的优化器的，test_imperative_optimizer_v2.py主要用来测试2.0版本的优化器的，但在一开始改动test_imperative_optimizer.py文件的时候只是将优化器替换为2.0的实现，并没有将其删除，所以导致现在两个文件越来相似，后面考虑将test_imperative_optimizer.py进行删除

zoooo0820 · 2023-08-07T09:30:44Z

test/prim/model/test_resnet_cinn.py

+    9.8336181640625,
+    8.22379207611084,
+    8.195695877075195,
+    10.508796691894531,


2.0版本的paddle.optimizer.Momentum会自动fuse L2Decay，此时和fluid存在数值差异。其他Decay或No Decay结果一致，本单测属于前一种场景。

zoooo0820 · 2023-08-07T09:31:04Z

test/prim/model/test_resnet_prim.py

+    9.569124221801758,
+    8.251557350158691,
+    8.513609886169434,
+    10.603094100952148,


zoooo0820 · 2023-08-07T09:31:14Z

test/prim/model/test_resnet_prim_cinn.py

+    9.559739112854004,
+    8.430597305297852,
+    8.109201431274414,
+    10.224763870239258,


zoooo0820 · 2023-08-07T10:19:21Z

test/legacy_test/test_trainable.py

@@ -68,7 +68,7 @@ def test_trainable(self):
        self.check_trainable(
            test_trainable,
            feed_dict,
-            op_count={'adam': 1, 'scale': 0, 'mul_grad': 0},
+            op_count={'adam': 1, 'scale': 0, 'mul_grad': 1},


该处理论上不应该改动，且下面一个关于adamax的检查已经被错误的修改。经查，属于静态图下Parameter未能正确设置stop_gradient属性导致(fluid.Optimizer检查的是trainable属性而非stop_gradient，paddle.Optimizer则相反）。

后续需要另提PR修复这个问题

zoooo0820

LGTM， trainable问题专门解决

zhangting2020

LGTM for amp

6clc

LGTM

jeff41404

LGTM

sunzhongkai588

LGTM

ExponentialMovingAverage 相应的中文文档是否需要更改位置？需要的话中文这边也提个PR叭 @longranger2
文档预览的ci现在出了点bug，暂时无法预览，之后预览有问题我再在这个comment下回复～

longranger2 · 2023-08-08T03:03:42Z

ExponentialMovingAverage 相应的中文文档是否需要更改位置？需要的话中文这边也提个PR叭 @longranger2

ExponentialMovingAverage 相应的中文文档应该是不需要更改位置，虽然对ExponentialMovingAverage进行迁移，但对外暴露的接口还是paddle.static.ExponentialMovingAverage，与文档是统一的～ @sunzhongkai588

XiaoguangHu01

LGTM

longranger2 added 30 commits April 17, 2023 00:02

change the optimizer base for SGDOptimizer

f28f6d4

change the optimizer base for SGDOptimizer

0a0aa08

Merge branch 'PaddlePaddle:develop' into SGDOptimizer

819cbbc

replace the SGDOptimizer with SGD

2cb38f2

fix bug of sgd

1b0203e

change the optimizer base for MomentumOptimizer

facca0a

fix the remaining tests

93c3e77

remove the Momentum in fluid/optimizer.py

fa088c8

Merge branch 'develop' into momentum

3d38988

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

70b012d

… momentum

fix bug

863e083

fix bug

6df5296

fix bug

1619183

fix bug

1dd7fa2

Merge branch 'PaddlePaddle:develop' into momentum

19937a0

Update test_resnet_cinn.py

f22a27a

Update test_resnet_prim_cinn.py

a501d76

fix conflict

74e7cc5

fix bug

f4e029f

fix bug

9dbf5f1

fix bug

2a92792

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

e494477

… SGDOptimizer

Merge branch 'PaddlePaddle:develop' into momentum

162ca01

remove the ModelAverage in fluid

1825663

remove the LookaheadOptimizer in fluid

6a4d5e6

fix bug

86a02ac

remove AdamOptimizer in fluid

48c5860

Update test_image_classification_fp16.py

44dd077

fix bug

fd47dff

relocate the ExponentialMovingAverage in fluid

6e2c2a8

longranger2 added 3 commits August 3, 2023 14:21

Merge branch 'Adam' of https://github.com/longranger2/Paddle into ada…

3973f85

…m_sgd_momentum

fix conflict

42556d8

fix conflict

8fc30a1

paddle-bot bot added contributor External developers status: proposed labels Aug 3, 2023

longranger2 changed the title ~~remove the adam、sgd、momentum in fluid~~ remove the AdamOptimizer、SGDOptimizer、MomentumOptimizer in fluid Aug 3, 2023

longranger2 added 3 commits August 3, 2023 15:33

change optimizer base for RecomputeOptimizer

25cb949

Merge branch 'LarsMomentum' of https://github.com/longranger2/Paddle …

9cc083a

…into LarsMomentum

fix conflict

a3dd808

longranger2 changed the title ~~remove the AdamOptimizer、SGDOptimizer、MomentumOptimizer in fluid~~ remove the AdamOptimizer、SGDOptimizer、MomentumOptimizer in fluid and change optimizer base for LarsMomentumOptimizer and RecomputeOptimizer Aug 4, 2023

longranger2 added 3 commits August 4, 2023 14:36

fix bug

82a9da3

fix bug

3648ec7

Merge branch 'ModelAverage' of https://github.com/longranger2/Paddle …

94a4c86

…into adam_sgd_momentum

Update test_imperative_optimizer_v2.py

2d9be2a

longranger2 mentioned this pull request Aug 5, 2023

test for #55970 #55997

Closed

Merge branch 'develop' into adam_sgd_momentum

fee3658

XieYunshen approved these changes Aug 7, 2023

View reviewed changes

zoooo0820 reviewed Aug 7, 2023

View reviewed changes

zoooo0820 approved these changes Aug 7, 2023

View reviewed changes

zhangting2020 approved these changes Aug 7, 2023

View reviewed changes

6clc approved these changes Aug 7, 2023

View reviewed changes

jeff41404 approved these changes Aug 7, 2023

View reviewed changes

tianshuo78520a approved these changes Aug 8, 2023

View reviewed changes

sunzhongkai588 approved these changes Aug 8, 2023

View reviewed changes

XiaoguangHu01 approved these changes Aug 9, 2023

View reviewed changes

jeff41404 merged commit 723c6f7 into PaddlePaddle:develop Aug 9, 2023

longranger2 deleted the adam_sgd_momentum branch August 9, 2023 03:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

longranger2 commented Aug 3, 2023 •

edited

Loading

paddle-bot bot commented Aug 3, 2023

XieYunshen left a comment

zoooo0820 Aug 7, 2023

zoooo0820 Aug 7, 2023

zoooo0820 Aug 7, 2023

longranger2 Aug 7, 2023

zoooo0820 Aug 7, 2023

zoooo0820 Aug 7, 2023

zoooo0820 Aug 7, 2023

zoooo0820 Aug 7, 2023

zoooo0820 left a comment

zhangting2020 left a comment

6clc left a comment

jeff41404 left a comment

sunzhongkai588 left a comment

longranger2 commented Aug 8, 2023 •

edited

Loading

XiaoguangHu01 left a comment

Conversation

longranger2 commented Aug 3, 2023 • edited Loading

PR types

PR changes

Description

paddle-bot bot commented Aug 3, 2023

XieYunshen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zoooo0820 left a comment

Choose a reason for hiding this comment

zhangting2020 left a comment

Choose a reason for hiding this comment

6clc left a comment

Choose a reason for hiding this comment

jeff41404 left a comment

Choose a reason for hiding this comment

sunzhongkai588 left a comment

Choose a reason for hiding this comment

longranger2 commented Aug 8, 2023 • edited Loading

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

longranger2 commented Aug 3, 2023 •

edited

Loading

longranger2 commented Aug 8, 2023 •

edited

Loading