Distributed Automatic SParsity with Fleet #33558

mingxu1067 · 2021-06-15T05:19:41Z

PR types

New features

PR changes

APIs

Describe

Implemented ASP MetaOptimizer in Fleet to enable distributed ASP training.
ASP MetaOptimizer could be the top of other MetaOptimizers.
Added a new attribute, asp, into DistributedStrategy .

Code example:

from paddle.fluid.contrib import sparsity

fleet.init(is_collective=True)
with fluid.program_guard(main_program, startup_program):
    input_data, loss = network()

    optimizer = fluid.optimizer.SGD(learning_rate=0.1)
	strategy = fleet.DistributedStragety()
    strategy.amp = True
    strategy.asp = True
    optimizer = fleet.distributed_optimizer(optimizer, strategy=strategy)
    optimizer.minimize(optimizer, loss, main_program, startup_program)

exe = fluid.Executor(place)
exe.run(startup_program)

# Must call `exe.run(startup_program)` first before calling `sparsity.prune_model`
sparsity.prune_model(place, main_program)

# Start training with ASP workflow.
for epoch in range(total_epochs):
    exe.run(main_program)

1. Added sparse mask generating functions, including 1D and 2Ds. 2. Added sparse pattern checking functions, including 1D and 2Ds.

1. Added model pruning functions. 2. Added optimizer decorator and sparse minimization functions.

… automatic_sparsity_utils

1. Added model pruning functions. 2. Added optimizer decorator and sparse minimization functions.

1. Added ASPOptimizer into Fleet metaoptimizer. 2. Added asp attribute into Fleet DistributedStrategy. 3. Added related test cases.

… automatic_sparsity_with_fleet

paddle-bot-old · 2021-06-15T05:19:45Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

… automatic_sparsity_with_fleet

qingqing01 · 2021-06-17T06:37:59Z

@mingxu1067 Do you have plan to support dygraph and inference?

mingxu1067 · 2021-06-17T07:04:47Z

@mingxu1067 Do you have plan to support dygraph and inference?

We do not have determined plan on DyGraph supporting so far.

Xreki · 2021-06-18T09:00:52Z

python/paddle/fluid/contrib/sparsity/asp.py

@@ -134,21 +130,36 @@ def prune_model(place,

            place = fluid.CUDAPlace(0)

+
+


这里空行加的有点多了。

Ok, 分散式那邊有沒有甚麼comment? 我一起修改後再commit一次

Xreki · 2021-06-18T09:22:30Z

python/paddle/fluid/contrib/sparsity/asp.py

+                optimizer = fluid.contrib.mixed_precision.decorator.decorate(optimizer )
+                # Calling sparsity.decorate() to wrap minimize() in optimizer, which 
+                # will insert necessary masking operations for ASP workflow.
+                optimizer = sparsity.decorate(optimizer )


现在是amp和asp一起用有问题？那这个示例里面先不要加amp吧？

ASP與AMP一起使用沒有問題，有成功使用PaddleNLP/bert的AMP + ASP做過sparse finetuning

paddle-bot-old · 2021-06-24T02:35:59Z

Sorry to inform you that b6f90d4's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

sandyhouse · 2021-07-05T09:20:34Z

python/paddle/distributed/fleet/meta_optimizers/asp_optimizer.py

+
+        return False
+
+    def _disable_strategy(self, dist_strategy):


set dist_strategy.asp = False ？

sandyhouse · 2021-07-05T09:21:01Z

python/paddle/distributed/fleet/meta_optimizers/asp_optimizer.py

+        return
+
+    def _enable_strategy(self, dist_strategy, context):
+        return


set dist_strategy.asp = True ？

1. Added enable/disable policy controlling in asp_optimizer. 2. Changed unittests.

sandyhouse

LGTM for Fleet strategy

mingxu1067 added 7 commits May 19, 2021 09:48

Automatic SParsity Utilities

ea668b2

1. Added sparse mask generating functions, including 1D and 2Ds. 2. Added sparse pattern checking functions, including 1D and 2Ds.

Automatic SParsity Helper

3725920

1. Added model pruning functions. 2. Added optimizer decorator and sparse minimization functions.

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

24ea128

… automatic_sparsity_utils

Automatic SParsity Helper

c0fc73b

1. Added model pruning functions. 2. Added optimizer decorator and sparse minimization functions.

ASP Fleet MetaOptimizer

5e4ab93

1. Added ASPOptimizer into Fleet metaoptimizer. 2. Added asp attribute into Fleet DistributedStrategy. 3. Added related test cases.

Merged from PaddlePaddle/Paddle develop

67e8a52

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

fb7c5d2

… automatic_sparsity_with_fleet

mingxu1067 added 3 commits June 16, 2021 10:25

Removed test_fleet_with_asp from nomal tests

30e3718

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

acd4ba1

… automatic_sparsity_with_fleet

Split test_fleet_with_asp to two tests for fitting time limition

b6f90d4

mingxu1067 mentioned this pull request Jun 17, 2021

Added an example to apply ASP on existed BERT training scripts PaddlePaddle/PaddleNLP#579

Merged

Xreki reviewed Jun 18, 2021

View reviewed changes

mingxu1067 added 13 commits June 25, 2021 15:58

Merged Paddle/develop 58e465a

c3693ac

Added enable_static() to example code.

1966f82

Slightly modified example code.

0ef2f7a

Set longer timeout for fleet_with_asp unit tests

dad3d13

Removed set timeout of fleet_with_asp tests due to approval required.

7990a46

Slightly changed example code in ASP.

5fec8a9

Added cuda_compiled checking in asp example code.

46053d4

Fix code example in asp.py and reduce net in asp unittest

6bc3628

Fix code example in asp.py

f097699

Fix fleet_with_asp test failed

9bb7e38

Fix fleet_with_asp test failed

efe8885

Set GPU devices for fleet with asp tests

fffab5f

Setup ip and port for fleet with asp tests

ff66b0b

sandyhouse reviewed Jul 5, 2021

View reviewed changes

mingxu1067 added 2 commits July 6, 2021 14:17

Modified ASP meta_optimizer and related tests

c642a66

1. Added enable/disable policy controlling in asp_optimizer. 2. Changed unittests.

Added ENDPOINT in unittests

0a05dda

sandyhouse approved these changes Jul 8, 2021

View reviewed changes

Xreki approved these changes Jul 8, 2021

View reviewed changes

Xreki merged commit 86cb3fb into PaddlePaddle:develop Jul 8, 2021

mingxu1067 deleted the automatic_sparsity_with_fleet branch October 15, 2021 06:10

mingxu1067 added the NVIDIA label Jan 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distributed Automatic SParsity with Fleet #33558

Distributed Automatic SParsity with Fleet #33558

mingxu1067 commented Jun 15, 2021

paddle-bot-old bot commented Jun 15, 2021

qingqing01 commented Jun 17, 2021

mingxu1067 commented Jun 17, 2021

Xreki Jun 18, 2021

mingxu1067 Jun 18, 2021

Xreki Jun 18, 2021

mingxu1067 Jun 18, 2021

paddle-bot-old bot commented Jun 24, 2021

sandyhouse Jul 5, 2021

sandyhouse Jul 5, 2021

sandyhouse left a comment

		@@ -134,21 +130,36 @@ def prune_model(place,

		place = fluid.CUDAPlace(0)

Distributed Automatic SParsity with Fleet #33558

Distributed Automatic SParsity with Fleet #33558

Conversation

mingxu1067 commented Jun 15, 2021

PR types

PR changes

Describe

paddle-bot-old bot commented Jun 15, 2021

qingqing01 commented Jun 17, 2021

mingxu1067 commented Jun 17, 2021

Xreki Jun 18, 2021

Choose a reason for hiding this comment

mingxu1067 Jun 18, 2021

Choose a reason for hiding this comment

Xreki Jun 18, 2021

Choose a reason for hiding this comment

mingxu1067 Jun 18, 2021

Choose a reason for hiding this comment

paddle-bot-old bot commented Jun 24, 2021

sandyhouse Jul 5, 2021

Choose a reason for hiding this comment

sandyhouse Jul 5, 2021

Choose a reason for hiding this comment

sandyhouse left a comment

Choose a reason for hiding this comment