[NPU] support global accumulator for adam #32780
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
phlrain
approved these changes
May 13, 2021
zhaoyinglia
pushed a commit
to zhaoyinglia/Paddle
that referenced
this pull request
Sep 2, 2021
* add use_global_beta_pow * add use_global_beta_pow * update npu kernel * update python api * refine code * add ut for use_global_beta_pow * fix npu kernel * add ut for api * add ut for exception * add ut for save/load
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
OPs
Describe
[NPU] support global accumulator for adam
As described in #32605, on ernie-3.0 model, we can see that there are several time bubbles between two adam kernels.
There are mainly two problems that cost much time,
update beta1/beta2 pow
convert beta1/beta2/epsilon to NPU tensor
PR #32605 solves one problem 2, and this PR tries to solve problem 1.
why
The original implementation of AdamOptimizer creates beta1_pow and beta2_pow for each parameter and updates them in adam op for each parameter.
It SHOULD be pointed out that, actually, the value of the beta1_pow and beta2_pow of each parameter is the same.
This works fine in adam CUDA kernel, since the beta1_pow and beta2_pow can be updated fast. However, in NPU kernel, it requires to call two
mulop and cost much time.How
So, we introduce
global beta_pow, which means only creates one beta1_pow and one beta2_pow for all the parameters of the whole model.Specificlly,
use_global_beta_powto adam op. If true, the outputs(Beta1PowOut, Beta2PowOut) will not be used in adam op, "and beta_pow will be updated after all adam op in the model.
use_global_beta_powtopaddle.fluid.optimizer.Adam, If true, Adam will use global beta_pow for whole model instead of creating beta_pow for each parameter.As can be seen in the timeline, there is no mul between two
ApplyAdamPerformance
before
after
22211 -> 24481 tokens/s, +10 %