[Distributed] Use custom overlapping method for backward chunks #74891
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Category
Auto Parallel
PR Types
Improvements
Description
在 DualPipeV 中,当执行 backward_pass 时,不直接调用 backwad_chunk.backward,而是调用模型的 overlapped_forward_backward 方法,但只传入 backward_chunk,不传入 forward_chunk:
这样允许模型自主编排 backward_chunk,实现 backward 之间的自 Overlap
该 PR 为 兼容性升级,不会破坏现有模型的兼容性,模型若未实现 backward 的自 Overlap 机制,则会跑原来不 overlap 的 backward,性能不变
DSV3 实现自 Overlap 机制:PaddlePaddle/PaddleNLP#11003
Pcard-85711