BMTrain New Version Release v1.0.0 #182
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
remove inappropriate import in __init__.py
* using hooks to implement ZeRO and Checkpoint --------- Co-authored-by: zhangkaihuo <zhangkaihuo@modelbest.cn>
FIX Error: tensor slice in gather()
only initialized tp_comm when tp_size > 1
This do not affect current BMTrain since BMTrain pass grain_size=0 in all parallel_for.
fix loss scale for tp
* fix adam bf16 load changed to fp16 * support save optim_manager.state (including optimizer, lr_scheduler, loss_scale)
FIX: allgather_object stuck
zkh2016
approved these changes
Feb 20, 2024
Optimizer load gathered state and record delta feature are supported now
Vocab parallel Embedding impl and make example work when tp_size > 1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
BMTrain New Version Release v1.0.0
Issue Reference
Issue #174
Description
This new version of BMTrain introduces Tensor Parallelism and a significant restructuring of the codebase. We have enhanced flexibility by allowing for more granular control over ZeRO levels and Activation Checkpointing. This control is now achievable at the
bmt.Block(alias of bmt.CheckpointBlock) level, enabling targeted optimizations. For instance, it is now possible to apply aggressive ZeRO settings or disable Checkpointing in specific layers as needed. Additionally, the updated BMTrain features a suite of operators designed explicitly for Tensor Parallel training.Type of Change
How Has This Been Tested?
(Describe the tests used to validate the changes. Please provide instructions for reproduction.)
Checklist
Additional Information
(Provide any additional information, configuration details, or data necessary for the review.)