Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix restore sequence parallel #7273

Merged
merged 4 commits into from
Aug 21, 2023
Merged

Fix restore sequence parallel #7273

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fbfa592
Fix restore
hsiehjackson Aug 18, 2023
56deb99
Merge branch 'main' into fix-restore-seq-parallel
hsiehjackson Aug 18, 2023
b51d6f6
reset and restore transformer config sequence parallel
blahBlahhhJ Aug 18, 2023
d2f3742
modify model parallel config as well
blahBlahhhJ Aug 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1348,6 +1348,8 @@ def _reset_sequence_parallelism_args(self):

# Reset config values. Needed for calling generate.
self.cfg.sequence_parallel = False
self.model_parallel_config.sequence_parallel = False
self.transformer_config.sequence_parallel = False

# Reset model parameters.
for module in self.get_gpt_module_list():
Expand All @@ -1362,6 +1364,8 @@ def _restore_sequence_parallelism_args(self):
"""
# Restore config values.
self.cfg.sequence_parallel = self.last_sequence_parallel
self.model_parallel_config.sequence_parallel = self.last_sequence_parallel
self.transformer_config.sequence_parallel = self.last_sequence_parallel

# Restore model parameters.
for module in self.get_gpt_module_list():
Expand Down
1 change: 1 addition & 0 deletions nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -583,6 +583,7 @@ def inference_epoch_end(self, outputs, mode, data_cfg):
# Merge the functionality of previous on_inference_epoch_end() within inference_epoch_end() func here
app_state = AppState()
self._restore_activation_checkpointing_args()
self._restore_sequence_parallelism_args()
if hasattr(self, "_train_ds"):
_reconfigure_microbatch_calculator(
rank=app_state.global_rank,
Expand Down
Loading