Skip to content

Commit

Permalink
remove skipping of checkpoint when checkpoint exists
Browse files Browse the repository at this point in the history
Signed-off-by: ashors1 <ashors@nvidia.com>
  • Loading branch information
ashors1 committed Nov 21, 2024
1 parent 34f7408 commit df6cc8d
Showing 1 changed file with 0 additions and 3 deletions.
3 changes: 0 additions & 3 deletions nemo/lightning/io/pl.py
Original file line number Diff line number Diff line change
Expand Up @@ -155,9 +155,6 @@ def save_checkpoint(self, checkpoint: Dict[str, Any], path: _PATH, storage_optio
checkpoint_dir = ckpt_to_weights_subdir(path, is_saving=True)

fs = get_filesystem(checkpoint_dir)
if fs.isdir(checkpoint_dir) and dist_checkpointing.check_is_distributed_checkpoint(checkpoint_dir):
logging.info(f'Distributed checkpoint at path {checkpoint_dir} already exists, skipping saving')
return
fs.makedirs(checkpoint_dir, exist_ok=True)

validate_sharding_integrity = not (self.validated_consistency and self.assume_constant_structure)
Expand Down

0 comments on commit df6cc8d

Please sign in to comment.