[NeMo-UX] Make TE and Apex dependencies optional #9550

ashors1 · 2024-06-26T22:28:06Z

What does this PR do ?

This PR provides a fallback codepath that defaults to pure Pytorch/jit when TE and Apex are not installed. Note that this is specifically designed to work with NeMo 2.0, so there are no guarantees when running with NeMo 1.0 models.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

nemo/lightning/apex_utils.py

+# def check_adlr_autoresume_termination(iteration, model, optimizer, lr_scheduler, save: bool):
+#     """Check for autoresume signal and exit if it is received."""
+#     from apex.ppu.checkpointing import save_checkpoint
+#
+#     autoresume = get_adlr_autoresume()
+#     # Add barrier to ensure consistency.
+#     torch.distributed.barrier()
+#     if autoresume.termination_requested():
+#         if save:
+#             save_checkpoint(iteration, model, optimizer, lr_scheduler)
+#         print_rank_0(">>> autoresume termination request found!")
+#         if torch.distributed.get_rank() == 0:
+#             autoresume.request_resume()
+#         print_rank_0(">>> training terminated. Returning")
+#         sys.exit(0)


nemo/lightning/apex_utils.py

nemo/lightning/data.py

nemo/lightning/apex_utils.py

examples/llm/megatron_gpt_pretraining.py

dimapihtar · 2024-07-04T11:40:03Z

nemo/collections/nlp/models/language_modeling/megatron_retro_model.py

@@ -99,6 +99,9 @@
    TransformerConfig = ApexGuardDefaults

    HAVE_MEGATRON_CORE = False
+    from typing import Any
+
+    RetroConfig = Any


do we want to do the same for all of others mcore's classes we are trying to import under "try" block?
also, why don't use ApexGuardDefaults?

do we want to do the same for all of others mcore's classes we are trying to import under "try" block?

So this PR is intended to provide a pure pytorch/jit path for NeMo 2.0, so things are not guaranteed to work with NeMo 1.0 models (I have updated the description to reflect this). I added this change because previously, the import of the Retro model in https://github.com/NVIDIA/NeMo/blob/bdb4e89d9ac33d733f8ea7b21552628dda798825/nemo/collections/nlp/models/language_modeling/__init__.py was breaking the 2.0 codepath, but it looks like that's no longer the case anymore. If it makes things clearer, I go ahead and revert this change.

nemo/lightning/data.py

nemo/lightning/megatron_parallel.py

nemo/lightning/pytorch/plugins/data_sampler.py

nemo/collections/llm/gpt/model/base.py

examples/llm/megatron_gpt_pretraining.py

nemo/lightning/data.py

+            setup_microbatch_calculator,
+        )
+
+        HAVE_APEX = True


Signed-off-by: ashors1 <ashors@nvidia.com>

Signed-off-by: ashors1 <ashors1@users.noreply.github.com>

ericharper

LGTM. Thanks!

* Provide a pure pytorch/jit path to avoid required dependency on TE and Apex Signed-off-by: ashors1 <ashors@nvidia.com> * add missing file Signed-off-by: ashors1 <ashors@nvidia.com> * add minimal gpt pretraining example Signed-off-by: ashors1 <ashors@nvidia.com> * fix pre-training datamodule initialization Signed-off-by: ashors1 <ashors@nvidia.com> * add non-te/non-apex test Signed-off-by: ashors1 <ashors@nvidia.com> * add comment to pretraining script Signed-off-by: ashors1 <ashors@nvidia.com> * use microbatch calculator from mcore Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix nemo 2 test name Signed-off-by: ashors1 <ashors@nvidia.com> * update Mcore commit for CI Signed-off-by: ashors1 <ashors@nvidia.com> * replace apex microbatch calculator with megatron's in more places Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix missing import Signed-off-by: ashors1 <ashors@nvidia.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * fix missed apex import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add types to command-line args Signed-off-by: ashors1 <ashors@nvidia.com> * bug fix Signed-off-by: ashors1 <ashors@nvidia.com> * fix path Signed-off-by: ashors1 <ashors@nvidia.com> * Disable distributed optimizer in nemo 2.0 test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix optimizer config Signed-off-by: ashors1 <ashors@nvidia.com> * update checkpointing Signed-off-by: ashors1 <ashors@nvidia.com> * move import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix failing unit test Signed-off-by: ashors1 <ashors@nvidia.com> * fix failing test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * Updating num_weights check of RETRO due to underlying changes from mcore RETRO MLM Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * remove stale warning Signed-off-by: ashors1 <ashors@nvidia.com> * fix lora notebook Signed-off-by: ashors1 <ashors@nvidia.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * add import guards to gemma2 Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com>

* [NeMo-UX] Make TE and Apex dependencies optional (#9550) * Provide a pure pytorch/jit path to avoid required dependency on TE and Apex Signed-off-by: ashors1 <ashors@nvidia.com> * add missing file Signed-off-by: ashors1 <ashors@nvidia.com> * add minimal gpt pretraining example Signed-off-by: ashors1 <ashors@nvidia.com> * fix pre-training datamodule initialization Signed-off-by: ashors1 <ashors@nvidia.com> * add non-te/non-apex test Signed-off-by: ashors1 <ashors@nvidia.com> * add comment to pretraining script Signed-off-by: ashors1 <ashors@nvidia.com> * use microbatch calculator from mcore Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix nemo 2 test name Signed-off-by: ashors1 <ashors@nvidia.com> * update Mcore commit for CI Signed-off-by: ashors1 <ashors@nvidia.com> * replace apex microbatch calculator with megatron's in more places Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix missing import Signed-off-by: ashors1 <ashors@nvidia.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * fix missed apex import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add types to command-line args Signed-off-by: ashors1 <ashors@nvidia.com> * bug fix Signed-off-by: ashors1 <ashors@nvidia.com> * fix path Signed-off-by: ashors1 <ashors@nvidia.com> * Disable distributed optimizer in nemo 2.0 test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix optimizer config Signed-off-by: ashors1 <ashors@nvidia.com> * update checkpointing Signed-off-by: ashors1 <ashors@nvidia.com> * move import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix failing unit test Signed-off-by: ashors1 <ashors@nvidia.com> * fix failing test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * Updating num_weights check of RETRO due to underlying changes from mcore RETRO MLM Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * remove stale warning Signed-off-by: ashors1 <ashors@nvidia.com> * fix lora notebook Signed-off-by: ashors1 <ashors@nvidia.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * add import guards to gemma2 Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix cherry-pick Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com>

* [NeMo-UX] Make TE and Apex dependencies optional (NVIDIA#9550) * Provide a pure pytorch/jit path to avoid required dependency on TE and Apex Signed-off-by: ashors1 <ashors@nvidia.com> * add missing file Signed-off-by: ashors1 <ashors@nvidia.com> * add minimal gpt pretraining example Signed-off-by: ashors1 <ashors@nvidia.com> * fix pre-training datamodule initialization Signed-off-by: ashors1 <ashors@nvidia.com> * add non-te/non-apex test Signed-off-by: ashors1 <ashors@nvidia.com> * add comment to pretraining script Signed-off-by: ashors1 <ashors@nvidia.com> * use microbatch calculator from mcore Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix nemo 2 test name Signed-off-by: ashors1 <ashors@nvidia.com> * update Mcore commit for CI Signed-off-by: ashors1 <ashors@nvidia.com> * replace apex microbatch calculator with megatron's in more places Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix missing import Signed-off-by: ashors1 <ashors@nvidia.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * fix missed apex import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add types to command-line args Signed-off-by: ashors1 <ashors@nvidia.com> * bug fix Signed-off-by: ashors1 <ashors@nvidia.com> * fix path Signed-off-by: ashors1 <ashors@nvidia.com> * Disable distributed optimizer in nemo 2.0 test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix optimizer config Signed-off-by: ashors1 <ashors@nvidia.com> * update checkpointing Signed-off-by: ashors1 <ashors@nvidia.com> * move import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix failing unit test Signed-off-by: ashors1 <ashors@nvidia.com> * fix failing test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * Updating num_weights check of RETRO due to underlying changes from mcore RETRO MLM Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * remove stale warning Signed-off-by: ashors1 <ashors@nvidia.com> * fix lora notebook Signed-off-by: ashors1 <ashors@nvidia.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * add import guards to gemma2 Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix cherry-pick Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com>

* [NeMo-UX] Make TE and Apex dependencies optional (#9550) * Provide a pure pytorch/jit path to avoid required dependency on TE and Apex Signed-off-by: ashors1 <ashors@nvidia.com> * add missing file Signed-off-by: ashors1 <ashors@nvidia.com> * add minimal gpt pretraining example Signed-off-by: ashors1 <ashors@nvidia.com> * fix pre-training datamodule initialization Signed-off-by: ashors1 <ashors@nvidia.com> * add non-te/non-apex test Signed-off-by: ashors1 <ashors@nvidia.com> * add comment to pretraining script Signed-off-by: ashors1 <ashors@nvidia.com> * use microbatch calculator from mcore Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix nemo 2 test name Signed-off-by: ashors1 <ashors@nvidia.com> * update Mcore commit for CI Signed-off-by: ashors1 <ashors@nvidia.com> * replace apex microbatch calculator with megatron's in more places Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix missing import Signed-off-by: ashors1 <ashors@nvidia.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * fix missed apex import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add types to command-line args Signed-off-by: ashors1 <ashors@nvidia.com> * bug fix Signed-off-by: ashors1 <ashors@nvidia.com> * fix path Signed-off-by: ashors1 <ashors@nvidia.com> * Disable distributed optimizer in nemo 2.0 test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix optimizer config Signed-off-by: ashors1 <ashors@nvidia.com> * update checkpointing Signed-off-by: ashors1 <ashors@nvidia.com> * move import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix failing unit test Signed-off-by: ashors1 <ashors@nvidia.com> * fix failing test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * Updating num_weights check of RETRO due to underlying changes from mcore RETRO MLM Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * remove stale warning Signed-off-by: ashors1 <ashors@nvidia.com> * fix lora notebook Signed-off-by: ashors1 <ashors@nvidia.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * add import guards to gemma2 Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix cherry-pick Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com>

* [NeMo-UX] Make TE and Apex dependencies optional (NVIDIA#9550) * Provide a pure pytorch/jit path to avoid required dependency on TE and Apex Signed-off-by: ashors1 <ashors@nvidia.com> * add missing file Signed-off-by: ashors1 <ashors@nvidia.com> * add minimal gpt pretraining example Signed-off-by: ashors1 <ashors@nvidia.com> * fix pre-training datamodule initialization Signed-off-by: ashors1 <ashors@nvidia.com> * add non-te/non-apex test Signed-off-by: ashors1 <ashors@nvidia.com> * add comment to pretraining script Signed-off-by: ashors1 <ashors@nvidia.com> * use microbatch calculator from mcore Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix nemo 2 test name Signed-off-by: ashors1 <ashors@nvidia.com> * update Mcore commit for CI Signed-off-by: ashors1 <ashors@nvidia.com> * replace apex microbatch calculator with megatron's in more places Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix missing import Signed-off-by: ashors1 <ashors@nvidia.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * fix missed apex import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add types to command-line args Signed-off-by: ashors1 <ashors@nvidia.com> * bug fix Signed-off-by: ashors1 <ashors@nvidia.com> * fix path Signed-off-by: ashors1 <ashors@nvidia.com> * Disable distributed optimizer in nemo 2.0 test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix optimizer config Signed-off-by: ashors1 <ashors@nvidia.com> * update checkpointing Signed-off-by: ashors1 <ashors@nvidia.com> * move import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix failing unit test Signed-off-by: ashors1 <ashors@nvidia.com> * fix failing test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * Updating num_weights check of RETRO due to underlying changes from mcore RETRO MLM Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * remove stale warning Signed-off-by: ashors1 <ashors@nvidia.com> * fix lora notebook Signed-off-by: ashors1 <ashors@nvidia.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * add import guards to gemma2 Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix cherry-pick Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> Signed-off-by: Malay Nagda <malayn@malayn-mlt.client.nvidia.com>

* [NeMo-UX] Make TE and Apex dependencies optional (NVIDIA#9550) * Provide a pure pytorch/jit path to avoid required dependency on TE and Apex Signed-off-by: ashors1 <ashors@nvidia.com> * add missing file Signed-off-by: ashors1 <ashors@nvidia.com> * add minimal gpt pretraining example Signed-off-by: ashors1 <ashors@nvidia.com> * fix pre-training datamodule initialization Signed-off-by: ashors1 <ashors@nvidia.com> * add non-te/non-apex test Signed-off-by: ashors1 <ashors@nvidia.com> * add comment to pretraining script Signed-off-by: ashors1 <ashors@nvidia.com> * use microbatch calculator from mcore Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix nemo 2 test name Signed-off-by: ashors1 <ashors@nvidia.com> * update Mcore commit for CI Signed-off-by: ashors1 <ashors@nvidia.com> * replace apex microbatch calculator with megatron's in more places Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix missing import Signed-off-by: ashors1 <ashors@nvidia.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * fix missed apex import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add types to command-line args Signed-off-by: ashors1 <ashors@nvidia.com> * bug fix Signed-off-by: ashors1 <ashors@nvidia.com> * fix path Signed-off-by: ashors1 <ashors@nvidia.com> * Disable distributed optimizer in nemo 2.0 test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix optimizer config Signed-off-by: ashors1 <ashors@nvidia.com> * update checkpointing Signed-off-by: ashors1 <ashors@nvidia.com> * move import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix failing unit test Signed-off-by: ashors1 <ashors@nvidia.com> * fix failing test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * Updating num_weights check of RETRO due to underlying changes from mcore RETRO MLM Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * remove stale warning Signed-off-by: ashors1 <ashors@nvidia.com> * fix lora notebook Signed-off-by: ashors1 <ashors@nvidia.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * add import guards to gemma2 Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix cherry-pick Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> Signed-off-by: tonyjie <jl4257@cornell.edu>

* Provide a pure pytorch/jit path to avoid required dependency on TE and Apex Signed-off-by: ashors1 <ashors@nvidia.com> * add missing file Signed-off-by: ashors1 <ashors@nvidia.com> * add minimal gpt pretraining example Signed-off-by: ashors1 <ashors@nvidia.com> * fix pre-training datamodule initialization Signed-off-by: ashors1 <ashors@nvidia.com> * add non-te/non-apex test Signed-off-by: ashors1 <ashors@nvidia.com> * add comment to pretraining script Signed-off-by: ashors1 <ashors@nvidia.com> * use microbatch calculator from mcore Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix nemo 2 test name Signed-off-by: ashors1 <ashors@nvidia.com> * update Mcore commit for CI Signed-off-by: ashors1 <ashors@nvidia.com> * replace apex microbatch calculator with megatron's in more places Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix missing import Signed-off-by: ashors1 <ashors@nvidia.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * fix missed apex import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add types to command-line args Signed-off-by: ashors1 <ashors@nvidia.com> * bug fix Signed-off-by: ashors1 <ashors@nvidia.com> * fix path Signed-off-by: ashors1 <ashors@nvidia.com> * Disable distributed optimizer in nemo 2.0 test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix optimizer config Signed-off-by: ashors1 <ashors@nvidia.com> * update checkpointing Signed-off-by: ashors1 <ashors@nvidia.com> * move import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix failing unit test Signed-off-by: ashors1 <ashors@nvidia.com> * fix failing test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * Updating num_weights check of RETRO due to underlying changes from mcore RETRO MLM Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * remove stale warning Signed-off-by: ashors1 <ashors@nvidia.com> * fix lora notebook Signed-off-by: ashors1 <ashors@nvidia.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * add import guards to gemma2 Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com>

* [NeMo-UX] Make TE and Apex dependencies optional (#9550) * Provide a pure pytorch/jit path to avoid required dependency on TE and Apex Signed-off-by: ashors1 <ashors@nvidia.com> * add missing file Signed-off-by: ashors1 <ashors@nvidia.com> * add minimal gpt pretraining example Signed-off-by: ashors1 <ashors@nvidia.com> * fix pre-training datamodule initialization Signed-off-by: ashors1 <ashors@nvidia.com> * add non-te/non-apex test Signed-off-by: ashors1 <ashors@nvidia.com> * add comment to pretraining script Signed-off-by: ashors1 <ashors@nvidia.com> * use microbatch calculator from mcore Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix nemo 2 test name Signed-off-by: ashors1 <ashors@nvidia.com> * update Mcore commit for CI Signed-off-by: ashors1 <ashors@nvidia.com> * replace apex microbatch calculator with megatron's in more places Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix missing import Signed-off-by: ashors1 <ashors@nvidia.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * fix missed apex import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add types to command-line args Signed-off-by: ashors1 <ashors@nvidia.com> * bug fix Signed-off-by: ashors1 <ashors@nvidia.com> * fix path Signed-off-by: ashors1 <ashors@nvidia.com> * Disable distributed optimizer in nemo 2.0 test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix optimizer config Signed-off-by: ashors1 <ashors@nvidia.com> * update checkpointing Signed-off-by: ashors1 <ashors@nvidia.com> * move import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix failing unit test Signed-off-by: ashors1 <ashors@nvidia.com> * fix failing test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * Updating num_weights check of RETRO due to underlying changes from mcore RETRO MLM Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * remove stale warning Signed-off-by: ashors1 <ashors@nvidia.com> * fix lora notebook Signed-off-by: ashors1 <ashors@nvidia.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * add import guards to gemma2 Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix cherry-pick Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com>

* [NeMo-UX] Make TE and Apex dependencies optional (NVIDIA#9550) * Provide a pure pytorch/jit path to avoid required dependency on TE and Apex Signed-off-by: ashors1 <ashors@nvidia.com> * add missing file Signed-off-by: ashors1 <ashors@nvidia.com> * add minimal gpt pretraining example Signed-off-by: ashors1 <ashors@nvidia.com> * fix pre-training datamodule initialization Signed-off-by: ashors1 <ashors@nvidia.com> * add non-te/non-apex test Signed-off-by: ashors1 <ashors@nvidia.com> * add comment to pretraining script Signed-off-by: ashors1 <ashors@nvidia.com> * use microbatch calculator from mcore Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix nemo 2 test name Signed-off-by: ashors1 <ashors@nvidia.com> * update Mcore commit for CI Signed-off-by: ashors1 <ashors@nvidia.com> * replace apex microbatch calculator with megatron's in more places Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix missing import Signed-off-by: ashors1 <ashors@nvidia.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * fix missed apex import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add types to command-line args Signed-off-by: ashors1 <ashors@nvidia.com> * bug fix Signed-off-by: ashors1 <ashors@nvidia.com> * fix path Signed-off-by: ashors1 <ashors@nvidia.com> * Disable distributed optimizer in nemo 2.0 test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix optimizer config Signed-off-by: ashors1 <ashors@nvidia.com> * update checkpointing Signed-off-by: ashors1 <ashors@nvidia.com> * move import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix failing unit test Signed-off-by: ashors1 <ashors@nvidia.com> * fix failing test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * Updating num_weights check of RETRO due to underlying changes from mcore RETRO MLM Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * remove stale warning Signed-off-by: ashors1 <ashors@nvidia.com> * fix lora notebook Signed-off-by: ashors1 <ashors@nvidia.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * add import guards to gemma2 Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix cherry-pick Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>

ashors1 requested review from marcromeyn and akoumpa June 26, 2024 22:28

github-actions bot added core Changes to NeMo Core NLP labels Jun 26, 2024

ashors1 changed the title ~~Ashors/remove te apex deps~~ Make TE and Apex dependencies optional Jun 26, 2024

github-actions bot removed the core Changes to NeMo Core label Jun 26, 2024

ashors1 marked this pull request as draft June 26, 2024 23:39

github-advanced-security bot found potential problems Jun 27, 2024

View reviewed changes

examples/llm/megatron_gpt_pretraining.py Fixed Show fixed Hide fixed

github-actions bot added the CI label Jun 30, 2024

ericharper requested review from dimapihtar and removed request for marcromeyn and akoumpa July 3, 2024 19:12

dimapihtar reviewed Jul 4, 2024

View reviewed changes

ashors1 marked this pull request as ready for review July 5, 2024 06:47

ashors1 changed the title ~~Make TE and Apex dependencies optional~~ [NeMo-UX] Make TE and Apex dependencies optional Jul 5, 2024

github-advanced-security bot found potential problems Jul 5, 2024

View reviewed changes

examples/llm/megatron_gpt_pretraining.py Fixed Show fixed Hide fixed

nemo/lightning/data.py Outdated

setup_microbatch_calculator,

)

HAVE_APEX = True

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable HAVE_APEX is not used.

ashors1 force-pushed the ashors/remove-te-apex-deps branch from ed46aec to 9e0fbbd Compare July 5, 2024 19:50

github-actions bot added core Changes to NeMo Core TTS ASR common Multi Modal audio labels Jul 5, 2024

ashors1 marked this pull request as draft July 5, 2024 19:56

ericharper added Run CICD and removed Run CICD labels Jul 5, 2024

ashors1 force-pushed the ashors/remove-te-apex-deps branch from 6f3f671 to 6abeff6 Compare July 5, 2024 20:05

ashors1 marked this pull request as ready for review July 5, 2024 20:06

github-actions bot removed the core Changes to NeMo Core label Jul 5, 2024

ashors1 added the Run CICD label Jul 12, 2024

Merge branch 'r2.0.0rc1' into ashors/remove-te-apex-deps

2f42db7