Skip to content

Commit

Permalink
NeMo dev doc restructure (NVIDIA#8896)
Browse files Browse the repository at this point in the history
* Update intro and why nemo in dev doc

* Categorize tutorials

* Update tutorials link

* update index

* Restructure

* Restructure

* Restructure

* Restructure

* Restructure

* Restructure

* Restructure

* Restructure

* Update flash attention

* Update flash attention

* Fix few structure issue

* Fix migration

* Fix structure

* Fix structure

* Few updates

* Add few more scripts

* Fix scripts

* Fix few things

* Fix tutorial table

* Restructure

* Rename

* Few fixes and moves

* Move sections

* Fix bib

* Refactor files

* Fixes

* Fix

* Fix few issues

* remove scripts

* Update docs

---------

Co-authored-by: Pablo Garay <palenq@gmail.com>
  • Loading branch information
yaoyu-33 and pablo-garay authored Apr 25, 2024
1 parent e710433 commit 557ab0d
Show file tree
Hide file tree
Showing 26 changed files with 513 additions and 617 deletions.
6 changes: 3 additions & 3 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Latest News
<details open>
<summary><b>Large Language Models and Multimodal</b></summary>
<details>
<summary><a href="https://cloud.google.com/blog/products/compute/gke-and-nvidia-nemo-framework-to-train-generative-ai-models">Accelerate your generative AI journey with NVIDIA NeMo framework on GKE</a> (2024/03/16) </summary>
<summary><a href="https://cloud.google.com/blog/products/compute/gke-and-nvidia-nemo-framework-to-train-generative-ai-models">Accelerate your generative AI journey with NVIDIA NeMo Framework on GKE</a> (2024/03/16) </summary>

An end-to-end walkthrough to train generative AI models on the Google Kubernetes Engine (GKE) using the NVIDIA NeMo Framework is available at https://github.com/GoogleCloudPlatform/nvidia-nemo-on-gke. The walkthrough includes detailed instructions on how to set up a Google Cloud Project and pre-train a GPT model using the NeMo Framework.
<br><br>
Expand All @@ -71,7 +71,7 @@ Latest News
<details>
<summary><a href="https://blogs.nvidia.com/blog/nemo-amazon-titan/">NVIDIA now powers training for Amazon Titan Foundation models</a> (2023/11/28) </summary>

NVIDIA NeMo framework now empowers the Amazon Titan foundation models (FM) with efficient training of large language models (LLMs). The Titan FMs form the basis of Amazon’s generative AI service, Amazon Bedrock. The NeMo Framework provides a versatile framework for building, customizing, and running LLMs.
NVIDIA NeMo Framework now empowers the Amazon Titan foundation models (FM) with efficient training of large language models (LLMs). The Titan FMs form the basis of Amazon’s generative AI service, Amazon Bedrock. The NeMo Framework provides a versatile framework for building, customizing, and running LLMs.
<br><br>
</details>

Expand Down Expand Up @@ -486,7 +486,7 @@ We welcome community contributions! Please refer to `CONTRIBUTING.md <https://gi
Publications
------------

We provide an ever-growing list of `publications <https://nvidia.github.io/NeMo/publications/>`_ that utilize the NeMo framework.
We provide an ever-growing list of `publications <https://nvidia.github.io/NeMo/publications/>`_ that utilize the NeMo Framework.

If you would like to add your own article to the list, you are welcome to do so via a pull request to this repository's ``gh-pages-src`` branch.
Please refer to the instructions in the `README of that branch <https://github.com/NVIDIA/NeMo/tree/gh-pages-src#readme>`_.
Expand Down
32 changes: 32 additions & 0 deletions docs/source/ckpt_converters/convert_mlm.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
Converting from Megatron-LM
===========================

NVIDIA NeMo and NVIDIA Megatron-LM share many underlying technologies. This document provides guidance for migrating your project from Megatron-LM to NVIDIA NeMo.

Converting Checkpoints
----------------------

You can convert your GPT-style model checkpoints trained with Megatron-LM into the NeMo Framework using the provided example script. This script facilitates the conversion of Megatron-LM checkpoints to NeMo compatible formats.

.. code-block:: bash
<NeMo_ROOT_FOLDER>/examples/nlp/language_modeling/megatron_lm_ckpt_to_nemo.py \
--checkpoint_folder <path_to_PTL_checkpoints_folder> \
--checkpoint_name megatron_gpt--val_loss=99.99-step={steps}-consumed_samples={consumed}.0 \
--nemo_file_path <path_to_output_nemo_file> \
--model_type <megatron_model_type> \
--tensor_model_parallel_size <tensor_model_parallel_size> \
--pipeline_model_parallel_size <pipeline_model_parallel_size> \
--gpus_per_node <gpus_per_node>
Resuming Training
-----------------

To resume training from a converted Megatron-LM checkpoint, it is crucial to correctly set up the training parameters to match the previous learning rate schedule. Use the following setting for the `trainer.max_steps` parameter in your NeMo training configuration:

.. code-block:: none
trainer.max_steps=round(lr-warmup-fraction * lr-decay-iters + lr-decay-iters)
This configuration ensures that the learning rate scheduler in NeMo continues from where it left off in Megatron-LM, using the `lr-warmup-fraction` and `lr-decay-iters` arguments from the original Megatron-LM training setup.

22 changes: 22 additions & 0 deletions docs/source/ckpt_converters/intro.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
Community Checkpoint Converter
==============================

We provide easy-to-use tools that enable users to convert community checkpoints into the NeMo format. These tools facilitate various operations, including resuming training, Sparse Fine-Tuning (SFT), Parameter-Efficient Fine-Tuning (PEFT), and deployment. For detailed instructions and guidelines, please refer to our documentation.

We offer comprehensive guides to assist both end users and developers:

- **User Guide**: Detailed steps on how to convert community model checkpoints for further training or deployment within NeMo. For more information, please see our :doc:`user_guide`.

- **Developer Guide**: Instructions for developers on how to implement converters for community model checkpoints, allowing for broader compatibility and integration within the NeMo ecosystem. For development details, refer to our :doc:`dev_guide`.

- **Megatron-LM Checkpoint Conversion**: NVIDIA NeMo and NVIDIA Megatron-LM share several foundational technologies. You can convert your GPT-style model checkpoints trained with Megatron-LM into the NeMo Framework using our scripts, see our :doc:`convert_mlm`.

Access the user and developer guides directly through the links below:

.. toctree::
:maxdepth: 1
:caption: Conversion Guides

user_guide
dev_guide
convert_mlm
70 changes: 70 additions & 0 deletions docs/source/collections.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
================
NeMo Collections
================

Documentation for the individual collections

.. toctree::
:maxdepth: 1
:caption: Large Language Models (LLMs)
:name: Large Language Models
:titlesonly:

nlp/nemo_megatron/intro
nlp/models
nlp/machine_translation/machine_translation
nlp/megatron_onnx_export
nlp/quantization
nlp/api


.. toctree::
:maxdepth: 1
:caption: Speech AI
:name: Speech AI
:titlesonly:

asr/intro
asr/speech_classification/intro
asr/speaker_recognition/intro
asr/speaker_diarization/intro
asr/ssl/intro
asr/speech_intent_slot/intro


.. toctree::
:maxdepth: 1
:caption: Multimodal Models (MMs)
:name: Multimodal
:titlesonly:

multimodal/mllm/intro
multimodal/vlm/intro
multimodal/text2img/intro
multimodal/nerf/intro
multimodal/api


.. toctree::
:maxdepth: 1
:caption: Text To Speech (TTS)
:name: Text To Speech
:titlesonly:

tts/intro

.. toctree::
:maxdepth: 1
:caption: Vision (CV)
:name: vision
:titlesonly:

vision/intro

.. toctree::
:maxdepth: 1
:caption: Common
:name: Common
:titlesonly:

common/intro
4 changes: 2 additions & 2 deletions docs/source/core/core_index.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
=========
NeMo Core
NeMo APIs
=========

You can learn more about the underlying principles of the NeMo codebase in this section.
Expand Down Expand Up @@ -30,7 +30,7 @@ Alternatively, you can jump straight to the documentation for the individual col

* :doc:`Automatic Speech Recognition (ASR) <../asr/intro>`

* :doc:`Multimodal (MM) Models <../multimodal/mllm/intro>`
* :doc:`Multimodal Models (MMs) <../multimodal/mllm/intro>`

* :doc:`Text-to-Speech (TTS) <../tts/intro>`

Expand Down
48 changes: 48 additions & 0 deletions docs/source/features/memory_optimizations.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
Memory Optimizations
====================

Parallelism
-----------
Refer to :doc:`Parallelism <./parallelism>`.

Flash Attention
---------------

Overview
^^^^^^^^

Flash Attention is a method designed to enhance the efficiency of Transformer models, which are widely utilized in applications such as Natural Language Processing (NLP). Traditional Transformers are slow and consume a lot of memory, especially with long sequences, due to the quadratic time and memory complexity of self-attention. FlashAttention, an IO-aware exact attention algorithm that leverages tiling to minimize the number of memory reads/writes between the GPU's high bandwidth memory (HBM) and on-chip SRAM. This approach is designed to be more efficient in terms of IO complexity compared to standard attention mechanisms.

Turn Flash Attention On and Off
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In the NeMo Framework, Flash Attention is supported through the Transformer Engine with the inclusion of Flash Attention 2. By default, Flash Attention is enabled, but the Transformer Engine may switch to a different kernel if the tensor dimensions are not optimal for Flash Attention. Users can completely disable Flash Attention by setting the environment variable ``NVTE_FLASH_ATTN=0``.

For more details on the supported Dot Attention backend, please refer to the Transformer Engine source code available at `Transformer Engine's Attention Mechanism <https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention.py>`_.

.. bibliography:: ./nlp_all.bib
:style: plain
:labelprefix: nlp-megatron
:keyprefix: nlp-megatron-

Overview
^^^^^^^^

Full Activation Recomputation
"""""""""""""""""""""""""""""
This method recalculates all the intermediate activations during the backward pass of a model's training, instead of storing them during the forward pass. This technique maximizes memory efficiency at the cost of computational overhead, as each activation is recomputed when needed.

Partial Activation Recomputation
""""""""""""""""""""""""""""""""
This method recomputes only a subset of layers during the backward phase. It is a trade-off between the full recomputation and no recomputation, balancing memory savings with computational efficiency.

Selective Activation Recomputation
""""""""""""""""""""""""""""""""""
This method reduces memory footprint of activations significantly via smart activation checkpointing. This approach involves selectively storing only crucial activations and recomputing the others as needed. It is particularly useful in large models to minimize memory usage while controlling the computational cost.

Refer to "Reducing Activation Recomputation in Large Transformer Models" for more details: https://arxiv.org/abs/2205.05198

.. bibliography:: ./nlp_all.bib
:style: plain
:labelprefix: nlp-megatron
:keyprefix: nlp-megatron-
6 changes: 6 additions & 0 deletions docs/source/features/mixed_precision.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.. _mix_precision:

Mixed Precision Training
------------------------

Mixed precision training significantly enhances computational efficiency by conducting operations in half-precision and fp8 formats, while selectively maintaining minimal data in single-precision to preserve critical information throughout key areas of the network. NeMo now supports FP16, BF16, and FP8 (via Transformer Engine) across most models. Further details will be provided shortly.
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@
Parallelisms
------------

NeMo Megatron supports 5 types of parallelisms (which can be mixed together arbitraritly):
NeMo Megatron supports 5 types of parallelisms (which can be mixed together arbitrarily):

Distributed Data parallelism
Distributed Data Parallelism
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Distributed Data parallelism (DDP) creates idential copies of the model across multiple GPUs.
Distributed Data Parallelism (DDP) creates idential copies of the model across multiple GPUs.

.. image:: images/ddp.gif
.. image:: ../nlp/nemo_megatron/images/ddp.gif
:align: center
:width: 800px
:alt: Distributed Data Parallel
Expand All @@ -20,7 +20,7 @@ Tensor Parallelism
With Tensor Paralellism (TP) a tensor is split into non-overlapping pieces and
different parts are distributed and processed on separate GPUs.

.. image:: images/tp.gif
.. image:: ../nlp/nemo_megatron/images/tp.gif
:align: center
:width: 800px
:alt: Tensor Parallel
Expand All @@ -29,15 +29,15 @@ Pipeline Parallelism
^^^^^^^^^^^^^^^^^^^^
With Pipeline Paralellism (PP) consecutive layer chunks are assigned to different GPUs.

.. image:: images/pp.gif
.. image:: ../nlp/nemo_megatron/images/pp.gif
:align: center
:width: 800px
:alt: Pipeline Parallel

Sequence Parallelism
^^^^^^^^^^^^^^^^^^^^

.. image:: images/sp.gif
.. image:: ../nlp/nemo_megatron/images/sp.gif
:align: center
:width: 800px
:alt: Sequence Parallel
Expand All @@ -47,7 +47,7 @@ Expert Parallelism
Expert Paralellim (EP) distributes experts across GPUs.


.. image:: images/ep.png
.. image:: ../nlp/nemo_megatron/images/ep.png
:align: center
:width: 800px
:alt: Expert Parallelism
Expand All @@ -57,7 +57,7 @@ Parallelism nomenclature

When reading and modifying NeMo Megatron code you will encounter the following terms.

.. image:: images/pnom.gif
.. image:: ../nlp/nemo_megatron/images/pnom.gif
:align: center
:width: 800px
:alt: Parallelism nomenclature
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
Throughput Optimizations
========================

Sequence Packing for SFT/PEFT
-----------------------------


Overview
^^^^^^^^

Expand Down Expand Up @@ -133,6 +135,10 @@ To train with packed sequences, you need to change four items in the SFT/PEFT co
Now you are all set to finetune your model with a much improved throughput!

Communication Overlap
---------------------
NeMo leverages Megatron-Core's optimizations to enhance bandwidth utilization and effectively overlap computation with communication. Additional details will be provided soon.


.. rubric:: Footnotes

Expand Down
Loading

0 comments on commit 557ab0d

Please sign in to comment.