Mistral + Mixtral Support for NeVa #9459
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: paul-gibbons <paul@gibbonspaul.com>
Signed-off-by: paul-gibbons <paul@gibbonspaul.com>
Signed-off-by: paul-gibbons <paul@gibbonspaul.com>
Signed-off-by: paul-gibbons <paul@gibbonspaul.com>
Signed-off-by: paul-gibbons <paul@gibbonspaul.com>
Signed-off-by: paul-gibbons <paul@gibbonspaul.com>
Signed-off-by: paul-gibbons <paul@gibbonspaul.com>
Signed-off-by: paul-gibbons <paul@gibbonspaul.com>
paul-gibbons
changed the title
Signed-off-by: paul-gibbons <paul@gibbonspaul.com>
Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>
yaoyu-33
previously approved these changes
Jun 13, 2024
Signed-off-by: paul-gibbons <paul@gibbonspaul.com>
Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @@ -75,7 +75,7 @@ | |||
| HAVE_APEX = False | |||
|
|
|||
| try: | |||
| from megatron.core import InferenceParams, dist_checkpointing, parallel_state | |||
| from megatron.core import InferenceParams, dist_checkpointing, parallel_state, tensor_parallel | |||
Check notice
Code scanning / CodeQL
Unused import Note
yaoyu-33
approved these changes
Jul 2, 2024
yaoyu-33
approved these changes
Jul 8, 2024
pablo-garay
pushed a commit
that referenced
this pull request
Jul 8, 2024
* mistral template support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * get_specs neva fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fixed mistral tokenization Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * text_gen_strategy add mistral support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral text_gen fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Cleaning up neva config Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fix llama_2 default text_gen_strategy Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * fix forward() to account for new embedding optimization in MCore Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com>
ertkonuk
pushed a commit
that referenced
this pull request
Jul 19, 2024
* mistral template support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * get_specs neva fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fixed mistral tokenization Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * text_gen_strategy add mistral support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral text_gen fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Cleaning up neva config Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fix llama_2 default text_gen_strategy Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * fix forward() to account for new embedding optimization in MCore Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com>
akoumpa
added a commit
that referenced
this pull request
Jul 25, 2024
* Adding context- & expert-parallism to MegatronStrategy (#9525) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add CICD test for Stable Diffusion (#9464) * Add CICD test for Stable Diffusion Signed-off-by: Michal Futrega <mfutrega@nvidia.com> * Update cicd-main.yml Signed-off-by: Michal Futrega <mfutrega@nvidia.com> * Use single gpu runner Signed-off-by: Michal Futrega <mfutrega@nvidia.com> --------- Signed-off-by: Michal Futrega <mfutrega@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Akoumparouli/nemo ux mixtral (#9446) * use default collate if dataset does not have one Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mixtral config Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add convert_state Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix StateDictTransform for 2D layers, e.g. MoE Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pass num_moe_experts to specs Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * udpate MixtralModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mini docstring Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * update mcoreddp call (#9345) * update mcoreddp call Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update mcore commits Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Llama and Gemma (#9528) * add llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add llama3 Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * fix typo Signed-off-by: Chen Cui <chcui@nvidia.com> * enable importers with multiple models Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add gemma Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * checks Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] minor logging bug fixes (#9529) * minor exp_manager bug fixes * remove print statement * fix docstring * fix AppState defaults --------- Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * mcore distOpt restore fix (#9421) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Custom Tiktoken tokenizer. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fixed the tokenizer decoding on special tokens. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Apply isort and black reformatting Signed-off-by: ertkonuk <ertkonuk@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Added token_to_id() method. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update neva conversion script from and to HF (#9296) * Update NeMo script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix example scripts Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * Update convert_llava_nemo_to_hf.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * vLLM Export Support (#9381) * Export implementation for vLLM 0.4.3. Supports LLAMA2, Mistral, Mixtral (unverified), Gemma and StarCoder2 models. The nemo.export.tensorrt_llm alias was removed to avoid initializing TRT-LLM when importing anything from nemo.export. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Fixed some CodeQL warnings. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Removed empty files. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Updated the integration for vLLM 0.5.0. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Updated the vLLM deployment interface to use max_output_len instead of max_output_token. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Moved the Exporter class to nemo/export and renamed its file to vllm_exporter.py, to be more similar to TRT-LLM. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Implemented vLLM support in the export tests, added functional testing, implemented forward evaluation on vLLM without Triton. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Moved the vLLM deployment functionality to the common deploy_triton.py script. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Fixed the CodeQL discovered issues. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Fixed one more return of a wrong dimensionality... Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * More wrong dimensionality returns. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> --------- Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * PL: Delete precision if using plugin. TODO switch to MegatronTrainerBuilder (#9535) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add page context fmha (#9526) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * extend get_gpt_layer_modelopt_spec to support MoE (#9532) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix mock data generation for legacy dataset (#9530) Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] IO fixes (#9512) * Improve IOMixin.io_transform_args to handle dataclasses better * Dump task json + img inside NeMoLogger * Adding store_io to train task * Update opt.connect to also propagate to __io__ * Rename opt to optim for consistency * Moving to using safe serialization using fiddle, only use cloudpickle when needed * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Using Config from fiddle instead of sdk for now * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move enable_nemo_ckpt_io from MegatronStrategy to ModelCheckpoint * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move nemo-ckpt to _get_finalize_save_checkpoint_callback * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Update TrainerContext & io.load_ckpt * Use renamed TrainerContext inside ModelCheckpoint * Remove double io saving * Rename lightning.pytorch.opt -> optim * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove store_io from train-task * Adding fiddle-extension for torch * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move fdl_torch import * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding dtype to serialization * Some fixes * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Make TransformerConfig inherit from IOMixin to fix serialization error * Make TransformerConfig inherit from IOMixin to fix serialization error * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Add support for BuiltinFunctionType * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Add missing import * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix dataclass fields --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Test C++ runtime on demand in nemo_export.py to avoid possible OOMs (#9544) * Add test_cpp_runtime flag Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Apply isort and black reformatting Signed-off-by: janekl <janekl@users.noreply.github.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: janekl <janekl@users.noreply.github.com> Co-authored-by: janekl <janekl@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix lhotse tests for v1.24.2 (#9546) * Fix lhotse tests for v1.24.0 Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Fix RIR test Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * gpu_unitTests_notOptional (#9551) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * add reset learning rate functionality (#9372) * add reset_lr functionality Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix reset_lr logic Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * move reset_lr from optim section Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * add reset_lr value to config Signed-off-by: dimapihtar <dpihtar@gmail.com> * set reset_lr False by default Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove extra line Signed-off-by: dimapihtar <dpihtar@gmail.com> * add reset_lr test Signed-off-by: dimapihtar <dpihtar@gmail.com> * add reset_lr test Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove extra quote Signed-off-by: dimapihtar <dpihtar@gmail.com> * add ability to reset schedule's max_steps and decay_steps Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * change scheduler's first step logic when using reset_lr Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix reset_lr logic Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * update reset_lr comments Signed-off-by: dimapihtar <dpihtar@gmail.com> * add use cases for reset_lr feature Signed-off-by: dimapihtar <dpihtar@gmail.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add Python AIStore SDK to container and bump min Lhotse version (#9537) * Add Python AIStore SDK to requirements and bump min Lhotse version Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Move AIStore Python SDK to Dockerfile, remove matplotlib/ipywidgets deps Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export() (#9147) * Ininial WARs to implement dynamo option for export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * including weights in .onnx Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * dynamo_export works for many small models Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * External weights behaviour fixed Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Cleanup Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Apply isort and black reformatting Signed-off-by: borisfom <borisfom@users.noreply.github.com> * print cleaned up Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Added overloadable dynamic_shapes_for_export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Addressing code review Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Fixing CI issues Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Fixing CI test failure Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Eliminated test cross-contamination Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Signed-off-by: borisfom <borisfom@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Fix tokenizer IO (#9555) * Adding tokenizer to io-test + making it pass * Handling tokenizer correctly inside dump_io * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Removing not used import --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo UX] Move mistral_7b.py to mistral.py (#9545) * Move mistral_7b.py to mistral.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename MixtralConfig to MixtralConfig8x7B Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mistral rename: mistralconfig7b & mistralmodel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Use closed-formula to round by multiple (#9307) * Use closed-formula to round by multiple Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * ci: Do not attempt to send slack on fork (#9556) * ci: Do not attempt to send slack on fork Signed-off-by: Oliver Koenig <okoenig@nvidia.com> * test Signed-off-by: Oliver Koenig <okoenig@nvidia.com> --------- Signed-off-by: Oliver Koenig <okoenig@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix nemo export test (#9547) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix export test Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix SDXL incorrect name in docs (#9534) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * GPU unit tests: Mark flaky tests to be fixed (#9559) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Bump PTL version (#9557) Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Resiliency] Straggler detection (#9473) * Initial straggler det impl Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed CI code checks Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Removed unused import Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * remove submodule Signed-off-by: Maanu Grover <maanug@nvidia.com> * Updated documentation; Updated callback params; Cosmetic changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed straggler det config; Added basic test Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixes in test_straggler_det.py Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Updated straggler callback API Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * stop_if_detected=False by default Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * switch to torch_dist as default dist checkpointing backend (#9541) Signed-off-by: ashors1 <ashors@nvidia.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Checkpointing bug fixes (#9562) * fix checkpoint loading * fix * fixes * another fix * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add tps and pps params to the export script (#9558) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix export test Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * remove n_gpus param Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * add and fix parameters Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix deploy script Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * rename tps and pps params Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Consolidate gpt continue training script into pretraining script (#9413) * Consolidate gpt continue training with pretraining Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix default config Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add github action cicd Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * extract _integrate_original_checkpoint_data as a method Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix getattr Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Revert "Add github action cicd" This reverts commit a453f16ba2be6413db932623009da893208acdd5. * Update comments in nlp_overrides.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add support to change Multi task model prompt (#9542) * Add support to change Multi task model prompt Signed-off-by: smajumdar <titu1994@gmail.com> * Add support to change Multi task model prompt Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Update nemo/collections/common/prompts/formatter.py Co-authored-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Somshubra Majumdar <titu1994@gmail.com> * Address comments Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Address comments Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Signed-off-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add Multimodal Exporter (#9256) * Add video-neva TRT export * Add TRT inference * Change config * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change export params * Remove unused import * Add neva export * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change unpack nemo * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Add trt infer config * Fix neva trt inference * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Add exporter * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix infer * Add PyTriton * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix deploy wrong dim * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change to pass PIL Image * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix video neva deploy * Change query * Change deploy * Remove unused import * Change ptuning * Change to mm exporter * Add script * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix script --------- Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> Co-authored-by: meatybobby <meatybobby@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Enable encoder adapters for Canary and MultiTaskAED models (#9409) * Fix assertions for adapter types Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Cleanup Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Finalize support for decoder adapters Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * fix the freeze/unfreeze problem by replacing as_frozen with torch.inference_mode * Apply isort and black reformatting Signed-off-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> * Update tests to new generic way of module update Signed-off-by: smajumdar <titu1994@gmail.com> * Finalize code for update module Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Fix variable name Signed-off-by: smajumdar <titu1994@gmail.com> * Finalize projection support for transformer mha adapters Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Correct implementation of freeze restore Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Corrects the implementation of replace_adapter_modules to limit to just the top level modules Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Remove registration of Transformer MHA Signed-off-by: smajumdar <titu1994@gmail.com> * Remove registration of Transformer MHA Signed-off-by: smajumdar <titu1994@gmail.com> * Address reviewer comments Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Signed-off-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> Co-authored-by: Weiqing Wang <weiqingw@nvidia.com> Co-authored-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * pass option through (#9570) Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * PTQ refinements (#9574) * Rename megatron_gpt_quantization -> megatron_gpt_ptq Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Configure export.save_path as dir or tarball Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * PTQ docs update Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Make model_type optional in case of quantized checkpoints Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Drop unused save_nemo_model_config argument Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Audio model collection (#9263) * Audio model collection Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Apply isort and black reformatting Signed-off-by: anteju <anteju@users.noreply.github.com> * Fix imports Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Addressed PR comments Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Apply isort and black reformatting Signed-off-by: anteju <anteju@users.noreply.github.com> --------- Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: anteju <anteju@users.noreply.github.com> Co-authored-by: anteju <anteju@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Fix Trainer serialization (#9571) * Fix Trainer serialization * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update click version requirement (#9580) Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> Co-authored-by: Dong Hyuk Chang <donghyukc@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Fault tolerance] Heartbeat detection (#9352) * Fault tolerance related changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Cosmetic changes in documentation Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Doc update round2 Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: jbieniusiewi <152396322+jbieniusiewi@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add ModelOpt QAT example for Llama2 SFT model (#9326) * add INT4 QAT example for Llama2 SFT model Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * Add config parameter to control kv cache quantization Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * Fix typo in cicd-main.yml for QAT test Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * fix nlp_overrides.py Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * address reviewer feedback Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * quantize unwrapped model Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * add compress export argument for qat config Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set TE flag in legacy -> mcore conversion script (#9585) * set TE flag Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Add fabric-API for manual forward-pass (#9577) * First pass over fabric-API * Adding Trainer -> Fabric conversion * Some small fixes to get a forward-pass in Fabric working * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding doc-string to Fabric.import_model * Adding track_io to io_init of Fabric * Fix Fabric.load_model + add doc-string * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove unused import * Some small fixes * Fix failing test --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Add SDK-factories to llm-collection (#9589) * Adding sdk-factories to llm-collection * Removing _model from mistral + mixtral * Expose lr_scheduler inside lightning * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Multimodal projection layer adapter fix for PP>1 (#9445) * enabling multimodal adapters to load in PP>1 Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * parameterizing validate_access_integrity, set to false when PP>1 Signed-off-by: paul-gibbons <paul@gibbonspaul.com> formatting fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * update nlp_model.py Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * update modelPT with validate_access_integrity Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * updating save_restore_connector w/ validate_access_integrity Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * addressing comment Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * adding validate_access_integrity to super().load_config_and_state_dict() Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * testing reorder of validate_access_integrity for CI failures Signed-off-by: paul-gibbons <paul@gibbonspaul.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add offline quantization script for QLoRA deployment (#9455) * add qlora offline quantization script Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * clean Signed-off-by: Chen Cui <chcui@nvidia.com> * docstring Signed-off-by: Chen Cui <chcui@nvidia.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * qlora support more models (#9488) Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Some improvements to NeMoLogger (#9591) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set n_gpu to None in nemo export (#9593) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * set ngpus to None Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Inflight nemo model export support (#9527) * online model conversion and refit Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * clean code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * add refit, cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * combine weight conversion functions Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * Apply isort and black reformatting Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> * remove debug print Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * fix single gpu and cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * Apply isort and black reformatting Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> --------- Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * vLLM Export Improvements (#9596) * Separated the vLLM export functionality from the common deployment script into deploy_vllm_triton.py. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Fixed vocab_size for LLAMA3. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Export test: fixed deployment testing w/o Megatron, made functional tests optional, added --gpu_memory_utilization. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Addressing review and CodeQL comments. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> --------- Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set finalize_model_grads_func in on_fit_start instead to make sure it's being called (#9599) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set no_sync_func & grad_sync_fucn (#9601) * Set no_sync_func & grad_sync_fucn Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * set overlap_param_sync Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * small nemo logger bug fix (#9607) Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix the dict format returned by scheduler method (#9609) Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Dataloading enhancements and bug fixes (#9595) * fix dataloading + checkpoint restore * clean up data sampler * fix typo * support passing multiple paths to data module * fix validation dataloader * fix dataloader len when using gradient accumulation * fix progress bar * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix step count in loggers * fix blended dataset * address comments * address comment * move step logging into strategy * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix serialization of AutoResume (#9616) * fix serialization of autoresume * update undefined variables Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Chat template support for megatron_gpt_eval.py (#9354) * Bump PTL version (#9557) Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [Resiliency] Straggler detection (#9473) * Initial straggler det impl Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed CI code checks Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Removed unused import Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * remove submodule Signed-off-by: Maanu Grover <maanug@nvidia.com> * Updated documentation; Updated callback params; Cosmetic changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed straggler det config; Added basic test Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixes in test_straggler_det.py Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Updated straggler callback API Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * stop_if_detected=False by default Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move model loading to separate function; call toContainer once; pad using closed formula Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * read prompts from file Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * If input prompt contains dict, apply model.tokenizer.chat_template Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * apply @Gal Leibovich's patch Taken from: https://github.com/NVIDIA/NeMo/commit/17572905344db4692583e72799d55801a8860f35 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename prompts_file to prompts_jsonl Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat_template param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add ChatTemplateMixin to SentencePieceTokenizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat-template to text-gen-strat Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move load prompts to separate file Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove chat-template from text-gen-utils Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make chat-template more generic Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add assert message Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * small refactor for chat_template_mixin Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * undo ckpt conv changes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move rounding to function Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com> Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Co-authored-by: jbieniusiewi <152396322+jbieniusiewi@users.noreply.github.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Jsonl support (#9611) * Adding support to preprocess .jsonl and .jsonl.gz files in input directory Signed-off-by: adityavavre <avavre@nvidia.com> * Adding support to preprocess .jsonl and .jsonl.gz files in input directory Signed-off-by: adityavavre <avavre@nvidia.com> * Apply isort and black reformatting Signed-off-by: adityavavre <adityavavre@users.noreply.github.com> --------- Signed-off-by: adityavavre <avavre@nvidia.com> Signed-off-by: adityavavre <adityavavre@users.noreply.github.com> Co-authored-by: adityavavre <adityavavre@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Add PEFT (#9490) * initial commit for PEFT in nemo2 * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * address comments Signed-off-by: Chen Cui <chcui@nvidia.com> * make import easier Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * address comments Signed-off-by: Chen Cui <chcui@nvidia.com> * Update nemo/collections/llm/peft/lora.py Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Some small fixes + adding more doc-strings * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding ModelTransform callback * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fixing type-hint for model_transform * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * fix import Signed-off-by: Chen Cui <chcui@nvidia.com> * model transform for gemma llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * fix model transform Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * change lora target default to all linear modules Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * Small fix in mixtral * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Integrating PEFT to the public-API + some fixes * Big refactor to allow to load adapter-states * Some fixes to support adapter_path * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Disabling ckpt reloading when adapter_path is passed * Fix CLI * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove commented-out code * Remove commented-out code * Remove un-used import * Fix callback imports * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fixing llm.pretrain * Some small fixes * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix missing import + type-hint in finetune * Adding PreemptionCallback + some more tests * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Clean up imports & clean up llm.api * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Trying to fix failing tests * Remove __init__.py 2 * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix failing test * Trying to fix last failing test --------- Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Akoumparouli/mistral import instruct chat template fix (#9567) * use bf16 by defualt mistral conv Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat template Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use capitalized role names Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Remove .cuda calls, use device isntead (#9602) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix converter defautl args (#9565) * fix converter defautl args Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * mixtral export (#9603) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix: remove non_blocking from PTL's .cuda call (#9618) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Alit/mamba tmp (#9612) * adding mamba support * fix import mixins * rm convert jamba * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * more cleanups * use GPT text gen * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * fixing gbs in TP convetor * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add reqs * add tutorial * minor fix to tutorial * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add mamba_tmp * remove mamba import * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> --------- Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com> Co-authored-by: JRD971000 <JRD971000@users.noreply.github.com> Co-authored-by: arendu <adithya.r@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * TitaNet Batch Verify Speaker (#9337) * add batch_inference for verify_speakers method Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * remove not used package Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * change batch inference logic Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * fixup Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * requested changes Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * add verify_speakers_batch to docs Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * handle None durations in manifest Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * change logging text Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * check duration presence Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * add channel_selector to dataset configs Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> --------- Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> Co-authored-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Enable MCore checkpointing optimizations (#9505) * Expose num processes in PyT Dist Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add parallel save/load optimizations from MCore Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Remove async utils from MCore Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Enable DistOpt paralell R/W Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Enable PyT Dist caching Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Small fixes Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Make sure DistCkptIO is instantiated from config Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Bump MCore version to v0.7 Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Print load strategy Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Forward MCore to model space DistOpt Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add separate flag to control DistOpt paralell R/W Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Turn off parallel save by default Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Change mixtral moe key name for trt-llm (#9620) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * change moe key values Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * add weight to the key Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix ckpt load bug (#9621) * fix ckpt load bug Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * NeVA Minor Fixes (#9608) * fix neva resume with empty param loaded for some pp stage Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix crop size check Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix pretrianing data sizes and weights (#9627) Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Alit/mamba (#9575) * adding mamba support * fix import mixins * rm convert jamba * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * more cleanups * use GPT text gen * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * fixing gbs in TP convetor * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add reqs * add tutorial * minor fix to tutorial * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * add mamba dependancies * add mcore tag * modify dockerfile ci * modify dockerfile ci --------- Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com> Co-authored-by: JRD971000 <JRD971000@users.noreply.github.com> Co-authored-by: arendu <adithya.r@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] async checkpointing support (#9466) * add async checkpointing support * fixes * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add parallel read/write support and other optimizations * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * address comments, make dist checkpointing args configurable * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * Update default sharding type Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> * Update default sharding type Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix the arguments of forward_for_export function in msdd_models (#9624) * Fix the arguments of forward_for_export function Signed-off-by: Taejin Park <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> --------- Signed-off-by: Taejin Park <tango4j@gmail.com> Signed-off-by: tango4j <tango4j@users.noreply.github.com> Co-authored-by: tango4j <tango4j@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Change default parallel_save to False (#9632) Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Unwrap ckpt_io for model opt (async save) (#9622) Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * MCore T5 support for NeMo - Training (#9432) * huvu/mcore_t5 first commit from local * removing DEBUGGING prints * cleaning megatron_lm_encoder_decoder_model.py code * cleaning code * adding Github action test * only run mcore T5 test * only run mcore T5 test * only run mcore T5 test * only run mcore T5 test * reset .github/workflows/cicd-main.yml * reset .github/workflows/cicd-main.yml * adding condition self.mcore_t5 when running self.build_transformer_config() * refractor megatron_lm_encoder_decoder_model.py to not use self.model * only run T5-related tests * remove all self.model * reset cicd file * reset cicd file * updating codes remove duplicate if/else; adding mcore/transformer_engine to config file * adjust +model.mcore_t5=True * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> --------- Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: Huy Vu2 <huvu@login-eos01.eos.clusters.nvidia.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Expose transformer_layer_spec inside GPTConfig (#9592) * Expose transformer_layer_spec inside GPTConfig * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Expose layer-specs * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update NeMo Clip to Use MCore Modules (#9594) * update clip model and config file Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update clip for mcore Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * MCore CLIP Fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix no mask Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * few neva fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip module Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * add siglip loss Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix collate fn Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip conversion script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip convert Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * clip fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * clean up script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * clip fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix code styles Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update siglip_loss.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add REST API to deploy module (#9539) * Add REST API and FastAPI to deploy module Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add NemoQuery and requirements Signed-off-by: Abhishree <abhishreetm@gmail.com> * Edit path for config.json Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add modifications for REST API for the correct functionality Move service dir under deploy Use NeMoQueryLLM instead of NemoQuery Signed-off-by: Abhishree <abhishreetm@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply isort and black reformatting Signed-off-by: pre-commit-ci[bot] <pre-commit-ci[bot]@users.noreply.github.com> * Change default port for REST Service Change default port for REST service as Triton server also used the same port as default. Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: athitten <athitten@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: pre-commit-ci[bot] <pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Signed-off-by: athitten <athitten@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: athitten <athitten@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Mistral + Mixtral Support for NeVa (#9459) * mistral template support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * get_specs neva fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fixed mistral tokenization Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * t…
BoxiangW
pushed a commit
to BoxiangW/NeMo
that referenced
this pull request
Jul 30, 2024
* Adding context- & expert-parallism to MegatronStrategy (#9525) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add CICD test for Stable Diffusion (#9464) * Add CICD test for Stable Diffusion Signed-off-by: Michal Futrega <mfutrega@nvidia.com> * Update cicd-main.yml Signed-off-by: Michal Futrega <mfutrega@nvidia.com> * Use single gpu runner Signed-off-by: Michal Futrega <mfutrega@nvidia.com> --------- Signed-off-by: Michal Futrega <mfutrega@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Akoumparouli/nemo ux mixtral (#9446) * use default collate if dataset does not have one Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mixtral config Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add convert_state Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix StateDictTransform for 2D layers, e.g. MoE Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pass num_moe_experts to specs Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * udpate MixtralModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mini docstring Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * update mcoreddp call (#9345) * update mcoreddp call Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update mcore commits Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Llama and Gemma (#9528) * add llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add llama3 Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * fix typo Signed-off-by: Chen Cui <chcui@nvidia.com> * enable importers with multiple models Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add gemma Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * checks Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] minor logging bug fixes (#9529) * minor exp_manager bug fixes * remove print statement * fix docstring * fix AppState defaults --------- Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * mcore distOpt restore fix (#9421) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Custom Tiktoken tokenizer. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fixed the tokenizer decoding on special tokens. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Apply isort and black reformatting Signed-off-by: ertkonuk <ertkonuk@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Added token_to_id() method. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update neva conversion script from and to HF (#9296) * Update NeMo script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix example scripts Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * Update convert_llava_nemo_to_hf.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * vLLM Export Support (#9381) * Export implementation for vLLM 0.4.3. Supports LLAMA2, Mistral, Mixtral (unverified), Gemma and StarCoder2 models. The nemo.export.tensorrt_llm alias was removed to avoid initializing TRT-LLM when importing anything from nemo.export. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Fixed some CodeQL warnings. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Removed empty files. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Updated the integration for vLLM 0.5.0. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Updated the vLLM deployment interface to use max_output_len instead of max_output_token. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Moved the Exporter class to nemo/export and renamed its file to vllm_exporter.py, to be more similar to TRT-LLM. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Implemented vLLM support in the export tests, added functional testing, implemented forward evaluation on vLLM without Triton. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Moved the vLLM deployment functionality to the common deploy_triton.py script. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Fixed the CodeQL discovered issues. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Fixed one more return of a wrong dimensionality... Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * More wrong dimensionality returns. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> --------- Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * PL: Delete precision if using plugin. TODO switch to MegatronTrainerBuilder (#9535) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add page context fmha (#9526) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * extend get_gpt_layer_modelopt_spec to support MoE (#9532) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix mock data generation for legacy dataset (#9530) Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] IO fixes (#9512) * Improve IOMixin.io_transform_args to handle dataclasses better * Dump task json + img inside NeMoLogger * Adding store_io to train task * Update opt.connect to also propagate to __io__ * Rename opt to optim for consistency * Moving to using safe serialization using fiddle, only use cloudpickle when needed * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Using Config from fiddle instead of sdk for now * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move enable_nemo_ckpt_io from MegatronStrategy to ModelCheckpoint * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move nemo-ckpt to _get_finalize_save_checkpoint_callback * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Update TrainerContext & io.load_ckpt * Use renamed TrainerContext inside ModelCheckpoint * Remove double io saving * Rename lightning.pytorch.opt -> optim * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove store_io from train-task * Adding fiddle-extension for torch * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move fdl_torch import * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding dtype to serialization * Some fixes * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Make TransformerConfig inherit from IOMixin to fix serialization error * Make TransformerConfig inherit from IOMixin to fix serialization error * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Add support for BuiltinFunctionType * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Add missing import * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix dataclass fields --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Test C++ runtime on demand in nemo_export.py to avoid possible OOMs (#9544) * Add test_cpp_runtime flag Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Apply isort and black reformatting Signed-off-by: janekl <janekl@users.noreply.github.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: janekl <janekl@users.noreply.github.com> Co-authored-by: janekl <janekl@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix lhotse tests for v1.24.2 (#9546) * Fix lhotse tests for v1.24.0 Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Fix RIR test Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * gpu_unitTests_notOptional (#9551) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * add reset learning rate functionality (#9372) * add reset_lr functionality Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix reset_lr logic Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * move reset_lr from optim section Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * add reset_lr value to config Signed-off-by: dimapihtar <dpihtar@gmail.com> * set reset_lr False by default Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove extra line Signed-off-by: dimapihtar <dpihtar@gmail.com> * add reset_lr test Signed-off-by: dimapihtar <dpihtar@gmail.com> * add reset_lr test Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove extra quote Signed-off-by: dimapihtar <dpihtar@gmail.com> * add ability to reset schedule's max_steps and decay_steps Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * change scheduler's first step logic when using reset_lr Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix reset_lr logic Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * update reset_lr comments Signed-off-by: dimapihtar <dpihtar@gmail.com> * add use cases for reset_lr feature Signed-off-by: dimapihtar <dpihtar@gmail.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add Python AIStore SDK to container and bump min Lhotse version (#9537) * Add Python AIStore SDK to requirements and bump min Lhotse version Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Move AIStore Python SDK to Dockerfile, remove matplotlib/ipywidgets deps Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export() (#9147) * Ininial WARs to implement dynamo option for export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * including weights in .onnx Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * dynamo_export works for many small models Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * External weights behaviour fixed Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Cleanup Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Apply isort and black reformatting Signed-off-by: borisfom <borisfom@users.noreply.github.com> * print cleaned up Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Added overloadable dynamic_shapes_for_export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Addressing code review Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Fixing CI issues Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Fixing CI test failure Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Eliminated test cross-contamination Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Signed-off-by: borisfom <borisfom@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Fix tokenizer IO (#9555) * Adding tokenizer to io-test + making it pass * Handling tokenizer correctly inside dump_io * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Removing not used import --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo UX] Move mistral_7b.py to mistral.py (#9545) * Move mistral_7b.py to mistral.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename MixtralConfig to MixtralConfig8x7B Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mistral rename: mistralconfig7b & mistralmodel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Use closed-formula to round by multiple (#9307) * Use closed-formula to round by multiple Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * ci: Do not attempt to send slack on fork (#9556) * ci: Do not attempt to send slack on fork Signed-off-by: Oliver Koenig <okoenig@nvidia.com> * test Signed-off-by: Oliver Koenig <okoenig@nvidia.com> --------- Signed-off-by: Oliver Koenig <okoenig@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix nemo export test (#9547) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix export test Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix SDXL incorrect name in docs (#9534) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * GPU unit tests: Mark flaky tests to be fixed (#9559) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Bump PTL version (#9557) Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Resiliency] Straggler detection (#9473) * Initial straggler det impl Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed CI code checks Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Removed unused import Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * remove submodule Signed-off-by: Maanu Grover <maanug@nvidia.com> * Updated documentation; Updated callback params; Cosmetic changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed straggler det config; Added basic test Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixes in test_straggler_det.py Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Updated straggler callback API Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * stop_if_detected=False by default Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * switch to torch_dist as default dist checkpointing backend (#9541) Signed-off-by: ashors1 <ashors@nvidia.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Checkpointing bug fixes (#9562) * fix checkpoint loading * fix * fixes * another fix * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add tps and pps params to the export script (#9558) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix export test Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * remove n_gpus param Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * add and fix parameters Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix deploy script Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * rename tps and pps params Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Consolidate gpt continue training script into pretraining script (#9413) * Consolidate gpt continue training with pretraining Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix default config Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add github action cicd Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * extract _integrate_original_checkpoint_data as a method Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix getattr Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Revert "Add github action cicd" This reverts commit a453f16ba2be6413db932623009da893208acdd5. * Update comments in nlp_overrides.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add support to change Multi task model prompt (#9542) * Add support to change Multi task model prompt Signed-off-by: smajumdar <titu1994@gmail.com> * Add support to change Multi task model prompt Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Update nemo/collections/common/prompts/formatter.py Co-authored-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Somshubra Majumdar <titu1994@gmail.com> * Address comments Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Address comments Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Signed-off-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add Multimodal Exporter (#9256) * Add video-neva TRT export * Add TRT inference * Change config * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change export params * Remove unused import * Add neva export * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change unpack nemo * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Add trt infer config * Fix neva trt inference * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Add exporter * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix infer * Add PyTriton * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix deploy wrong dim * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change to pass PIL Image * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix video neva deploy * Change query * Change deploy * Remove unused import * Change ptuning * Change to mm exporter * Add script * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix script --------- Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> Co-authored-by: meatybobby <meatybobby@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Enable encoder adapters for Canary and MultiTaskAED models (#9409) * Fix assertions for adapter types Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Cleanup Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Finalize support for decoder adapters Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * fix the freeze/unfreeze problem by replacing as_frozen with torch.inference_mode * Apply isort and black reformatting Signed-off-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> * Update tests to new generic way of module update Signed-off-by: smajumdar <titu1994@gmail.com> * Finalize code for update module Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Fix variable name Signed-off-by: smajumdar <titu1994@gmail.com> * Finalize projection support for transformer mha adapters Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Correct implementation of freeze restore Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Corrects the implementation of replace_adapter_modules to limit to just the top level modules Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Remove registration of Transformer MHA Signed-off-by: smajumdar <titu1994@gmail.com> * Remove registration of Transformer MHA Signed-off-by: smajumdar <titu1994@gmail.com> * Address reviewer comments Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Signed-off-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> Co-authored-by: Weiqing Wang <weiqingw@nvidia.com> Co-authored-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * pass option through (#9570) Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * PTQ refinements (#9574) * Rename megatron_gpt_quantization -> megatron_gpt_ptq Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Configure export.save_path as dir or tarball Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * PTQ docs update Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Make model_type optional in case of quantized checkpoints Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Drop unused save_nemo_model_config argument Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Audio model collection (#9263) * Audio model collection Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Apply isort and black reformatting Signed-off-by: anteju <anteju@users.noreply.github.com> * Fix imports Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Addressed PR comments Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Apply isort and black reformatting Signed-off-by: anteju <anteju@users.noreply.github.com> --------- Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: anteju <anteju@users.noreply.github.com> Co-authored-by: anteju <anteju@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Fix Trainer serialization (#9571) * Fix Trainer serialization * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update click version requirement (#9580) Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> Co-authored-by: Dong Hyuk Chang <donghyukc@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Fault tolerance] Heartbeat detection (#9352) * Fault tolerance related changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Cosmetic changes in documentation Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Doc update round2 Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: jbieniusiewi <152396322+jbieniusiewi@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add ModelOpt QAT example for Llama2 SFT model (#9326) * add INT4 QAT example for Llama2 SFT model Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * Add config parameter to control kv cache quantization Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * Fix typo in cicd-main.yml for QAT test Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * fix nlp_overrides.py Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * address reviewer feedback Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * quantize unwrapped model Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * add compress export argument for qat config Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set TE flag in legacy -> mcore conversion script (#9585) * set TE flag Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Add fabric-API for manual forward-pass (#9577) * First pass over fabric-API * Adding Trainer -> Fabric conversion * Some small fixes to get a forward-pass in Fabric working * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding doc-string to Fabric.import_model * Adding track_io to io_init of Fabric * Fix Fabric.load_model + add doc-string * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove unused import * Some small fixes * Fix failing test --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Add SDK-factories to llm-collection (#9589) * Adding sdk-factories to llm-collection * Removing _model from mistral + mixtral * Expose lr_scheduler inside lightning * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Multimodal projection layer adapter fix for PP>1 (#9445) * enabling multimodal adapters to load in PP>1 Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * parameterizing validate_access_integrity, set to false when PP>1 Signed-off-by: paul-gibbons <paul@gibbonspaul.com> formatting fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * update nlp_model.py Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * update modelPT with validate_access_integrity Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * updating save_restore_connector w/ validate_access_integrity Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * addressing comment Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * adding validate_access_integrity to super().load_config_and_state_dict() Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * testing reorder of validate_access_integrity for CI failures Signed-off-by: paul-gibbons <paul@gibbonspaul.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add offline quantization script for QLoRA deployment (#9455) * add qlora offline quantization script Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * clean Signed-off-by: Chen Cui <chcui@nvidia.com> * docstring Signed-off-by: Chen Cui <chcui@nvidia.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * qlora support more models (#9488) Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Some improvements to NeMoLogger (#9591) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set n_gpu to None in nemo export (#9593) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * set ngpus to None Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Inflight nemo model export support (#9527) * online model conversion and refit Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * clean code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * add refit, cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * combine weight conversion functions Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * Apply isort and black reformatting Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> * remove debug print Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * fix single gpu and cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * Apply isort and black reformatting Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> --------- Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * vLLM Export Improvements (#9596) * Separated the vLLM export functionality from the common deployment script into deploy_vllm_triton.py. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Fixed vocab_size for LLAMA3. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Export test: fixed deployment testing w/o Megatron, made functional tests optional, added --gpu_memory_utilization. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Addressing review and CodeQL comments. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> --------- Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set finalize_model_grads_func in on_fit_start instead to make sure it's being called (#9599) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set no_sync_func & grad_sync_fucn (#9601) * Set no_sync_func & grad_sync_fucn Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * set overlap_param_sync Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * small nemo logger bug fix (#9607) Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix the dict format returned by scheduler method (#9609) Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Dataloading enhancements and bug fixes (#9595) * fix dataloading + checkpoint restore * clean up data sampler * fix typo * support passing multiple paths to data module * fix validation dataloader * fix dataloader len when using gradient accumulation * fix progress bar * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix step count in loggers * fix blended dataset * address comments * address comment * move step logging into strategy * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix serialization of AutoResume (#9616) * fix serialization of autoresume * update undefined variables Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Chat template support for megatron_gpt_eval.py (#9354) * Bump PTL version (#9557) Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [Resiliency] Straggler detection (#9473) * Initial straggler det impl Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed CI code checks Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Removed unused import Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * remove submodule Signed-off-by: Maanu Grover <maanug@nvidia.com> * Updated documentation; Updated callback params; Cosmetic changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed straggler det config; Added basic test Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixes in test_straggler_det.py Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Updated straggler callback API Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * stop_if_detected=False by default Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move model loading to separate function; call toContainer once; pad using closed formula Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * read prompts from file Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * If input prompt contains dict, apply model.tokenizer.chat_template Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * apply @Gal Leibovich's patch Taken from: https://github.com/NVIDIA/NeMo/commit/17572905344db4692583e72799d55801a8860f35 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename prompts_file to prompts_jsonl Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat_template param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add ChatTemplateMixin to SentencePieceTokenizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat-template to text-gen-strat Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move load prompts to separate file Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove chat-template from text-gen-utils Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make chat-template more generic Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add assert message Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * small refactor for chat_template_mixin Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * undo ckpt conv changes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move rounding to function Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com> Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Co-authored-by: jbieniusiewi <152396322+jbieniusiewi@users.noreply.github.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Jsonl support (#9611) * Adding support to preprocess .jsonl and .jsonl.gz files in input directory Signed-off-by: adityavavre <avavre@nvidia.com> * Adding support to preprocess .jsonl and .jsonl.gz files in input directory Signed-off-by: adityavavre <avavre@nvidia.com> * Apply isort and black reformatting Signed-off-by: adityavavre <adityavavre@users.noreply.github.com> --------- Signed-off-by: adityavavre <avavre@nvidia.com> Signed-off-by: adityavavre <adityavavre@users.noreply.github.com> Co-authored-by: adityavavre <adityavavre@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Add PEFT (#9490) * initial commit for PEFT in nemo2 * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * address comments Signed-off-by: Chen Cui <chcui@nvidia.com> * make import easier Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * address comments Signed-off-by: Chen Cui <chcui@nvidia.com> * Update nemo/collections/llm/peft/lora.py Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Some small fixes + adding more doc-strings * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding ModelTransform callback * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fixing type-hint for model_transform * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * fix import Signed-off-by: Chen Cui <chcui@nvidia.com> * model transform for gemma llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * fix model transform Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * change lora target default to all linear modules Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * Small fix in mixtral * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Integrating PEFT to the public-API + some fixes * Big refactor to allow to load adapter-states * Some fixes to support adapter_path * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Disabling ckpt reloading when adapter_path is passed * Fix CLI * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove commented-out code * Remove commented-out code * Remove un-used import * Fix callback imports * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fixing llm.pretrain * Some small fixes * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix missing import + type-hint in finetune * Adding PreemptionCallback + some more tests * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Clean up imports & clean up llm.api * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Trying to fix failing tests * Remove __init__.py 2 * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix failing test * Trying to fix last failing test --------- Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Akoumparouli/mistral import instruct chat template fix (#9567) * use bf16 by defualt mistral conv Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat template Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use capitalized role names Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Remove .cuda calls, use device isntead (#9602) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix converter defautl args (#9565) * fix converter defautl args Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * mixtral export (#9603) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix: remove non_blocking from PTL's .cuda call (#9618) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Alit/mamba tmp (#9612) * adding mamba support * fix import mixins * rm convert jamba * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * more cleanups * use GPT text gen * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * fixing gbs in TP convetor * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add reqs * add tutorial * minor fix to tutorial * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add mamba_tmp * remove mamba import * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> --------- Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com> Co-authored-by: JRD971000 <JRD971000@users.noreply.github.com> Co-authored-by: arendu <adithya.r@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * TitaNet Batch Verify Speaker (#9337) * add batch_inference for verify_speakers method Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * remove not used package Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * change batch inference logic Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * fixup Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * requested changes Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * add verify_speakers_batch to docs Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * handle None durations in manifest Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * change logging text Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * check duration presence Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * add channel_selector to dataset configs Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> --------- Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> Co-authored-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Enable MCore checkpointing optimizations (#9505) * Expose num processes in PyT Dist Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add parallel save/load optimizations from MCore Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Remove async utils from MCore Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Enable DistOpt paralell R/W Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Enable PyT Dist caching Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Small fixes Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Make sure DistCkptIO is instantiated from config Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Bump MCore version to v0.7 Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Print load strategy Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Forward MCore to model space DistOpt Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add separate flag to control DistOpt paralell R/W Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Turn off parallel save by default Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Change mixtral moe key name for trt-llm (#9620) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * change moe key values Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * add weight to the key Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix ckpt load bug (#9621) * fix ckpt load bug Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * NeVA Minor Fixes (#9608) * fix neva resume with empty param loaded for some pp stage Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix crop size check Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix pretrianing data sizes and weights (#9627) Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Alit/mamba (#9575) * adding mamba support * fix import mixins * rm convert jamba * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * more cleanups * use GPT text gen * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * fixing gbs in TP convetor * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add reqs * add tutorial * minor fix to tutorial * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * add mamba dependancies * add mcore tag * modify dockerfile ci * modify dockerfile ci --------- Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com> Co-authored-by: JRD971000 <JRD971000@users.noreply.github.com> Co-authored-by: arendu <adithya.r@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] async checkpointing support (#9466) * add async checkpointing support * fixes * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add parallel read/write support and other optimizations * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * address comments, make dist checkpointing args configurable * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * Update default sharding type Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> * Update default sharding type Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix the arguments of forward_for_export function in msdd_models (#9624) * Fix the arguments of forward_for_export function Signed-off-by: Taejin Park <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> --------- Signed-off-by: Taejin Park <tango4j@gmail.com> Signed-off-by: tango4j <tango4j@users.noreply.github.com> Co-authored-by: tango4j <tango4j@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Change default parallel_save to False (#9632) Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Unwrap ckpt_io for model opt (async save) (#9622) Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * MCore T5 support for NeMo - Training (#9432) * huvu/mcore_t5 first commit from local * removing DEBUGGING prints * cleaning megatron_lm_encoder_decoder_model.py code * cleaning code * adding Github action test * only run mcore T5 test * only run mcore T5 test * only run mcore T5 test * only run mcore T5 test * reset .github/workflows/cicd-main.yml * reset .github/workflows/cicd-main.yml * adding condition self.mcore_t5 when running self.build_transformer_config() * refractor megatron_lm_encoder_decoder_model.py to not use self.model * only run T5-related tests * remove all self.model * reset cicd file * reset cicd file * updating codes remove duplicate if/else; adding mcore/transformer_engine to config file * adjust +model.mcore_t5=True * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> --------- Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: Huy Vu2 <huvu@login-eos01.eos.clusters.nvidia.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Expose transformer_layer_spec inside GPTConfig (#9592) * Expose transformer_layer_spec inside GPTConfig * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Expose layer-specs * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update NeMo Clip to Use MCore Modules (#9594) * update clip model and config file Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update clip for mcore Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * MCore CLIP Fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix no mask Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * few neva fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip module Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * add siglip loss Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix collate fn Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip conversion script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip convert Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * clip fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * clean up script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * clip fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix code styles Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update siglip_loss.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add REST API to deploy module (#9539) * Add REST API and FastAPI to deploy module Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add NemoQuery and requirements Signed-off-by: Abhishree <abhishreetm@gmail.com> * Edit path for config.json Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add modifications for REST API for the correct functionality Move service dir under deploy Use NeMoQueryLLM instead of NemoQuery Signed-off-by: Abhishree <abhishreetm@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply isort and black reformatting Signed-off-by: pre-commit-ci[bot] <pre-commit-ci[bot]@users.noreply.github.com> * Change default port for REST Service Change default port for REST service as Triton server also used the same port as default. Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: athitten <athitten@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: pre-commit-ci[bot] <pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Signed-off-by: athitten <athitten@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: athitten <athitten@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Mistral + Mixtral Support for NeVa (#9459) * mistral template support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * get_specs neva fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fixed mistral tokenization Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * t…
tonyjie
pushed a commit
to tonyjie/NeMo
that referenced
this pull request
Aug 6, 2024
* mistral template support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * get_specs neva fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fixed mistral tokenization Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * text_gen_strategy add mistral support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral text_gen fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Cleaning up neva config Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fix llama_2 default text_gen_strategy Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * fix forward() to account for new embedding optimization in MCore Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Signed-off-by: tonyjie <jl4257@cornell.edu>
gshennvm
pushed a commit
that referenced
this pull request
Oct 2, 2024
* Adding context- & expert-parallism to MegatronStrategy (#9525) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add CICD test for Stable Diffusion (#9464) * Add CICD test for Stable Diffusion Signed-off-by: Michal Futrega <mfutrega@nvidia.com> * Update cicd-main.yml Signed-off-by: Michal Futrega <mfutrega@nvidia.com> * Use single gpu runner Signed-off-by: Michal Futrega <mfutrega@nvidia.com> --------- Signed-off-by: Michal Futrega <mfutrega@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Akoumparouli/nemo ux mixtral (#9446) * use default collate if dataset does not have one Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mixtral config Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add convert_state Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix StateDictTransform for 2D layers, e.g. MoE Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pass num_moe_experts to specs Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * udpate MixtralModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mini docstring Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * update mcoreddp call (#9345) * update mcoreddp call Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update mcore commits Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Llama and Gemma (#9528) * add llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add llama3 Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * fix typo Signed-off-by: Chen Cui <chcui@nvidia.com> * enable importers with multiple models Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add gemma Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * checks Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] minor logging bug fixes (#9529) * minor exp_manager bug fixes * remove print statement * fix docstring * fix AppState defaults --------- Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * mcore distOpt restore fix (#9421) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Custom Tiktoken tokenizer. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fixed the tokenizer decoding on special tokens. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Apply isort and black reformatting Signed-off-by: ertkonuk <ertkonuk@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Added token_to_id() method. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update neva conversion script from and to HF (#9296) * Update NeMo script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix example scripts Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * Update convert_llava_nemo_to_hf.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * vLLM Export Support (#9381) * Export implementation for vLLM 0.4.3. Supports LLAMA2, Mistral, Mixtral (unverified), Gemma and StarCoder2 models. The nemo.export.tensorrt_llm alias was removed to avoid initializing TRT-LLM when importing anything from nemo.export. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Fixed some CodeQL warnings. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Removed empty files. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Updated the integration for vLLM 0.5.0. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Updated the vLLM deployment interface to use max_output_len instead of max_output_token. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Moved the Exporter class to nemo/export and renamed its file to vllm_exporter.py, to be more similar to TRT-LLM. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Implemented vLLM support in the export tests, added functional testing, implemented forward evaluation on vLLM without Triton. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Moved the vLLM deployment functionality to the common deploy_triton.py script. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Fixed the CodeQL discovered issues. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Fixed one more return of a wrong dimensionality... Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * More wrong dimensionality returns. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> --------- Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * PL: Delete precision if using plugin. TODO switch to MegatronTrainerBuilder (#9535) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add page context fmha (#9526) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * extend get_gpt_layer_modelopt_spec to support MoE (#9532) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix mock data generation for legacy dataset (#9530) Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] IO fixes (#9512) * Improve IOMixin.io_transform_args to handle dataclasses better * Dump task json + img inside NeMoLogger * Adding store_io to train task * Update opt.connect to also propagate to __io__ * Rename opt to optim for consistency * Moving to using safe serialization using fiddle, only use cloudpickle when needed * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Using Config from fiddle instead of sdk for now * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move enable_nemo_ckpt_io from MegatronStrategy to ModelCheckpoint * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move nemo-ckpt to _get_finalize_save_checkpoint_callback * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Update TrainerContext & io.load_ckpt * Use renamed TrainerContext inside ModelCheckpoint * Remove double io saving * Rename lightning.pytorch.opt -> optim * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove store_io from train-task * Adding fiddle-extension for torch * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move fdl_torch import * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding dtype to serialization * Some fixes * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Make TransformerConfig inherit from IOMixin to fix serialization error * Make TransformerConfig inherit from IOMixin to fix serialization error * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Add support for BuiltinFunctionType * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Add missing import * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix dataclass fields --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Test C++ runtime on demand in nemo_export.py to avoid possible OOMs (#9544) * Add test_cpp_runtime flag Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Apply isort and black reformatting Signed-off-by: janekl <janekl@users.noreply.github.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: janekl <janekl@users.noreply.github.com> Co-authored-by: janekl <janekl@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix lhotse tests for v1.24.2 (#9546) * Fix lhotse tests for v1.24.0 Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Fix RIR test Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * gpu_unitTests_notOptional (#9551) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * add reset learning rate functionality (#9372) * add reset_lr functionality Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix reset_lr logic Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * move reset_lr from optim section Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * add reset_lr value to config Signed-off-by: dimapihtar <dpihtar@gmail.com> * set reset_lr False by default Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove extra line Signed-off-by: dimapihtar <dpihtar@gmail.com> * add reset_lr test Signed-off-by: dimapihtar <dpihtar@gmail.com> * add reset_lr test Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove extra quote Signed-off-by: dimapihtar <dpihtar@gmail.com> * add ability to reset schedule's max_steps and decay_steps Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * change scheduler's first step logic when using reset_lr Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix reset_lr logic Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * update reset_lr comments Signed-off-by: dimapihtar <dpihtar@gmail.com> * add use cases for reset_lr feature Signed-off-by: dimapihtar <dpihtar@gmail.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add Python AIStore SDK to container and bump min Lhotse version (#9537) * Add Python AIStore SDK to requirements and bump min Lhotse version Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Move AIStore Python SDK to Dockerfile, remove matplotlib/ipywidgets deps Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export() (#9147) * Ininial WARs to implement dynamo option for export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * including weights in .onnx Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * dynamo_export works for many small models Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * External weights behaviour fixed Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Cleanup Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Apply isort and black reformatting Signed-off-by: borisfom <borisfom@users.noreply.github.com> * print cleaned up Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Added overloadable dynamic_shapes_for_export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Addressing code review Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Fixing CI issues Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Fixing CI test failure Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Eliminated test cross-contamination Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Signed-off-by: borisfom <borisfom@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Fix tokenizer IO (#9555) * Adding tokenizer to io-test + making it pass * Handling tokenizer correctly inside dump_io * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Removing not used import --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo UX] Move mistral_7b.py to mistral.py (#9545) * Move mistral_7b.py to mistral.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename MixtralConfig to MixtralConfig8x7B Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mistral rename: mistralconfig7b & mistralmodel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Use closed-formula to round by multiple (#9307) * Use closed-formula to round by multiple Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * ci: Do not attempt to send slack on fork (#9556) * ci: Do not attempt to send slack on fork Signed-off-by: Oliver Koenig <okoenig@nvidia.com> * test Signed-off-by: Oliver Koenig <okoenig@nvidia.com> --------- Signed-off-by: Oliver Koenig <okoenig@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix nemo export test (#9547) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix export test Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix SDXL incorrect name in docs (#9534) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * GPU unit tests: Mark flaky tests to be fixed (#9559) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Bump PTL version (#9557) Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Resiliency] Straggler detection (#9473) * Initial straggler det impl Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed CI code checks Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Removed unused import Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * remove submodule Signed-off-by: Maanu Grover <maanug@nvidia.com> * Updated documentation; Updated callback params; Cosmetic changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed straggler det config; Added basic test Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixes in test_straggler_det.py Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Updated straggler callback API Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * stop_if_detected=False by default Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * switch to torch_dist as default dist checkpointing backend (#9541) Signed-off-by: ashors1 <ashors@nvidia.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Checkpointing bug fixes (#9562) * fix checkpoint loading * fix * fixes * another fix * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add tps and pps params to the export script (#9558) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix export test Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * remove n_gpus param Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * add and fix parameters Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix deploy script Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * rename tps and pps params Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Consolidate gpt continue training script into pretraining script (#9413) * Consolidate gpt continue training with pretraining Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix default config Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add github action cicd Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * extract _integrate_original_checkpoint_data as a method Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix getattr Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Revert "Add github action cicd" This reverts commit a453f16ba2be6413db932623009da893208acdd5. * Update comments in nlp_overrides.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add support to change Multi task model prompt (#9542) * Add support to change Multi task model prompt Signed-off-by: smajumdar <titu1994@gmail.com> * Add support to change Multi task model prompt Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Update nemo/collections/common/prompts/formatter.py Co-authored-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Somshubra Majumdar <titu1994@gmail.com> * Address comments Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Address comments Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Signed-off-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add Multimodal Exporter (#9256) * Add video-neva TRT export * Add TRT inference * Change config * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change export params * Remove unused import * Add neva export * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change unpack nemo * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Add trt infer config * Fix neva trt inference * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Add exporter * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix infer * Add PyTriton * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix deploy wrong dim * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change to pass PIL Image * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix video neva deploy * Change query * Change deploy * Remove unused import * Change ptuning * Change to mm exporter * Add script * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix script --------- Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> Co-authored-by: meatybobby <meatybobby@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Enable encoder adapters for Canary and MultiTaskAED models (#9409) * Fix assertions for adapter types Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Cleanup Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Finalize support for decoder adapters Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * fix the freeze/unfreeze problem by replacing as_frozen with torch.inference_mode * Apply isort and black reformatting Signed-off-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> * Update tests to new generic way of module update Signed-off-by: smajumdar <titu1994@gmail.com> * Finalize code for update module Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Fix variable name Signed-off-by: smajumdar <titu1994@gmail.com> * Finalize projection support for transformer mha adapters Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Correct implementation of freeze restore Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Corrects the implementation of replace_adapter_modules to limit to just the top level modules Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Remove registration of Transformer MHA Signed-off-by: smajumdar <titu1994@gmail.com> * Remove registration of Transformer MHA Signed-off-by: smajumdar <titu1994@gmail.com> * Address reviewer comments Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Signed-off-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> Co-authored-by: Weiqing Wang <weiqingw@nvidia.com> Co-authored-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * pass option through (#9570) Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * PTQ refinements (#9574) * Rename megatron_gpt_quantization -> megatron_gpt_ptq Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Configure export.save_path as dir or tarball Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * PTQ docs update Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Make model_type optional in case of quantized checkpoints Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Drop unused save_nemo_model_config argument Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Audio model collection (#9263) * Audio model collection Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Apply isort and black reformatting Signed-off-by: anteju <anteju@users.noreply.github.com> * Fix imports Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Addressed PR comments Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Apply isort and black reformatting Signed-off-by: anteju <anteju@users.noreply.github.com> --------- Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: anteju <anteju@users.noreply.github.com> Co-authored-by: anteju <anteju@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Fix Trainer serialization (#9571) * Fix Trainer serialization * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update click version requirement (#9580) Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> Co-authored-by: Dong Hyuk Chang <donghyukc@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Fault tolerance] Heartbeat detection (#9352) * Fault tolerance related changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Cosmetic changes in documentation Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Doc update round2 Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: jbieniusiewi <152396322+jbieniusiewi@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add ModelOpt QAT example for Llama2 SFT model (#9326) * add INT4 QAT example for Llama2 SFT model Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * Add config parameter to control kv cache quantization Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * Fix typo in cicd-main.yml for QAT test Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * fix nlp_overrides.py Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * address reviewer feedback Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * quantize unwrapped model Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * add compress export argument for qat config Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set TE flag in legacy -> mcore conversion script (#9585) * set TE flag Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Add fabric-API for manual forward-pass (#9577) * First pass over fabric-API * Adding Trainer -> Fabric conversion * Some small fixes to get a forward-pass in Fabric working * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding doc-string to Fabric.import_model * Adding track_io to io_init of Fabric * Fix Fabric.load_model + add doc-string * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove unused import * Some small fixes * Fix failing test --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Add SDK-factories to llm-collection (#9589) * Adding sdk-factories to llm-collection * Removing _model from mistral + mixtral * Expose lr_scheduler inside lightning * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Multimodal projection layer adapter fix for PP>1 (#9445) * enabling multimodal adapters to load in PP>1 Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * parameterizing validate_access_integrity, set to false when PP>1 Signed-off-by: paul-gibbons <paul@gibbonspaul.com> formatting fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * update nlp_model.py Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * update modelPT with validate_access_integrity Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * updating save_restore_connector w/ validate_access_integrity Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * addressing comment Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * adding validate_access_integrity to super().load_config_and_state_dict() Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * testing reorder of validate_access_integrity for CI failures Signed-off-by: paul-gibbons <paul@gibbonspaul.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add offline quantization script for QLoRA deployment (#9455) * add qlora offline quantization script Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * clean Signed-off-by: Chen Cui <chcui@nvidia.com> * docstring Signed-off-by: Chen Cui <chcui@nvidia.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * qlora support more models (#9488) Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Some improvements to NeMoLogger (#9591) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set n_gpu to None in nemo export (#9593) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * set ngpus to None Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Inflight nemo model export support (#9527) * online model conversion and refit Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * clean code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * add refit, cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * combine weight conversion functions Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * Apply isort and black reformatting Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> * remove debug print Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * fix single gpu and cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * Apply isort and black reformatting Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> --------- Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * vLLM Export Improvements (#9596) * Separated the vLLM export functionality from the common deployment script into deploy_vllm_triton.py. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Fixed vocab_size for LLAMA3. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Export test: fixed deployment testing w/o Megatron, made functional tests optional, added --gpu_memory_utilization. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Addressing review and CodeQL comments. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> --------- Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set finalize_model_grads_func in on_fit_start instead to make sure it's being called (#9599) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set no_sync_func & grad_sync_fucn (#9601) * Set no_sync_func & grad_sync_fucn Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * set overlap_param_sync Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * small nemo logger bug fix (#9607) Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix the dict format returned by scheduler method (#9609) Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Dataloading enhancements and bug fixes (#9595) * fix dataloading + checkpoint restore * clean up data sampler * fix typo * support passing multiple paths to data module * fix validation dataloader * fix dataloader len when using gradient accumulation * fix progress bar * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix step count in loggers * fix blended dataset * address comments * address comment * move step logging into strategy * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix serialization of AutoResume (#9616) * fix serialization of autoresume * update undefined variables Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Chat template support for megatron_gpt_eval.py (#9354) * Bump PTL version (#9557) Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [Resiliency] Straggler detection (#9473) * Initial straggler det impl Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed CI code checks Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Removed unused import Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * remove submodule Signed-off-by: Maanu Grover <maanug@nvidia.com> * Updated documentation; Updated callback params; Cosmetic changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed straggler det config; Added basic test Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixes in test_straggler_det.py Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Updated straggler callback API Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * stop_if_detected=False by default Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move model loading to separate function; call toContainer once; pad using closed formula Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * read prompts from file Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * If input prompt contains dict, apply model.tokenizer.chat_template Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * apply @Gal Leibovich's patch Taken from: https://github.com/NVIDIA/NeMo/commit/17572905344db4692583e72799d55801a8860f35 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename prompts_file to prompts_jsonl Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat_template param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add ChatTemplateMixin to SentencePieceTokenizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat-template to text-gen-strat Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move load prompts to separate file Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove chat-template from text-gen-utils Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make chat-template more generic Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add assert message Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * small refactor for chat_template_mixin Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * undo ckpt conv changes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move rounding to function Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com> Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Co-authored-by: jbieniusiewi <152396322+jbieniusiewi@users.noreply.github.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Jsonl support (#9611) * Adding support to preprocess .jsonl and .jsonl.gz files in input directory Signed-off-by: adityavavre <avavre@nvidia.com> * Adding support to preprocess .jsonl and .jsonl.gz files in input directory Signed-off-by: adityavavre <avavre@nvidia.com> * Apply isort and black reformatting Signed-off-by: adityavavre <adityavavre@users.noreply.github.com> --------- Signed-off-by: adityavavre <avavre@nvidia.com> Signed-off-by: adityavavre <adityavavre@users.noreply.github.com> Co-authored-by: adityavavre <adityavavre@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Add PEFT (#9490) * initial commit for PEFT in nemo2 * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * address comments Signed-off-by: Chen Cui <chcui@nvidia.com> * make import easier Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * address comments Signed-off-by: Chen Cui <chcui@nvidia.com> * Update nemo/collections/llm/peft/lora.py Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Some small fixes + adding more doc-strings * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding ModelTransform callback * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fixing type-hint for model_transform * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * fix import Signed-off-by: Chen Cui <chcui@nvidia.com> * model transform for gemma llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * fix model transform Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * change lora target default to all linear modules Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * Small fix in mixtral * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Integrating PEFT to the public-API + some fixes * Big refactor to allow to load adapter-states * Some fixes to support adapter_path * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Disabling ckpt reloading when adapter_path is passed * Fix CLI * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove commented-out code * Remove commented-out code * Remove un-used import * Fix callback imports * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fixing llm.pretrain * Some small fixes * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix missing import + type-hint in finetune * Adding PreemptionCallback + some more tests * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Clean up imports & clean up llm.api * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Trying to fix failing tests * Remove __init__.py 2 * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix failing test * Trying to fix last failing test --------- Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Akoumparouli/mistral import instruct chat template fix (#9567) * use bf16 by defualt mistral conv Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat template Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use capitalized role names Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Remove .cuda calls, use device isntead (#9602) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix converter defautl args (#9565) * fix converter defautl args Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * mixtral export (#9603) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix: remove non_blocking from PTL's .cuda call (#9618) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Alit/mamba tmp (#9612) * adding mamba support * fix import mixins * rm convert jamba * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * more cleanups * use GPT text gen * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * fixing gbs in TP convetor * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add reqs * add tutorial * minor fix to tutorial * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add mamba_tmp * remove mamba import * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> --------- Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com> Co-authored-by: JRD971000 <JRD971000@users.noreply.github.com> Co-authored-by: arendu <adithya.r@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * TitaNet Batch Verify Speaker (#9337) * add batch_inference for verify_speakers method Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * remove not used package Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * change batch inference logic Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * fixup Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * requested changes Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * add verify_speakers_batch to docs Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * handle None durations in manifest Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * change logging text Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * check duration presence Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * add channel_selector to dataset configs Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> --------- Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> Co-authored-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Enable MCore checkpointing optimizations (#9505) * Expose num processes in PyT Dist Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add parallel save/load optimizations from MCore Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Remove async utils from MCore Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Enable DistOpt paralell R/W Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Enable PyT Dist caching Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Small fixes Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Make sure DistCkptIO is instantiated from config Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Bump MCore version to v0.7 Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Print load strategy Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Forward MCore to model space DistOpt Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add separate flag to control DistOpt paralell R/W Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Turn off parallel save by default Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Change mixtral moe key name for trt-llm (#9620) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * change moe key values Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * add weight to the key Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix ckpt load bug (#9621) * fix ckpt load bug Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * NeVA Minor Fixes (#9608) * fix neva resume with empty param loaded for some pp stage Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix crop size check Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix pretrianing data sizes and weights (#9627) Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Alit/mamba (#9575) * adding mamba support * fix import mixins * rm convert jamba * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * more cleanups * use GPT text gen * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * fixing gbs in TP convetor * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add reqs * add tutorial * minor fix to tutorial * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * add mamba dependancies * add mcore tag * modify dockerfile ci * modify dockerfile ci --------- Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com> Co-authored-by: JRD971000 <JRD971000@users.noreply.github.com> Co-authored-by: arendu <adithya.r@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] async checkpointing support (#9466) * add async checkpointing support * fixes * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add parallel read/write support and other optimizations * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * address comments, make dist checkpointing args configurable * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * Update default sharding type Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> * Update default sharding type Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix the arguments of forward_for_export function in msdd_models (#9624) * Fix the arguments of forward_for_export function Signed-off-by: Taejin Park <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> --------- Signed-off-by: Taejin Park <tango4j@gmail.com> Signed-off-by: tango4j <tango4j@users.noreply.github.com> Co-authored-by: tango4j <tango4j@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Change default parallel_save to False (#9632) Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Unwrap ckpt_io for model opt (async save) (#9622) Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * MCore T5 support for NeMo - Training (#9432) * huvu/mcore_t5 first commit from local * removing DEBUGGING prints * cleaning megatron_lm_encoder_decoder_model.py code * cleaning code * adding Github action test * only run mcore T5 test * only run mcore T5 test * only run mcore T5 test * only run mcore T5 test * reset .github/workflows/cicd-main.yml * reset .github/workflows/cicd-main.yml * adding condition self.mcore_t5 when running self.build_transformer_config() * refractor megatron_lm_encoder_decoder_model.py to not use self.model * only run T5-related tests * remove all self.model * reset cicd file * reset cicd file * updating codes remove duplicate if/else; adding mcore/transformer_engine to config file * adjust +model.mcore_t5=True * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> --------- Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: Huy Vu2 <huvu@login-eos01.eos.clusters.nvidia.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Expose transformer_layer_spec inside GPTConfig (#9592) * Expose transformer_layer_spec inside GPTConfig * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Expose layer-specs * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update NeMo Clip to Use MCore Modules (#9594) * update clip model and config file Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update clip for mcore Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * MCore CLIP Fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix no mask Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * few neva fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip module Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * add siglip loss Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix collate fn Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip conversion script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip convert Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * clip fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * clean up script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * clip fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix code styles Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update siglip_loss.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add REST API to deploy module (#9539) * Add REST API and FastAPI to deploy module Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add NemoQuery and requirements Signed-off-by: Abhishree <abhishreetm@gmail.com> * Edit path for config.json Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add modifications for REST API for the correct functionality Move service dir under deploy Use NeMoQueryLLM instead of NemoQuery Signed-off-by: Abhishree <abhishreetm@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply isort and black reformatting Signed-off-by: pre-commit-ci[bot] <pre-commit-ci[bot]@users.noreply.github.com> * Change default port for REST Service Change default port for REST service as Triton server also used the same port as default. Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: athitten <athitten@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: pre-commit-ci[bot] <pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Signed-off-by: athitten <athitten@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: athitten <athitten@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Mistral + Mixtral Support for NeVa (#9459) * mistral template support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * get_specs neva fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fixed mistral tokenization Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * t…
hainan-xv
pushed a commit
to hainan-xv/NeMo
that referenced
this pull request
Nov 5, 2024
* mistral template support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * get_specs neva fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fixed mistral tokenization Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * text_gen_strategy add mistral support Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * mistral text_gen fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Cleaning up neva config Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fix llama_2 default text_gen_strategy Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * fix forward() to account for new embedding optimization in MCore Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Adds support for Mistral7B-Instruct and Mixtral8x7B-Instruct models in NeVa.
Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information