Make tensor split contiguous for qkv and kv in attention #6580
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
github-actions bot
pushed a commit
that referenced
this pull request
May 8, 2023
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
aklife97
added a commit
that referenced
this pull request
May 9, 2023
KunalDhawan
added a commit
that referenced
this pull request
May 18, 2023
* Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) (#6553) * Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by * updated ASR docs with the fastconformer hybrid checkpoints * added the fastconformer RNNT and CTC models --------- Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> * Add scores for FastConformer models (#6557) (#6558) Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * Fix fp16 (#6543) (#6544) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> * Patch transcribe and support offline transcribe for hybrid model (#6550) (#6559) Signed-off-by: fayejf <fayejf07@gmail.com> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> * Fix notebook bad json (#6561) Signed-off-by: smajumdar <titu1994@gmail.com> * Change Megatron Enc Dec model to use persistent_workers (#6548) (#6552) * persistent workers * fix --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Make KenLM with PC for AggregateTokenizer and merge it (#6081) * do_lowercase, rm_punctuation Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * support beam_strategy = beam Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * black Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix config and^Cunctuation capitalization Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * rm math Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * update kenlm Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * black Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add opengrm Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * mv install_beamsearch_decoders Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * punctuation_to_preserve Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only tikenizer opion Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * Black Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * DEFAULT_TOKEN_OFFSET Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * aggregate_tokenizer Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install kenlm with more than 5gram Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * install_beamsearch_decoders Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * ngram_bin_path kenlm_bin_path Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * black Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * fix greedy PC bug Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * move global params Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * fix description and perplexity Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * fix description Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * NEMO_PATH Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * nemo:23.01 Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * License Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * description Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * isinstance Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * refactor kenlm stdin Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * black Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * add cmd arg Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * use new iter_files Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * EncDecHybridRNNTCTCModel Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * punctuation Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * train_kenlm args Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * add docstrings Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add ngram_merge docs Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * ngram_prune Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rename to ngram_merge Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rename to ngram Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * add comments Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * Ngram Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * nemo_model_file Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install_opengrm_ngram Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * install opengrm Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rename to install_opengrm.sh Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rm extra import Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * train_paths Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * text_processing Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * fix ngram_bin_path Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * DECODERS_PATH Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * farcompile Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rm text processing Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * text_processing Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * AggregateTokenizer.DummyTokenizer Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * comments Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * TextProcessingConfig Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * typo Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * doc Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * types Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * nemo_model_file Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rm assert Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * import kenlm_utils Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * return None Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * Copyright Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * 2022 Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * 2023 Signed-off-by: Nikolay Karpov <karpnv@gmail.com> --------- Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> Signed-off-by: Nikolay Karpov <karpnv@gmail.com> Co-authored-by: Nikolay Karpov <nkarpov@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * temp rtd fix (#6568) (#6569) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * [TTS] Add script for mapping speaker names to indices (#6509) Signed-off-by: Ryan <rlangman@nvidia.com> * whitespace (#6574) Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * Update manifest.py for speedup (#6565) (#6573) * Update manifest.py Re-order the checks for faster processing audio filepaths that are already absolute paths * Update manifest.py --------- Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * More streaming conformer export fixes (#6567) (#6578) Signed-off-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * user selected max_seq_len should be less than model's max_seq_len (#6333) (#6386) * user selection should not break model max limit * eval max seq length --------- Signed-off-by: arendu <adithya.r@gmail.com> Signed-off-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com> Co-authored-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Eric Harper <complex451@gmail.com> * Framework for PEFT via mixins (#6391) * init commit ptuning via mixin Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <adithya.r@gmail.com> * gpt ptuning places virtual tokens on the left only Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * encoder input modified when pre_process is true Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * optimizer group and state dict updates Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adapter ptuning working for pp>1 Signed-off-by: arendu <adithya.r@gmail.com> * adapter defaults Signed-off-by: arendu <adithya.r@gmail.com> * adapter ptuining config defaults Signed-off-by: arendu <adithya.r@gmail.com> * training works Signed-off-by: arendu <adithya.r@gmail.com> * loading and saving adapter only params during training Signed-off-by: arendu <adithya.r@gmail.com> * added checks and comments Signed-off-by: arendu <adithya.r@gmail.com> * clean up Signed-off-by: arendu <adithya.r@gmail.com> * checks for grad is None before calling all_reduce Signed-off-by: arendu <adithya.r@gmail.com> * load adapter .nemo file working Signed-off-by: arendu <adithya.r@gmail.com> * resume training for adapters Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * peft tuning Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor Signed-off-by: arendu <adithya.r@gmail.com> * file not needed Signed-off-by: arendu <adithya.r@gmail.com> * undo prompt learning dataset changes Signed-off-by: arendu <adithya.r@gmail.com> * undo updates to gpt prompt learning model Signed-off-by: arendu <adithya.r@gmail.com> * naming updates Signed-off-by: arendu <adithya.r@gmail.com> * decoding Signed-off-by: arendu <adithya.r@gmail.com> * predict_step in gpt_sft_model Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed inference from tuning config Signed-off-by: arendu <adithya.r@gmail.com> * no test in peft training Signed-off-by: arendu <adithya.r@gmail.com> * answer only loss and correct defaults for val_loss Signed-off-by: arendu <adithya.r@gmail.com> * hybrid adapters and ptuning Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * eval working.. Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * prepending tokens for ptuning Signed-off-by: arendu <adithya.r@gmail.com> * cleaned up eval config Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * clean up Signed-off-by: arendu <adithya.r@gmail.com> * update Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * default prompt template Signed-off-by: arendu <adithya.r@gmail.com> * Lora added Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support synamic length with GPT SFT Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * make branch functional Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * defaults to max_pad_length=False in GPT SFT dataset Signed-off-by: arendu <adithya.r@gmail.com> * adapter parallel_adapters to support Lora Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added early stopping by default Signed-off-by: arendu <adithya.r@gmail.com> * eval script for peft and eval config. bug fixes in predict step and added out_features to t5 adapter config Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * better defaults Signed-off-by: arendu <adithya.r@gmail.com> * updates Signed-off-by: arendu <adithya.r@gmail.com> * update Signed-off-by: arendu <adithya.r@gmail.com> * docs Signed-off-by: arendu <adithya.r@gmail.com> --------- Signed-off-by: arendu <adithya.r@gmail.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * cache and reuse inputs (#6422) (#6452) Co-authored-by: Sangkug Lym <slym@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Add patches for Virtual Parallel conversion (#6589) * Add patches for Virtual Parllel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Pass `.scale` instead of scaler object to core (#6551) * pass .scale instead of scaler object to core (#6545) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Update megatron_gpt_model.py Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * scale changes for main Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Documentation for ASR-TTS models (#6594) (#6595) * Add docs about hybrid ASR-TTS models * Add docs about text-only datasets * Add docs about ASR-TTS checkpoints * Add docs about ASR-TTS configs and training * Clean up * ASR-TTS docs: add to api, fix imports * Clean up * Wrap optional import * Revert general ASR import --------- Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Vladimir Bataev <vbataev@nvidia.com> * [TTS] Fix aligner nan loss in fp32 (#6435) * Fix nan loss in fp32 Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update SDP docs (#6485) (#6596) * add info about SDP e.g. processor classes in docs * add link to SDP docs in README * address code review comments and add SDP overview diagram * Fix spelling typo --------- Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com> * Bug/typo fixes (#6599) Signed-off-by: Igor Gitman <igitman@nvidia.com> * Manual garbage collection with an interval (#6469) (#6482) * Manual garbage collection with an interval * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * use trainer.global_step for tracking the interval of GC --------- Signed-off-by: Sangkug Lym <slym@nvidia.com> Co-authored-by: Sangkug Lym <slym@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Make tensor split contiguous (#6580) (#6593) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * [ASR] Fix for old models in change_attention_model (#6608) * fixes Signed-off-by: sam1373 <samuelkriman@gmail.com> * done already Signed-off-by: sam1373 <samuelkriman@gmail.com> --------- Signed-off-by: sam1373 <samuelkriman@gmail.com> * Update manifest.py to use os.path for get_full_path (#6598) * Update manifest.py to use os.path for get_full_path Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py to get rid of pathlib Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update manifest.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * Cherry pick commits in #6601 to main (#6611) * fix write Signed-off-by: fayejf <fayejf07@gmail.com> * decoding ctc Signed-off-by: fayejf <fayejf07@gmail.com> * temp set rnnt decoding return_best_hypothesis to true Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to transcribe_speech as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_rnnt as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_ctc as requested Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * reflect change in asr_evaluator Signed-off-by: fayejf <fayejf07@gmail.com> * reflect som and vahid comment Signed-off-by: fayejf <fayejf07@gmail.com> * remove return_best_hy=true in transcribe_speech Signed-off-by: fayejf <fayejf07@gmail.com> * no text skip Signed-off-by: fayejf <fayejf07@gmail.com> * revert partial Signed-off-by: fayejf <fayejf07@gmail.com> --------- Signed-off-by: fayejf <fayejf07@gmail.com> * Create dummy iters to satisy len checks (#6600) (#6603) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * add GPT eval mode fix for interleaved to main (#6610) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Fix batch size reconf for T5 FT for multi-validation (#6582) (#6588) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Not doing CastToFloat by default (#6524) (#6563) * Not doing CastToFloat by default * Added docustring * Dummy commit --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Turn autocast off when precision is fp32 (#6576) * Turn autocast off when precision is fp32 (#6554) * Turn autocast off when precision is fp32 Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * address review Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * merge Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * correct auto-merge Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * correct auto-merge Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * add to GPT SFT Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * update core commit hash in readme (#6622) (#6623) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * add hat image to docs (#6619) (#6621) Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> * Allow indices exchange via distributed (#6618) (#6624) Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> * Offline and streaming inference support for hybrid model (#6570) * streaming buffered for hybrid + ctc Signed-off-by: fayejf <fayejf07@gmail.com> * change default model_stride in eval.yaml Signed-off-by: fayejf <fayejf07@gmail.com> * add fc model_stride Signed-off-by: fayejf <fayejf07@gmail.com> * small fix Signed-off-by: fayejf <fayejf07@gmail.com> * check whether model and decoding match Signed-off-by: fayejf <fayejf07@gmail.com> * small fix Signed-off-by: fayejf <fayejf07@gmail.com> * streaming buffered for hybrid + rnnt Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * fix yaml Signed-off-by: fayejf <fayejf07@gmail.com> * reflect comment wip Signed-off-by: fayejf <fayejf07@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: fayejf <fayejf07@gmail.com> * refactor and verified Signed-off-by: fayejf <fayejf07@gmail.com> * add get_full_path to buffered Signed-off-by: fayejf <fayejf07@gmail.com> * small fix Signed-off-by: fayejf <fayejf07@gmail.com> * add RNNTDecodingConfig Signed-off-by: fayejf <fayejf07@gmail.com> * model name & instruction of changing decoding Signed-off-by: fayejf <fayejf07@gmail.com> --------- Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Patch decoding for PC models (#6630) (#6631) * Patch decoding logic for PC models * Patch decoding logic for PC models --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * Fix wer.py where 'errors' variable was not set (#6633) (#6634) Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Restore GPT support for interleaved pipeline parallelism (#6528) (#6613) * Restore logic for data-parallel communication with pipeline parallelism in GPT * Support dynamic attention masks in GPT * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug typos * Debug data iterator caching with interleaved pipeline parallelism Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Megatron-LM commit * Distinguish between list of data iterators and data iterator that is a list * Create dummy iters to satisy len checks * Kludge while waiting for Megatron-LM update * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set transformers offline to avoid rate limiting --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * bugfix (#6636) Signed-off-by: fayejf <fayejf07@gmail.com> * Disable interctc tests (#6638) Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add megatron_core to requirements (#6639) (#6640) * add megatron_core to requirements * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Remove from jenkins (#6642) * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * remove from jenkins Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: ericharper <complex451@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove dup Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * sft model can use this script for eval (#6637) * sft model can use this script for eval Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * please fix me Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor Signed-off-by: arendu <adithya.r@gmail.com> --------- Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] Fix TTS audio preprocessing bugs (#6628) Signed-off-by: Ryan <rlangman@nvidia.com> * Move black parameters to pyproject.toml (#6647) Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * ASR-TTS Models: Support hybrid RNNT-CTC, improve docs. (#6620) * ASR-TTS: support hybrid RNNT-CTC models * Do not warn on optional import * Explain adding options to config * Fix import guard docs * Add docs for ConcatDataset * Add explanation for sampling parameters * Initial docs for the enhancer model * Fix use_start_end_token parameter usage --------- Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * fix conversion and eval (#6648) * fix conversion and eval Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Confidence ensembles implementation (#6614) * Working version to train conf model + save ensemble class Signed-off-by: Igor Gitman <igitman@nvidia.com> * Working version Signed-off-by: Igor Gitman <igitman@nvidia.com> * Remove copy of transcribe_speech.py Signed-off-by: Igor Gitman <igitman@nvidia.com> * Move models parameter to config Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add explicit parameters to transcribe Signed-off-by: Igor Gitman <igitman@nvidia.com> * Small cleanups Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add temperature and integration tests Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add more tests Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add pc removal config Signed-off-by: Igor Gitman <igitman@nvidia.com> * Cleanup Signed-off-by: Igor Gitman <igitman@nvidia.com> * Fix typo Signed-off-by: Igor Gitman <igitman@nvidia.com> * Address review comments Signed-off-by: Igor Gitman <igitman@nvidia.com> --------- Signed-off-by: Igor Gitman <igitman@nvidia.com> * Patch memory used for NeMo Megatron models (#6615) * Patch memory used for NeMo Megatron models Signed-off-by: smajumdar <titu1994@gmail.com> * Cleanup the dtype of embeddings Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor util function for parsing precision Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor util function for parsing precision Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Try patch for Megatron O2 Signed-off-by: smajumdar <titu1994@gmail.com> * Refactor to incorporate megatron amp 02 state Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor to incorporate megatron amp 02 state Signed-off-by: smajumdar <titu1994@gmail.com> * Correct indent Signed-off-by: smajumdar <titu1994@gmail.com> * Correct utils import Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * handle artifacts when path is dir (#6658) Signed-off-by: arendu <adithya.r@gmail.com> * remove upgrading setuptools in reinstall.sh (#6659) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> * merge lora weights into base model (#6597) * merge lora weights into base model Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * typo fix Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor update Signed-off-by: arendu <adithya.r@gmail.com> * update copyright Signed-off-by: arendu <adithya.r@gmail.com> * eval needs to know the PEFT class Signed-off-by: arendu <adithya.r@gmail.com> * add target class in training script so that we can use it in eval Signed-off-by: arendu <adithya.r@gmail.com> * update Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update to work for tp1 Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set restore model path Signed-off-by: arendu <adithya.r@gmail.com> * peft can be none Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated merge script so that eval works easily Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * eval with peft or sft model Signed-off-by: arendu <adithya.r@gmail.com> * keep sentences in jsonl format Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * convert sft using correct classpath Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated to force sft yaml to have the correct target Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated docs Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix conversion and eval Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * upgrade to 23.04 (#6660) Signed-off-by: ericharper <complex451@gmail.com> * Merge r1.18.0 bugfixes and doc updates to main (#6655) * update branch Signed-off-by: ericharper <complex451@gmail.com> * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * remove from jenkins Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: ericharper <complex451@gmail.com> * remove dup Signed-off-by: ericharper <complex451@gmail.com> * update branch Signed-off-by: ericharper <complex451@gmail.com> * [TTS] reformat NeMo versions in the tts logging messages to avoid batch process them when upgrading NeMo versions. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> --------- Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Confidence ensembles: fix issues and add tuning functionality (#6657) * Implement compute confidence to properly handle blanks Signed-off-by: Igor Gitman <igitman@nvidia.com> * Implement proper confidence for transducers Signed-off-by: Igor Gitman <igitman@nvidia.com> * Implement tuning logic Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add tests for confidence tuning Signed-off-by: Igor Gitman <igitman@nvidia.com> * Remove unused imports Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add types/docs Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add comment about the main conf compute loop Signed-off-by: Igor Gitman <igitman@nvidia.com> --------- Signed-off-by: Igor Gitman <igitman@nvidia.com> * [TTS] Implement new TextToSpeech dataset (#6575) * [TTS] Implement new TextToSpeech dataset Signed-off-by: Ryan <rlangman@nvidia.com> * [TTS] Add unit tests Signed-off-by: Ryan <rlangman@nvidia.com> * [TTS] Fix defaulting of use_log_energy Signed-off-by: Ryan <rlangman@nvidia.com> * [TTS] Fix TTS export test Signed-off-by: Ryan <rlangman@nvidia.com> --------- Signed-off-by: Ryan <rlangman@nvidia.com> * Dialogue dataset (#6654) * chatbot interface Signed-off-by: Yi Dong <yidong@nvidia.com> * latest gradio Signed-off-by: Yi Dong <yidong@nvidia.com> * default greedy Signed-off-by: Yi Dong <yidong@nvidia.com> * better chatbot Signed-off-by: Yi Dong <yidong@nvidia.com> * handle preamble Signed-off-by: Yi Dong <yidong@nvidia.com> * added chatbot training capablity Signed-off-by: Yi Dong <yidong@nvidia.com> * added chatbot ui Signed-off-by: Yi Dong <yidong@nvidia.com> * remove debug code Signed-off-by: Yi Dong <yidong@nvidia.com> * default human Signed-off-by: Yi Dong <yidong@nvidia.com> * use special token for roles Signed-off-by: Yi Dong <yidong@nvidia.com> * special tokens Signed-off-by: Yi Dong <yidong@nvidia.com> * fix name Signed-off-by: Yi Dong <yidong@nvidia.com> * new chat dataset Signed-off-by: Yi Dong <yidong@nvidia.com> * fix the system token Signed-off-by: Yi Dong <yidong@nvidia.com> * upgrade gradio Signed-off-by: Yi Dong <yidong@nvidia.com> * save the chat history Signed-off-by: Yi Dong <yidong@nvidia.com> * update ui Signed-off-by: root <you@example.com> * update chat interface Signed-off-by: Yi Dong <yidong@nvidia.com> * handles canonical form Signed-off-by: Yi Dong <yidong@nvidia.com> * new sft chatbot Signed-off-by: Yi Dong <yidong@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * change format Signed-off-by: Yi Dong <yidong@nvidia.com> * check extra_id in the tokenizer Signed-off-by: Yi Dong <yidong@nvidia.com> * added vocab property check Signed-off-by: Yi Dong <yidong@nvidia.com> * added missing file Signed-off-by: Yi Dong <yidong@nvidia.com> --------- Signed-off-by: Yi Dong <yidong@nvidia.com> Signed-off-by: root <you@example.com> Co-authored-by: root <you@example.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> * Add support for RNNT/hybrid models to partial transcribe (#6609) * Add support for RNNT/hybrid models to partial transcribe Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update transcribe_utils.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update transcribe_speech.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update transcribe_utils.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * eval_beamsearch_ngram.py with hybrid ctc (#6656) * separate_punctuation = false * ctc decoding strategy = model.decoding * transcribe(files, logprobs=True) returns logprobs --------- Signed-off-by: Nikolay Karpov <karpnv@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix bucketing bug issue for picking new bucket (#6663) Signed-off-by: Nithin Rao Koluguri <nithinraok> Co-authored-by: Nithin Rao Koluguri <nithinraok> * minor fix for missing chat attr (#6671) Signed-off-by: arendu <adithya.r@gmail.com> * [TTS] Add callback for saving audio during FastPitch training (#6665) * [TTS] Add callback for saving audio during FastPitch training Signed-off-by: Ryan <rlangman@nvidia.com> * [TTS] Allow NGC model name for vocoder Signed-off-by: Ryan <rlangman@nvidia.com> --------- Signed-off-by: Ryan <rlangman@nvidia.com> --------- Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> Signed-off-by: Nikolay Karpov <karpnv@gmail.com> Signed-off-by: Ryan <rlangman@nvidia.com> Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Signed-off-by: Greg Clark <grclark@nvidia.com> Signed-off-by: arendu <adithya.r@gmail.com> Signed-off-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com> Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> Signed-off-by: Igor Gitman <igitman@nvidia.com> Signed-off-by: Sangkug Lym <slym@nvidia.com> Signed-off-by: sam1373 <samuelkriman@gmail.com> Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com> Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Yi Dong <yidong@nvidia.com> Signed-off-by: root <you@example.com> Signed-off-by: Nithin Rao Koluguri <nithinraok> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Nikolay Karpov <karpnv@gmail.com> Co-authored-by: Nikolay Karpov <nkarpov@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ryan Langman <rlangman@nvidia.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> Co-authored-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com> Co-authored-by: Sangkug Lym <slym@nvidia.com> Co-authored-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com> Co-authored-by: Igor Gitman <igitman@nvidia.com> Co-authored-by: Samuel Kriman <samuelkriman@gmail.com> Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: Adi Renduchintala <adithya.r@gmail.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Yi Dong <43824965+yidong72@users.noreply.github.com> Co-authored-by: root <you@example.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
hsiehjackson
pushed a commit
to hsiehjackson/NeMo
that referenced
this pull request
Jun 2, 2023
Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
ekmb
added a commit
that referenced
this pull request
Jun 12, 2023
…d Flash Attention (#6666) * move to nvidia megatron repo (#6465) (#6475) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Megatron KERPLE positional embeddings (#6478) (#6480) * [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- * [TTS] FastPitch speaker encoder (#6417) * Add initial codes * Remove wemb * Fix import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss * Add ConditionalInput * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix import error * Add initial codes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix dataset error * Remove reference speaker embedding * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove SV encoder * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix length type * Fix append * Move error msg * Add look-up into speaker encoder * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add valueerror msg * Move lookup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unused * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error * Rebase and Fix error * Fix spk encoder * Rename n_speakers * Follow comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error --------- * Sharded manifests for tarred datasets (#6395) * testing sharded manifests * compatibility * proper fixes * adding flag tot convert_to_tarred_audio_dataset * shard_manifests conf param * propagating the shard_manifests param * propagating the shard_manifests param * distributed checks * typo * typo * fixes * fixes * fixes * fixes * fixes * fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py * reversing manifest shards flag * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests * excluding manifests from webdataset url expansion * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore * explicit use of UTF-8 for manifest i/o --------- * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. --------- * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support * added tests for rampup batch size * fixed the typos * added assertions * changed assertion rules * deleted unused imports * changed tests for rampup batch size * updated rampup batch size tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling * rampup batch size tests changes --------- * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default * updated README with the new agg_tokenizer_manifest flag * fixed typo in scripts/speech_recognition/code_switching/README.md * changed agg_tokenizer_manifest to is_lid_manifest --------- * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script * add return encoding embedding option * update script * add rnnt and hat ngram decoding script * add some parameters * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig * replace return_encoder_embeddings parameter * generalization of scipt behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter * remove return_encoder_embeddings parameter * add manual encoder_embeddings calculation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 * fix rescoring description --------- * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- * fix replace_bos_with_pad not found (#6443) (#6450) * reduce workers on NMT CI (#6472) (#6474) * 1. Added KERPLE positional embeddings to encoder-decoder. * 1. Added a missing file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: Micha Livne <mlivne@nvidia.com> Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: Jim O’Regan <jaoregan@tcd.ie> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix an invalid link in get_data.py of ljspeech (#6456) Usage of the link in line 63 leads to downloading a html file not a tsv file, so we need to change it to a raw link. Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * 1. Added external index sample. (#6462) (#6483) Signed-off-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Update README to add core installation (#6488) (#6489) * update README for megatron-core * fix --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix cache aware hybrid bugs (#6466) (#6484) Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix typos (#6494) (#6495) Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add disclaimer about dataset for ASR (#6496) Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * fix (#6502) datastore_path_to_webdataset_url(p) if is_datastore_path(p) and is_tarred_path(p) else p NameError: name 'is_tarred_path' is not defined Co-authored-by: George <gzelenfroind@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * fix broken links r1.18.0 (#6501) (#6504) * fix broken links * fix broken links --------- Signed-off-by: Evelina <ebakhturina@nvidia.com> Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [TTS] Create functions for TTS preprocessing without dataloader (#6317) * [TTS] Create functions for TTS preprocessing without dataloader Signed-off-by: Ryan <rlangman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Cache aware streaming nfa (#6209) * add cache aware streaming to nemo aligner Signed-off-by: Slyne Deng <slyned@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator (#6499) * [BugFix] _get_batch_preds() is forced to keep logits in decoder timestamps generators Signed-off-by: Taejin Park <tango4j@gmail.com> * Ingnore keep_logits boolean in FrameASRBatchLogits Signed-off-by: Taejin Park <tango4j@gmail.com> --------- Signed-off-by: Taejin Park <tango4j@gmail.com> Co-authored-by: Jagadeesh Balam <4916480+jbalam-nv@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [TTS] Fix FastPitch energy code (#6511) Signed-off-by: Ryan <rlangman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * fix custom forward_torch_softmax (#6512) (#6517) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [TTS] fixed broken path. (#6514) (#6518) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix normalization of impulse response in ImpulsePerturbation (#6505) Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add interleaved pp support (#6498) * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Switch to megatron core Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix typos (#6523) * Fix typos Signed-off-by: smajumdar <titu1994@gmail.com> * Fix typos Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * New noise_norm perturbation based on Riva work (#6445) * Initial commit for new noise_norm perturbation Signed-off-by: Daniel Egert <degert@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Minor fix to random seed in perturb Signed-off-by: Daniel Egert <degert@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated code to reflect feedback Signed-off-by: Daniel Egert <degert@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates for feedback given by code reviewers Signed-off-by: Daniel Egert <degert@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates in response to PR feedback Signed-off-by: Daniel Egert <degert@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added comment about ref_mic being None Signed-off-by: Daniel Egert <degert@nvidia.com> * Updated perturb to use inspect module Signed-off-by: Daniel Egert <degert@nvidia.com> --------- Signed-off-by: Daniel Egert <degert@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [TTS] Add script for computing feature stats (#6508) * [TTS] Add script for computing feature stats Signed-off-by: Ryan <rlangman@nvidia.com> * [TTS] Add overwrite config Signed-off-by: Ryan <rlangman@nvidia.com> --------- Signed-off-by: Ryan <rlangman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add Frame-VAD model and datasets (#6441) * add model, dataset, necessary utils and tests Signed-off-by: stevehuang52 <heh@nvidia.com> * fix tarred data Signed-off-by: stevehuang52 <heh@nvidia.com> * fix typo Signed-off-by: stevehuang52 <heh@nvidia.com> * update docstring Signed-off-by: stevehuang52 <heh@nvidia.com> * update doc Signed-off-by: stevehuang52 <heh@nvidia.com> * update doc Signed-off-by: stevehuang52 <heh@nvidia.com> * update pretrained model info Signed-off-by: stevehuang52 <heh@nvidia.com> --------- Signed-off-by: stevehuang52 <heh@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Support dynamic length batches with GPT SFT (#6510) * Support synamic length with GPT SFT Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * make branch functional Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * added back the fast emit section to the configs. (#6540) (#6542) * added back the fast emit section to the configs. * added back the fast emit section to the configs. --------- Signed-off-by: Vahid <vnoroozi@nvidia.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * removing unnessary avoid_bfloat16_autocast_context (#6481) Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * FC models in menu (#6473) * FC models in menu Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [TTS] Add tutorials for FastPitch TTS speaker adaptation with adapters (#6431) * Add tts adapter tutorial Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Update main tutorial Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add tts adapter tutorial Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Update main tutorial Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Update tutorial Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix load .nemo error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support multi-speaker fine-tune Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Use .nemo Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Follow Comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix bug Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix bug Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix bug Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add precomputed speaker emb Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix space Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove repeated argument Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * optional batch size Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix comments in notebook Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [TTS] Create initial TTS dataset feature processors (#6507) Signed-off-by: Ryan <rlangman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * fix (#6529) (#6546) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) (#6553) * Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by * updated ASR docs with the fastconformer hybrid checkpoints * added the fastconformer RNNT and CTC models --------- Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add scores for FastConformer models (#6557) (#6558) Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix fp16 (#6543) (#6544) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Patch transcribe and support offline transcribe for hybrid model (#6550) (#6559) Signed-off-by: fayejf <fayejf07@gmail.com> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix notebook bad json (#6561) Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Change Megatron Enc Dec model to use persistent_workers (#6548) (#6552) * persistent workers * fix --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Make KenLM with PC for AggregateTokenizer and merge it (#6081) * do_lowercase, rm_punctuation Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * support beam_strategy = beam Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * black Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix config and^Cunctuation capitalization Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * rm math Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * update kenlm Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * black Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add opengrm Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * mv install_beamsearch_decoders Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * punctuation_to_preserve Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only tikenizer opion Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * Black Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * DEFAULT_TOKEN_OFFSET Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * aggregate_tokenizer Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install kenlm with more than 5gram Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * install_beamsearch_decoders Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * ngram_bin_path kenlm_bin_path Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * black Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * fix greedy PC bug Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * move global params Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * fix description and perplexity Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * fix description Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * NEMO_PATH Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * nemo:23.01 Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * License Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * description Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * isinstance Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * refactor kenlm stdin Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * black Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * add cmd arg Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * use new iter_files Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * EncDecHybridRNNTCTCModel Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * punctuation Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * train_kenlm args Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * add docstrings Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add ngram_merge docs Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * ngram_prune Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rename to ngram_merge Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rename to ngram Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * add comments Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * Ngram Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * nemo_model_file Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install_opengrm_ngram Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * install opengrm Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rename to install_opengrm.sh Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rm extra import Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * train_paths Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * text_processing Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * fix ngram_bin_path Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * DECODERS_PATH Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * farcompile Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rm text processing Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * text_processing Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * AggregateTokenizer.DummyTokenizer Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * comments Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * TextProcessingConfig Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * typo Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * doc Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * types Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * nemo_model_file Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * rm assert Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * import kenlm_utils Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * return None Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * Copyright Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * 2022 Signed-off-by: Nikolay Karpov <karpnv@gmail.com> * 2023 Signed-off-by: Nikolay Karpov <karpnv@gmail.com> --------- Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> Signed-off-by: Nikolay Karpov <karpnv@gmail.com> Co-authored-by: Nikolay Karpov <nkarpov@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * fix for running on 1 GPU. Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * temp rtd fix (#6568) (#6569) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [TTS] Add script for mapping speaker names to indices (#6509) Signed-off-by: Ryan <rlangman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * whitespace (#6574) Signed-off-by: Nikolay Karpov <karpnv@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Update manifest.py for speedup (#6565) (#6573) * Update manifest.py Re-order the checks for faster processing audio filepaths that are already absolute paths * Update manifest.py --------- Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * More streaming conformer export fixes (#6567) (#6578) Signed-off-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * user selected max_seq_len should be less than model's max_seq_len (#6333) (#6386) * user selection should not break model max limit * eval max seq length --------- Signed-off-by: arendu <adithya.r@gmail.com> Signed-off-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com> Co-authored-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Framework for PEFT via mixins (#6391) * init commit ptuning via mixin Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <adithya.r@gmail.com> * gpt ptuning places virtual tokens on the left only Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * encoder input modified when pre_process is true Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * optimizer group and state dict updates Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adapter ptuning working for pp>1 Signed-off-by: arendu <adithya.r@gmail.com> * adapter defaults Signed-off-by: arendu <adithya.r@gmail.com> * adapter ptuining config defaults Signed-off-by: arendu <adithya.r@gmail.com> * training works Signed-off-by: arendu <adithya.r@gmail.com> * loading and saving adapter only params during training Signed-off-by: arendu <adithya.r@gmail.com> * added checks and comments Signed-off-by: arendu <adithya.r@gmail.com> * clean up Signed-off-by: arendu <adithya.r@gmail.com> * checks for grad is None before calling all_reduce Signed-off-by: arendu <adithya.r@gmail.com> * load adapter .nemo file working Signed-off-by: arendu <adithya.r@gmail.com> * resume training for adapters Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * peft tuning Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor Signed-off-by: arendu <adithya.r@gmail.com> * file not needed Signed-off-by: arendu <adithya.r@gmail.com> * undo prompt learning dataset changes Signed-off-by: arendu <adithya.r@gmail.com> * undo updates to gpt prompt learning model Signed-off-by: arendu <adithya.r@gmail.com> * naming updates Signed-off-by: arendu <adithya.r@gmail.com> * decoding Signed-off-by: arendu <adithya.r@gmail.com> * predict_step in gpt_sft_model Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed inference from tuning config Signed-off-by: arendu <adithya.r@gmail.com> * no test in peft training Signed-off-by: arendu <adithya.r@gmail.com> * answer only loss and correct defaults for val_loss Signed-off-by: arendu <adithya.r@gmail.com> * hybrid adapters and ptuning Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * eval working.. Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * prepending tokens for ptuning Signed-off-by: arendu <adithya.r@gmail.com> * cleaned up eval config Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * clean up Signed-off-by: arendu <adithya.r@gmail.com> * update Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * default prompt template Signed-off-by: arendu <adithya.r@gmail.com> * Lora added Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support synamic length with GPT SFT Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * make branch functional Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * defaults to max_pad_length=False in GPT SFT dataset Signed-off-by: arendu <adithya.r@gmail.com> * adapter parallel_adapters to support Lora Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added early stopping by default Signed-off-by: arendu <adithya.r@gmail.com> * eval script for peft and eval config. bug fixes in predict step and added out_features to t5 adapter config Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * better defaults Signed-off-by: arendu <adithya.r@gmail.com> * updates Signed-off-by: arendu <adithya.r@gmail.com> * update Signed-off-by: arendu <adithya.r@gmail.com> * docs Signed-off-by: arendu <adithya.r@gmail.com> --------- Signed-off-by: arendu <adithya.r@gmail.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * cache and reuse inputs (#6422) (#6452) Co-authored-by: Sangkug Lym <slym@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add patches for Virtual Parallel conversion (#6589) * Add patches for Virtual Parllel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Pass `.scale` instead of scaler object to core (#6551) * pass .scale instead of scaler object to core (#6545) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Update megatron_gpt_model.py Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * scale changes for main Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Documentation for ASR-TTS models (#6594) (#6595) * Add docs about hybrid ASR-TTS models * Add docs about text-only datasets * Add docs about ASR-TTS checkpoints * Add docs about ASR-TTS configs and training * Clean up * ASR-TTS docs: add to api, fix imports * Clean up * Wrap optional import * Revert general ASR import --------- Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [TTS] Fix aligner nan loss in fp32 (#6435) * Fix nan loss in fp32 Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Update SDP docs (#6485) (#6596) * add info about SDP e.g. processor classes in docs * add link to SDP docs in README * address code review comments and add SDP overview diagram * Fix spelling typo --------- Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Bug/typo fixes (#6599) Signed-off-by: Igor Gitman <igitman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Manual garbage collection with an interval (#6469) (#6482) * Manual garbage collection with an interval * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * use trainer.global_step for tracking the interval of GC --------- Signed-off-by: Sangkug Lym <slym@nvidia.com> Co-authored-by: Sangkug Lym <slym@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Make tensor split contiguous (#6580) (#6593) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [ASR] Fix for old models in change_attention_model (#6608) * fixes Signed-off-by: sam1373 <samuelkriman@gmail.com> * done already Signed-off-by: sam1373 <samuelkriman@gmail.com> --------- Signed-off-by: sam1373 <samuelkriman@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Update manifest.py to use os.path for get_full_path (#6598) * Update manifest.py to use os.path for get_full_path Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py to get rid of pathlib Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update manifest.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Cherry pick commits in #6601 to main (#6611) * fix write Signed-off-by: fayejf <fayejf07@gmail.com> * decoding ctc Signed-off-by: fayejf <fayejf07@gmail.com> * temp set rnnt decoding return_best_hypothesis to true Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to transcribe_speech as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_rnnt as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_ctc as requested Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * reflect change in asr_evaluator Signed-off-by: fayejf <fayejf07@gmail.com> * reflect som and vahid comment Signed-off-by: fayejf <fayejf07@gmail.com> * remove return_best_hy=true in transcribe_speech Signed-off-by: fayejf <fayejf07@gmail.com> * no text skip Signed-off-by: fayejf <fayejf07@gmail.com> * revert partial Signed-off-by: fayejf <fayejf07@gmail.com> --------- Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Create dummy iters to satisy len checks (#6600) (#6603) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * add GPT eval mode fix for interleaved to main (#6610) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix batch size reconf for T5 FT for multi-validation (#6582) (#6588) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Not doing CastToFloat by default (#6524) (#6563) * Not doing CastToFloat by default * Added docustring * Dummy commit --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Turn autocast off when precision is fp32 (#6576) * Turn autocast off when precision is fp32 (#6554) * Turn autocast off when precision is fp32 Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * address review Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * merge Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * correct auto-merge Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * correct auto-merge Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * add to GPT SFT Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * update core commit hash in readme (#6622) (#6623) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * add hat image to docs (#6619) (#6621) Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Allow indices exchange via distributed (#6618) (#6624) Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Offline and streaming inference support for hybrid model (#6570) * streaming buffered for hybrid + ctc Signed-off-by: fayejf <fayejf07@gmail.com> * change default model_stride in eval.yaml Signed-off-by: fayejf <fayejf07@gmail.com> * add fc model_stride Signed-off-by: fayejf <fayejf07@gmail.com> * small fix Signed-off-by: fayejf <fayejf07@gmail.com> * check whether model and decoding match Signed-off-by: fayejf <fayejf07@gmail.com> * small fix Signed-off-by: fayejf <fayejf07@gmail.com> * streaming buffered for hybrid + rnnt Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * fix yaml Signed-off-by: fayejf <fayejf07@gmail.com> * reflect comment wip Signed-off-by: fayejf <fayejf07@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: fayejf <fayejf07@gmail.com> * refactor and verified Signed-off-by: fayejf <fayejf07@gmail.com> * add get_full_path to buffered Signed-off-by: fayejf <fayejf07@gmail.com> * small fix Signed-off-by: fayejf <fayejf07@gmail.com> * add RNNTDecodingConfig Signed-off-by: fayejf <fayejf07@gmail.com> * model name & instruction of changing decoding Signed-off-by: fayejf <fayejf07@gmail.com> --------- Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Patch decoding for PC models (#6630) (#6631) * Patch decoding logic for PC models * Patch decoding logic for PC models --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix wer.py where 'errors' variable was not set (#6633) (#6634) Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Restore GPT support for interleaved pipeline parallelism (#6528) (#6613) * Restore logic for data-parallel communication with pipeline parallelism in GPT * Support dynamic attention masks in GPT * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug typos * Debug data iterator caching with interleaved pipeline parallelism Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Megatron-LM commit * Distinguish between list of data iterators and data iterator that is a list * Create dummy iters to satisy len checks * Kludge while waiting for Megatron-LM update * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set transformers offline to avoid rate limiting --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add FA Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix XPOS Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add warning Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix bugs Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix attention Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix comment Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix cast dtype Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Undo xpos Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * bugfix (#6636) Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Disable interctc tests (#6638) Signed-off-by: Igor Gitman <igitman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add megatron_core to requirements (#6639) (#6640) * add megatron_core to requirements * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove from jenkins (#6642) * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * remove from jenkins Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: ericharper <complex451@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove dup Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * sft model can use this script for eval (#6637) * sft model can use this script for eval Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * please fix me Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor Signed-off-by: arendu <adithya.r@gmail.com> --------- Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [TTS] Fix TTS audio preprocessing bugs (#6628) Signed-off-by: Ryan <rlangman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Move black parameters to pyproject.toml (#6647) Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * ASR-TTS Models: Support hybrid RNNT-CTC, improve docs. (#6620) * ASR-TTS: support hybrid RNNT-CTC models * Do not warn on optional import * Explain adding options to config * Fix import guard docs * Add docs for ConcatDataset * Add explanation for sampling parameters * Initial docs for the enhancer model * Fix use_start_end_token parameter usage --------- Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * fix conversion and eval (#6648) * fix conversion and eval Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Confidence ensembles implementation (#6614) * Working version to train conf model + save ensemble class Signed-off-by: Igor Gitman <igitman@nvidia.com> * Working version Signed-off-by: Igor Gitman <igitman@nvidia.com> * Remove copy of transcribe_speech.py Signed-off-by: Igor Gitman <igitman@nvidia.com> * Move models parameter to config Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add explicit parameters to transcribe Signed-off-by: Igor Gitman <igitman@nvidia.com> * Small cleanups Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add temperature and integration tests Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add more tests Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add pc removal config Signed-off-by: Igor Gitman <igitman@nvidia.com> * Cleanup Signed-off-by: Igor Gitman <igitman@nvidia.com> * Fix typo Signed-off-by: Igor Gitman <igitman@nvidia.com> * Address review comments Signed-off-by: Igor Gitman <igitman@nvidia.com> --------- Signed-off-by: Igor Gitman <igitman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Patch memory used for NeMo Megatron models (#6615) * Patch memory used for NeMo Megatron models Signed-off-by: smajumdar <titu1994@gmail.com> * Cleanup the dtype of embeddings Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor util function for parsing precision Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor util function for parsing precision Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Try patch for Megatron O2 Signed-off-by: smajumdar <titu1994@gmail.com> * Refactor to incorporate megatron amp 02 state Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor to incorporate megatron amp 02 state Signed-off-by: smajumdar <titu1994@gmail.com> * Correct indent Signed-off-by: smajumdar <titu1994@gmail.com> * Correct utils import Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * handle artifacts when path is dir (#6658) Signed-off-by: arendu <adithya.r@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * remove upgrading setuptools in reinstall.sh (#6659) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * merge lora weights into base model (#6597) * merge lora weights into base model Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * typo fix Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor update Signed-off-by: arendu <adithya.r@gmail.com> * update copyright Signed-off-by: arendu <adithya.r@gmail.com> * eval needs to know the PEFT class Signed-off-by: arendu <adithya.r@gmail.com> * add target class in training script so that we can use it in eval Signed-off-by: arendu <adithya.r@gmail.com> * update Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update to work for tp1 Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set restore model path Signed-off-by: arendu <adithya.r@gmail.com> * peft can be none Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated merge script so that eval works easily Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * eval with peft or sft model Signed-off-by: arendu <adithya.r@gmail.com> * keep sentences in jsonl format Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * convert sft using correct classpath Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated to force sft yaml to have the correct target Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated docs Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix conversion and eval Signed-off-by: arendu <adithya.r@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * upgrade to 23.04 (#6660) Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Merge r1.18.0 bugfixes and doc updates to main (#6655) * update branch Signed-off-by: ericharper <complex451@gmail.com> * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * remove from jenkins Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: ericharper <complex451@gmail.com> * remove dup Signed-off-by: ericharper <complex451@gmail.com> * update branch Signed-off-by: ericharper <complex451@gmail.com> * [TTS] reformat NeMo versions in the tts logging messages to avoid batch process them when upgrading NeMo versions. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> --------- Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Confidence ensembles: fix issues and add tuning functionality (#6657) * Implement compute confidence to properly handle blanks Signed-off-by: Igor Gitman <igitman@nvidia.com> * Implement proper confidence for transducers Signed-off-by: Igor Gitman <igitman@nvidia.com> * Implement tuning logic Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add tests for confidence tuning Signed-off-by: Igor Gitman <igitman@nvidia.com> * Remove unused imports Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add types/docs Signed-off-by: Igor Gitman <igitman@nvidia.com> * Add comment about the main conf compute loop Signed-off-by: Igor Gitman <igitman@nvidia.com> --------- Signed-off-by: Igor Gitman <igitman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [TTS] Implement new TextToSpeech dataset (#6575) * [TTS] Implement new TextToSpeech dataset Signed-off-by: Ryan <rlangman@nvidia.com> * [TTS] Add unit tests Signed-off-by: Ryan <rlangman@nvidia.com> * [TTS] Fix defaulting of use_log_energy Signed-off-by: Ryan <rlangman@nvidia.com> * [TTS] Fix TTS export test Signed-off-by: Ryan <rlangman@nvidia.com> --------- Signed-off-by: Ryan <rlangman@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Dialogue dataset (#6654) * chatbot interface Signed-off-by: Yi Dong <yidong@nvidia.com> * latest gradio Signed-off-by: Yi Dong <yidong@nvidia.com> * default greedy Signed-off-by: Yi Dong <yidong@nvidia.com> * better chatbot Signed-off-by: Yi Dong <yidong@nvidia.com> * handle preamble Signed-off-by: Yi Dong <yidong@nvidia.com> * added chatbot training capablity Signed-off-by: Yi Dong <yidong@nvidia.com> * added chatbot ui Signed-off-by: Yi Dong <yidong@nvidia.com> * remove debug code Signed-off-by: Yi Dong <yidong@nvidia.com> * default human Signed-off-by: Yi Dong <yidong@nvidia.com> * use special token for roles Signed-off-by: Yi Dong <yidong@nvidia.com> * special tokens Signed-off-by: Yi Dong <yidong@nvidia.com> * fix name Signed-off-by: Yi Dong <yidong@nvidia.com> * new chat dataset Signed-off-by: Yi Dong <yidong@nvidia.com> * fix the system token Signed-off-by: Yi Dong <yidong@nvidia.com> * upgrade gradio Signed-off-by: Yi Dong <yidong@nvidia.com> * save the chat history Signed-off-by: Yi Dong <yidong@nvidia.com> * update ui Signed-off-by: root <you@example.com> * update chat interface Signed-off-by: Yi Dong <yidong@nvidia.com> * handles canonical form Signed-off-by: Yi Dong <yidong@nvidia.com> * new sft chatbot Signed-off-by: Yi Dong <yidong@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * change format Signed-off-by: Yi Dong <yidong@nvidia.com> * check extra_id in the tokenizer Signed-off-by: Yi Dong <yidong@nvidia.com> * added vocab property check Signed-off-by: Yi Dong <yidong@nvidia.com> * added missing file Signed-off-by: Yi Dong <yidong@nvidia.com> --------- Signed-off-by: Yi Dong <yidong@nvidia.com> Signed-off-by: root <you@example.com> Co-authored-by: root <you@example.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add support for RNNT/hybrid models to partial transcribe (#6609) * Add support for RNNT/hybrid models to partial transcribe Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update transcribe_utils.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update transcribe_speech.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update transcr…
ericharper
added a commit
that referenced
this pull request
Aug 11, 2023
* [TTS] bugfix for missing configs. (#4725) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * docs typo fix Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> * Fix pynini install in TTS tutorials (#4729) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Fix ASR notebooks (#4738) Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: smajumdar <smajumdar@nvidia.com> * Multilingual VAD model (#4734) * add ngc link Signed-off-by: fayejf <fayejf07@gmail.com> * add tuned VAD config on ASR data Signed-off-by: fayejf <fayejf07@gmail.com> * yaml note Signed-off-by: fayejf <fayejf07@gmail.com> * update vad asr notebook with mVAD Signed-off-by: fayejf <fayejf07@gmail.com> * update vad infer config comment Signed-off-by: fayejf <fayejf07@gmail.com> * fix Signed-off-by: fayejf <fayejf07@gmail.com> * mvad sd config for ch109 Signed-off-by: fayejf <fayejf07@gmail.com> * update sd readme Signed-off-by: fayejf <fayejf07@gmail.com> * add new mVAD model to doc Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * update sd tutorial with mVAD Signed-off-by: fayejf <fayejf07@gmail.com> * typo fix Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: fayejf <fayejf07@gmail.com> * publish pretrained itn t5 model for English (#4748) Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> * Updated docs and doc paths (#4754) * Updated docs and doc paths Signed-off-by: Virginia Adams <vadams@nvidia.com> * Update Multitask_Prompt_and_PTuning.ipynb * Update README.rst * Changed branch name to use single quotes Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> * fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762) * [TTS] updated config with a German IPA phoneme tokenizer (#4756) * [TTS] added a German IPA phoneme tokenizer * [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence. * [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer. * [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Update r1.11 to new heteronyms list (#4745) * Update configs to new heteronyms list * Remove old heteronyms list, add alt 'merchandise' pron to CMUdict * Update remaining references to old heteronyms list Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Update CMUdict with more recent 0.7b entries (#4768) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Install pynini in docker container (#4733) Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Fix tutorial formatting (#4778) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * [TTS] deprecated old scripts for ljspeech. (#4780) * deprecated old scripts for ljspeech. * removed relevent function calls in TTS docs. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * update branch and typos (#4788) Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> * Adding support for models trained with full context for cache-aware streaming. (#4687) * added support for models trained with full context. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * dropped seq_range Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed indexing in caching methods. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addressed comments. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> * Update megatron encoder decoder model to support py37 for colab (#4791) * [ASR] Add pretrained ASR models for Croatian (#4682) * [ASR] Add pretrained ASR models for Croatian Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Fix style for import Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * added/fixed export for Megatron models (#4712) * added/fixed export for Megatron models Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * fixed style Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * fixed FusedScaleMaskSoftmax in BioMegatron Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * included comments Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> * update branch for qa notebook Signed-off-by: ericharper <complex451@gmail.com> * Fix initializing weights from ptl ckpt with exclude (#4807) Signed-off-by: sam1373 <samuelkriman@gmail.com> Signed-off-by: sam1373 <samuelkriman@gmail.com> * Fix index error from addition of voiced_mask and p_voiced (#4811) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * T5 prompt learning fixes (#4771) * RPE, hidden size and config fixes Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update to reflect new config names Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Sentencepiece fixes Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix finetuning Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add encoder seq len to gpt Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add finetune eval script Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix name Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update Jenkinsfile Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update config Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix CI test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update check Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Backward compat Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update CI test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Split rank for Enc-Dec models Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Address comments Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com> * G2P docs (#4841) * g2p docs added Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix references Signed-off-by: ekmb <ebakhturina@nvidia.com> * address review feedback Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: ekmb <ebakhturina@nvidia.com> * Fix providing glue in seq2seq eval (#4843) * Fix providing glue in seq2seq eval Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Updated inference code and squad scripts (#4835) * Updated inference code and squad scripts Signed-off-by: Virginia Adams <vadams@nvidia.com> * Reverted GPT & T5 inference files back to use NLPDDPlugin Signed-off-by: Virginia Adams <vadams@nvidia.com> * Overwrite frozen LM to use fused adam Signed-off-by: Virginia Adams <vadams@nvidia.com> * Added padded vocab size Signed-off-by: Virginia Adams <vadams@nvidia.com> * Fixed val check interval value Signed-off-by: Virginia Adams <vadams@nvidia.com> * Python format fix Signed-off-by: Virginia Adams <vadams@nvidia.com> * Make t5 prompt learning preds write to file Signed-off-by: Virginia Adams <vadams@nvidia.com> * Added back dp=1 check Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> * Update README.rst * Fix uppercasing mismatch for IPA heteronyms (#4860) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790) * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Revert pin memory Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Sean Naren <sean.narenthiran@gmail.com> * Fix mha (#4866) * fix bug in mha forward function related to cache update return type Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * fix lgtm Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Co-authored-by: Sean Naren <snarenthiran@nvidia.com> * ipa bug fix (#4871) Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: ekmb <ebakhturina@nvidia.com> * Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884) * Fix nmt and ckpt_to_nemo Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * added utf8 encoding (#4892) Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> * 1. Applying the same patch to r1.11.0 (#4894) Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * Update tutorials.rst (#4897) * update readme with apex commit Signed-off-by: ericharper <complex451@gmail.com> * Add support for Apex distributed Adam optimizer with GPT-3 (#4487) * Add support for Apex distributed Adam optimizer with GPT-3 Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix bug in grad clipping with dist Adam Grad norm was computed over all params, not respecting model parallelism. Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix bug with DDP initialization Signed-off-by: Tim Moon <tmoon@nvidia.com> * Make distopt dependent on megatron_amp_o2 Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix code formatting Signed-off-by: Tim Moon <tmoon@nvidia.com> * Handle dist Adam in optimizer unit tests Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> * update readme Signed-off-by: ericharper <complex451@gmail.com> * update readme Signed-off-by: ericharper <complex451@gmail.com> * latent model support * 1. Debugging. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. * update branch Signed-off-by: ericharper <complex451@gmail.com> * fix replace_bos_with_pad not found (#6443) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Support Swiglu in TP PP Conversion (#6437) * Support Swiglu in TP PP Conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Guard activation Signed-off-by: smajumdar <titu1994@gmail.com> * Guard activation Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> * BERT pre-training mp fork to spawn (#6442) * change bert fork to spawn Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * num_workers=0 fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Meagtron encoder decoder fix for empty validation outputs (#6459) * 1. Meagtron encoder decoder fix for empty validation outputs. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. --------- Signed-off-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> * Added/updated new Conformer configs (#6426) * updated conf files. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * moved longconformer confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated readme. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated readme. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated batch sizes and added fastconformer ctc streaming configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated batch sizes. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added hybrid support. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added hybrid support. Signed-off-by: Vahid <vnoroozi@nvidia.com> --------- Signed-off-by: Vahid <vnoroozi@nvidia.com> * reduce workers on NMT CI (#6472) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * move to nvidia megatron repo (#6465) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Megatron KERPLE positional embeddings (#6478) * [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] FastPitch speaker encoder (#6417) * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove wemb Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix import Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add ConditionalInput Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix import error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix dataset error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove reference speaker embedding Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove SV encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix length type Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix append Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Move error msg Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add look-up into speaker encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add valueerror msg Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Move lookup Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove unused Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Rebase and Fix error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix spk encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Rename n_speakers Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Sharded manifests for tarred datasets (#6395) * testing sharded manifests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * compatibility Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * proper fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * adding flag tot convert_to_tarred_audio_dataset Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * shard_manifests conf param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * distributed checks Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * typo Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * typo Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * reversing manifest shards flag Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * excluding manifests from webdataset url expansion Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * explicit use of UTF-8 for manifest i/o Signed-off-by: Dima Rekesh <bmwshop@gmail.com> --------- Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> --------- Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * added tests for rampup batch size Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * fixed the typos Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * added assertions Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * changed assertion rules Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * deleted unused imports Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * changed tests for rampup batch size Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * updated rampup batch size tests Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * rampup batch size tests changes Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> --------- Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- Signed-off-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * updated README with the new agg_tokenizer_manifest flag Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * fixed typo in scripts/speech_recognition/code_switching/README.md Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * changed agg_tokenizer_manifest to is_lid_manifest Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> --------- Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add return encoding embedding option Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * update script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add rnnt and hat ngram decoding script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add some parameters Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * replace return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * generalization of scipt behavior Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add manual encoder_embeddings calculation Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> * fix rescoring description Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> --------- Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * fix replace_bos_with_pad not found (#6443) (#6450) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * reduce workers on NMT CI (#6472) (#6474) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * 1. Added KERPLE positional embeddings to encoder-decoder. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added a missing file. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: Micha Livne <mlivne@nvidia.com> Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: Jim O’Regan <jaoregan@tcd.ie> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * 1. Added external index sample. (#6462) Signed-off-by: Micha Livne <mlivne@nvidia.com> * Fix cache aware hybrid bugs (#6466) * Update README to add core installation (#6488) * update README for megatron-core Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Fix typos (#6494) Signed-off-by: smajumdar <titu1994@gmail.com> * fix broken links r1.18.0 (#6501) * fix broken links Signed-off-by: Evelina <ebakhturina@nvidia.com> * fix broken links Signed-off-by: Evelina <ebakhturina@nvidia.com> --------- Signed-off-by: Evelina <ebakhturina@nvidia.com> * 1. Fixed gaussian hidden transform. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Finished updating hidden loss for MIM. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix custom forward_torch_softmax (#6512) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500) * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0 Signed-off-by: Taejin Park <tango4j@gmail.com> * ignore keep_logits in FrameBatchASRLogits Signed-off-by: Taejin Park <tango4j@gmail.com> --------- Signed-off-by: Taejin Park <tango4j@gmail.com> * [TTS] fixed broken path. (#6514) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * 1. Added a hiddens module. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix typos (#6523) (#6539) * Fix typos Signed-off-by: smajumdar <titu1994@gmail.com> * Fix typos Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> (cherry picked from commit 5468077f5127be1a4c88065de2544f4268b9a6e4) * added back the fast emit section to the configs. (#6540) * added back the fast emit section to the configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added back the fast emit section to the configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> --------- Signed-off-by: Vahid <vnoroozi@nvidia.com> * Fix fp16 (#6543) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * fix (#6529) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * pass .scale instead of scaler object to core (#6545) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Change Megatron Enc Dec model to use persistent_workers (#6548) * persistent workers Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) * Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * updated ASR docs with the fastconformer hybrid checkpoints Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * added the fastconformer RNNT and CTC models Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> --------- Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * Add scores for FastConformer models (#6557) Signed-off-by: smajumdar <titu1994@gmail.com> * Patch transcribe and support offline transcribe for hybrid model (#6550) Signed-off-by: fayejf <fayejf07@gmail.com> * Not doing CastToFloat by default (#6524) * Not doing CastToFloat by default Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Added docustring Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Dummy commit Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * temp rtd fix (#6568) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py for speedup (#6565) * Update manifest.py Re-order the checks for faster processing audio filepaths that are already absolute paths Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update manifest.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> --------- Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * Turn autocast off when precision is fp32 (#6554) * Turn autocast off when precision is fp32 Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * address review Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * merge Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * More streaming conformer export fixes (#6567) Signed-off-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * Fix batch size reconf for T5 FT for multi-validation (#6582) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Updated Megatron LM encoder/decoder to use cfg for hiddens. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added support to register externalhidden loss / transforms. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make tensor split contiguous (#6580) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Patches from main to r1.18.0 for Virtual Parallel (#6592) * Add interleaved pp support (#6498) * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Switch to megatron core Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 892987169ef277f328e15b71a5a0c9bd961c8ee7) * Add patches for Virtual Parallel conversion (#6589) * Add patches for Virtual Parllel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 1d813a372ab51688e3af6395d905a4c0366ffd23) * Documentation for ASR-TTS models (#6594) * Add docs about hybrid ASR-TTS models Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about text-only datasets Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about ASR-TTS checkpoints Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about ASR-TTS configs and training Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Clean up Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * ASR-TTS docs: add to api, fix imports Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Clean up Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Wrap optional import Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Revert general ASR import Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> --------- Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Update SDP docs (#6485) * add info about SDP e.g. processor classes in docs Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * add link to SDP docs in README Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * address code review comments and add SDP overview diagram Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * Fix spelling typo Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> --------- Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * Create dummy iters to satisy len checks (#6600) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * Restore GPT support for interleaved pipeline parallelism (#6528) * Restore logic for data-parallel communication with pipeline parallelism in GPT Signed-off-by: Tim Moon <tmoon@nvidia.com> * Support dynamic attention masks in GPT Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug typos Signed-off-by: Tim Moon <tmoon@nvidia.com> * Debug data iterator caching with interleaved pipeline parallelism Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples. Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Megatron-LM commit Signed-off-by: Tim Moon <tmoon@nvidia.com> * Distinguish between list of data iterators and data iterator that is a list Signed-off-by: Tim Moon <tmoon@nvidia.com> * Create dummy iters to satisy len checks Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Kludge while waiting for Megatron-LM update Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set transformers offline to avoid rate limiting Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601) * fix write Signed-off-by: fayejf <fayejf07@gmail.com> * decoding ctc Signed-off-by: fayejf <fayejf07@gmail.com> * temp set rnnt decoding return_best_hypothesis to true Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to transcribe_speech as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_rnnt as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_ctc as requested Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * reflect change in asr_evaluator Signed-off-by: fayejf <fayejf07@gmail.com> * reflect som and vahid comment Signed-off-by: fayejf <fayejf07@gmail.com> * remove return_best_hy=true in transcribe_speech Signed-off-by: fayejf <fayejf07@gmail.com> * no text skip Signed-off-by: fayejf <fayejf07@gmail.com> --------- Signed-off-by: fayejf <fayejf07@gmail.com> * 1. Added example conf YAML. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support in tensor_parallel. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add hat image to docs (#6619) Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * update core commit hash in readme (#6622) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Patch decoding for PC models (#6630) * Patch decoding logic for PC models Signed-off-by: smajumdar <titu1994@gmail.com> * Patch decoding logic for PC models Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> * Fix wer.py where 'errors' variable was not set (#6633) Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * fix att_context_size bug for older models. (#6635) Signed-off-by: Vahid <vnoroozi@nvidia.com> * Add megatron_core to requirements (#6639) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * remove from jenkins Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: ericharper <complex451@gmail.com> * remove dup (#6643) Signed-off-by: ericharper <complex451@gmail.com> * 1. Fixed config to use names, and added better error messages. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support to pass extra data to hiddens for loss computation. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Working on passing extra data to hiddnes. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed support in loading .nemo without hiddnes module. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Improved and fixed logging of validation and testing. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed training logging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed logging of hidden loss. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed logging names. 2. Added logging to hiddens and tokens loss. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed conflicts. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> --------- Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Signed-off-by: sam1373 <samuelkriman@gmail.com> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Micha Livne <mlivne@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Signed-off-by: Evelina <ebakhturina@nvidia.com> Signed-off-by: Taejin Park <tango4j@gmail.com> Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Signed-off-by: Greg Clark <grclark@nvidia.com> Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> Co-authored-by: Jocelyn <jocelynh@nvidia.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: bene-ges <61418381+bene-ges@users.noreply.github.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com> Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com> Co-authored-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> Co-authored-by: anteju <108555623+anteju@users.noreply.github.com> Co-authored-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: David <amosalla@asu.edu> Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: Samuel Kriman <samuelkriman@gmail.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com> Co-authored-by: Sean Naren <sean.narenthiran@gmail.com> Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com> Co-authored-by: Sean Naren <snarenthiran@nvidia.com> Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: Neha Tadimeti <ntadimeti@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: Jim O’Regan <jaoregan@tcd.ie> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: Taejin Park <tango4j@gmail.com> Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>
guyueh1
pushed a commit
to guyueh1/NeMo
that referenced
this pull request
Aug 14, 2023
* [TTS] bugfix for missing configs. (#4725) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * docs typo fix Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> * Fix pynini install in TTS tutorials (#4729) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Fix ASR notebooks (#4738) Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: smajumdar <smajumdar@nvidia.com> * Multilingual VAD model (#4734) * add ngc link Signed-off-by: fayejf <fayejf07@gmail.com> * add tuned VAD config on ASR data Signed-off-by: fayejf <fayejf07@gmail.com> * yaml note Signed-off-by: fayejf <fayejf07@gmail.com> * update vad asr notebook with mVAD Signed-off-by: fayejf <fayejf07@gmail.com> * update vad infer config comment Signed-off-by: fayejf <fayejf07@gmail.com> * fix Signed-off-by: fayejf <fayejf07@gmail.com> * mvad sd config for ch109 Signed-off-by: fayejf <fayejf07@gmail.com> * update sd readme Signed-off-by: fayejf <fayejf07@gmail.com> * add new mVAD model to doc Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * update sd tutorial with mVAD Signed-off-by: fayejf <fayejf07@gmail.com> * typo fix Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: fayejf <fayejf07@gmail.com> * publish pretrained itn t5 model for English (#4748) Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> * Updated docs and doc paths (#4754) * Updated docs and doc paths Signed-off-by: Virginia Adams <vadams@nvidia.com> * Update Multitask_Prompt_and_PTuning.ipynb * Update README.rst * Changed branch name to use single quotes Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> * fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762) * [TTS] updated config with a German IPA phoneme tokenizer (#4756) * [TTS] added a German IPA phoneme tokenizer * [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence. * [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer. * [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Update r1.11 to new heteronyms list (#4745) * Update configs to new heteronyms list * Remove old heteronyms list, add alt 'merchandise' pron to CMUdict * Update remaining references to old heteronyms list Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Update CMUdict with more recent 0.7b entries (#4768) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Install pynini in docker container (#4733) Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Fix tutorial formatting (#4778) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * [TTS] deprecated old scripts for ljspeech. (#4780) * deprecated old scripts for ljspeech. * removed relevent function calls in TTS docs. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * update branch and typos (#4788) Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> * Adding support for models trained with full context for cache-aware streaming. (#4687) * added support for models trained with full context. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * dropped seq_range Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed indexing in caching methods. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addressed comments. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> * Update megatron encoder decoder model to support py37 for colab (#4791) * [ASR] Add pretrained ASR models for Croatian (#4682) * [ASR] Add pretrained ASR models for Croatian Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Fix style for import Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * added/fixed export for Megatron models (#4712) * added/fixed export for Megatron models Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * fixed style Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * fixed FusedScaleMaskSoftmax in BioMegatron Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * included comments Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> * update branch for qa notebook Signed-off-by: ericharper <complex451@gmail.com> * Fix initializing weights from ptl ckpt with exclude (#4807) Signed-off-by: sam1373 <samuelkriman@gmail.com> Signed-off-by: sam1373 <samuelkriman@gmail.com> * Fix index error from addition of voiced_mask and p_voiced (#4811) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * T5 prompt learning fixes (#4771) * RPE, hidden size and config fixes Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update to reflect new config names Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Sentencepiece fixes Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix finetuning Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add encoder seq len to gpt Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add finetune eval script Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix name Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update Jenkinsfile Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update config Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix CI test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update check Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Backward compat Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update CI test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Split rank for Enc-Dec models Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Address comments Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com> * G2P docs (#4841) * g2p docs added Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix references Signed-off-by: ekmb <ebakhturina@nvidia.com> * address review feedback Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: ekmb <ebakhturina@nvidia.com> * Fix providing glue in seq2seq eval (#4843) * Fix providing glue in seq2seq eval Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Updated inference code and squad scripts (#4835) * Updated inference code and squad scripts Signed-off-by: Virginia Adams <vadams@nvidia.com> * Reverted GPT & T5 inference files back to use NLPDDPlugin Signed-off-by: Virginia Adams <vadams@nvidia.com> * Overwrite frozen LM to use fused adam Signed-off-by: Virginia Adams <vadams@nvidia.com> * Added padded vocab size Signed-off-by: Virginia Adams <vadams@nvidia.com> * Fixed val check interval value Signed-off-by: Virginia Adams <vadams@nvidia.com> * Python format fix Signed-off-by: Virginia Adams <vadams@nvidia.com> * Make t5 prompt learning preds write to file Signed-off-by: Virginia Adams <vadams@nvidia.com> * Added back dp=1 check Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> * Update README.rst * Fix uppercasing mismatch for IPA heteronyms (#4860) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790) * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Revert pin memory Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Sean Naren <sean.narenthiran@gmail.com> * Fix mha (#4866) * fix bug in mha forward function related to cache update return type Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * fix lgtm Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Co-authored-by: Sean Naren <snarenthiran@nvidia.com> * ipa bug fix (#4871) Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: ekmb <ebakhturina@nvidia.com> * Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884) * Fix nmt and ckpt_to_nemo Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * added utf8 encoding (#4892) Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> * 1. Applying the same patch to r1.11.0 (#4894) Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * Update tutorials.rst (#4897) * update readme with apex commit Signed-off-by: ericharper <complex451@gmail.com> * Add support for Apex distributed Adam optimizer with GPT-3 (#4487) * Add support for Apex distributed Adam optimizer with GPT-3 Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix bug in grad clipping with dist Adam Grad norm was computed over all params, not respecting model parallelism. Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix bug with DDP initialization Signed-off-by: Tim Moon <tmoon@nvidia.com> * Make distopt dependent on megatron_amp_o2 Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix code formatting Signed-off-by: Tim Moon <tmoon@nvidia.com> * Handle dist Adam in optimizer unit tests Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> * update readme Signed-off-by: ericharper <complex451@gmail.com> * update readme Signed-off-by: ericharper <complex451@gmail.com> * latent model support * 1. Debugging. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. * update branch Signed-off-by: ericharper <complex451@gmail.com> * fix replace_bos_with_pad not found (#6443) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Support Swiglu in TP PP Conversion (#6437) * Support Swiglu in TP PP Conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Guard activation Signed-off-by: smajumdar <titu1994@gmail.com> * Guard activation Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> * BERT pre-training mp fork to spawn (#6442) * change bert fork to spawn Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * num_workers=0 fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Meagtron encoder decoder fix for empty validation outputs (#6459) * 1. Meagtron encoder decoder fix for empty validation outputs. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. --------- Signed-off-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> * Added/updated new Conformer configs (#6426) * updated conf files. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * moved longconformer confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated readme. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated readme. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated batch sizes and added fastconformer ctc streaming configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated batch sizes. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added hybrid support. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added hybrid support. Signed-off-by: Vahid <vnoroozi@nvidia.com> --------- Signed-off-by: Vahid <vnoroozi@nvidia.com> * reduce workers on NMT CI (#6472) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * move to nvidia megatron repo (#6465) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Megatron KERPLE positional embeddings (#6478) * [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] FastPitch speaker encoder (#6417) * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove wemb Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix import Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add ConditionalInput Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix import error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix dataset error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove reference speaker embedding Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove SV encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix length type Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix append Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Move error msg Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add look-up into speaker encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add valueerror msg Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Move lookup Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove unused Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Rebase and Fix error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix spk encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Rename n_speakers Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Sharded manifests for tarred datasets (#6395) * testing sharded manifests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * compatibility Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * proper fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * adding flag tot convert_to_tarred_audio_dataset Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * shard_manifests conf param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * distributed checks Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * typo Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * typo Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * reversing manifest shards flag Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * excluding manifests from webdataset url expansion Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * explicit use of UTF-8 for manifest i/o Signed-off-by: Dima Rekesh <bmwshop@gmail.com> --------- Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> --------- Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * added tests for rampup batch size Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * fixed the typos Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * added assertions Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * changed assertion rules Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * deleted unused imports Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * changed tests for rampup batch size Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * updated rampup batch size tests Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * rampup batch size tests changes Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> --------- Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- Signed-off-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * updated README with the new agg_tokenizer_manifest flag Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * fixed typo in scripts/speech_recognition/code_switching/README.md Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * changed agg_tokenizer_manifest to is_lid_manifest Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> --------- Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add return encoding embedding option Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * update script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add rnnt and hat ngram decoding script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add some parameters Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * replace return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * generalization of scipt behavior Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add manual encoder_embeddings calculation Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> * fix rescoring description Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> --------- Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * fix replace_bos_with_pad not found (#6443) (#6450) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * reduce workers on NMT CI (#6472) (#6474) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * 1. Added KERPLE positional embeddings to encoder-decoder. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added a missing file. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: Micha Livne <mlivne@nvidia.com> Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: Jim O’Regan <jaoregan@tcd.ie> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * 1. Added external index sample. (#6462) Signed-off-by: Micha Livne <mlivne@nvidia.com> * Fix cache aware hybrid bugs (#6466) * Update README to add core installation (#6488) * update README for megatron-core Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Fix typos (#6494) Signed-off-by: smajumdar <titu1994@gmail.com> * fix broken links r1.18.0 (#6501) * fix broken links Signed-off-by: Evelina <ebakhturina@nvidia.com> * fix broken links Signed-off-by: Evelina <ebakhturina@nvidia.com> --------- Signed-off-by: Evelina <ebakhturina@nvidia.com> * 1. Fixed gaussian hidden transform. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Finished updating hidden loss for MIM. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix custom forward_torch_softmax (#6512) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500) * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0 Signed-off-by: Taejin Park <tango4j@gmail.com> * ignore keep_logits in FrameBatchASRLogits Signed-off-by: Taejin Park <tango4j@gmail.com> --------- Signed-off-by: Taejin Park <tango4j@gmail.com> * [TTS] fixed broken path. (#6514) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * 1. Added a hiddens module. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix typos (#6523) (#6539) * Fix typos Signed-off-by: smajumdar <titu1994@gmail.com> * Fix typos Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> (cherry picked from commit 5468077f5127be1a4c88065de2544f4268b9a6e4) * added back the fast emit section to the configs. (#6540) * added back the fast emit section to the configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added back the fast emit section to the configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> --------- Signed-off-by: Vahid <vnoroozi@nvidia.com> * Fix fp16 (#6543) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * fix (#6529) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * pass .scale instead of scaler object to core (#6545) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Change Megatron Enc Dec model to use persistent_workers (#6548) * persistent workers Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) * Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * updated ASR docs with the fastconformer hybrid checkpoints Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * added the fastconformer RNNT and CTC models Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> --------- Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * Add scores for FastConformer models (#6557) Signed-off-by: smajumdar <titu1994@gmail.com> * Patch transcribe and support offline transcribe for hybrid model (#6550) Signed-off-by: fayejf <fayejf07@gmail.com> * Not doing CastToFloat by default (#6524) * Not doing CastToFloat by default Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Added docustring Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Dummy commit Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * temp rtd fix (#6568) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py for speedup (#6565) * Update manifest.py Re-order the checks for faster processing audio filepaths that are already absolute paths Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update manifest.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> --------- Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * Turn autocast off when precision is fp32 (#6554) * Turn autocast off when precision is fp32 Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * address review Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * merge Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * More streaming conformer export fixes (#6567) Signed-off-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * Fix batch size reconf for T5 FT for multi-validation (#6582) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Updated Megatron LM encoder/decoder to use cfg for hiddens. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added support to register externalhidden loss / transforms. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make tensor split contiguous (#6580) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Patches from main to r1.18.0 for Virtual Parallel (#6592) * Add interleaved pp support (#6498) * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Switch to megatron core Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 892987169ef277f328e15b71a5a0c9bd961c8ee7) * Add patches for Virtual Parallel conversion (#6589) * Add patches for Virtual Parllel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 1d813a372ab51688e3af6395d905a4c0366ffd23) * Documentation for ASR-TTS models (#6594) * Add docs about hybrid ASR-TTS models Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about text-only datasets Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about ASR-TTS checkpoints Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about ASR-TTS configs and training Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Clean up Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * ASR-TTS docs: add to api, fix imports Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Clean up Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Wrap optional import Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Revert general ASR import Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> --------- Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Update SDP docs (#6485) * add info about SDP e.g. processor classes in docs Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * add link to SDP docs in README Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * address code review comments and add SDP overview diagram Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * Fix spelling typo Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> --------- Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * Create dummy iters to satisy len checks (#6600) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * Restore GPT support for interleaved pipeline parallelism (#6528) * Restore logic for data-parallel communication with pipeline parallelism in GPT Signed-off-by: Tim Moon <tmoon@nvidia.com> * Support dynamic attention masks in GPT Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug typos Signed-off-by: Tim Moon <tmoon@nvidia.com> * Debug data iterator caching with interleaved pipeline parallelism Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples. Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Megatron-LM commit Signed-off-by: Tim Moon <tmoon@nvidia.com> * Distinguish between list of data iterators and data iterator that is a list Signed-off-by: Tim Moon <tmoon@nvidia.com> * Create dummy iters to satisy len checks Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Kludge while waiting for Megatron-LM update Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set transformers offline to avoid rate limiting Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601) * fix write Signed-off-by: fayejf <fayejf07@gmail.com> * decoding ctc Signed-off-by: fayejf <fayejf07@gmail.com> * temp set rnnt decoding return_best_hypothesis to true Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to transcribe_speech as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_rnnt as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_ctc as requested Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * reflect change in asr_evaluator Signed-off-by: fayejf <fayejf07@gmail.com> * reflect som and vahid comment Signed-off-by: fayejf <fayejf07@gmail.com> * remove return_best_hy=true in transcribe_speech Signed-off-by: fayejf <fayejf07@gmail.com> * no text skip Signed-off-by: fayejf <fayejf07@gmail.com> --------- Signed-off-by: fayejf <fayejf07@gmail.com> * 1. Added example conf YAML. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support in tensor_parallel. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add hat image to docs (#6619) Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * update core commit hash in readme (#6622) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Patch decoding for PC models (#6630) * Patch decoding logic for PC models Signed-off-by: smajumdar <titu1994@gmail.com> * Patch decoding logic for PC models Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> * Fix wer.py where 'errors' variable was not set (#6633) Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * fix att_context_size bug for older models. (#6635) Signed-off-by: Vahid <vnoroozi@nvidia.com> * Add megatron_core to requirements (#6639) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * remove from jenkins Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: ericharper <complex451@gmail.com> * remove dup (#6643) Signed-off-by: ericharper <complex451@gmail.com> * 1. Fixed config to use names, and added better error messages. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support to pass extra data to hiddens for loss computation. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Working on passing extra data to hiddnes. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed support in loading .nemo without hiddnes module. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Improved and fixed logging of validation and testing. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed training logging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed logging of hidden loss. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed logging names. 2. Added logging to hiddens and tokens loss. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed conflicts. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> --------- Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Signed-off-by: sam1373 <samuelkriman@gmail.com> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Micha Livne <mlivne@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Signed-off-by: Evelina <ebakhturina@nvidia.com> Signed-off-by: Taejin Park <tango4j@gmail.com> Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Signed-off-by: Greg Clark <grclark@nvidia.com> Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> Co-authored-by: Jocelyn <jocelynh@nvidia.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: bene-ges <61418381+bene-ges@users.noreply.github.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com> Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com> Co-authored-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> Co-authored-by: anteju <108555623+anteju@users.noreply.github.com> Co-authored-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: David <amosalla@asu.edu> Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: Samuel Kriman <samuelkriman@gmail.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com> Co-authored-by: Sean Naren <sean.narenthiran@gmail.com> Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com> Co-authored-by: Sean Naren <snarenthiran@nvidia.com> Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: Neha Tadimeti <ntadimeti@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: Jim O’Regan <jaoregan@tcd.ie> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: Taejin Park <tango4j@gmail.com> Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>
dorotat-nv
pushed a commit
to dorotat-nv/NeMo
that referenced
this pull request
Aug 24, 2023
* [TTS] bugfix for missing configs. (#4725) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * docs typo fix Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> * Fix pynini install in TTS tutorials (#4729) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Fix ASR notebooks (#4738) Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: smajumdar <smajumdar@nvidia.com> * Multilingual VAD model (#4734) * add ngc link Signed-off-by: fayejf <fayejf07@gmail.com> * add tuned VAD config on ASR data Signed-off-by: fayejf <fayejf07@gmail.com> * yaml note Signed-off-by: fayejf <fayejf07@gmail.com> * update vad asr notebook with mVAD Signed-off-by: fayejf <fayejf07@gmail.com> * update vad infer config comment Signed-off-by: fayejf <fayejf07@gmail.com> * fix Signed-off-by: fayejf <fayejf07@gmail.com> * mvad sd config for ch109 Signed-off-by: fayejf <fayejf07@gmail.com> * update sd readme Signed-off-by: fayejf <fayejf07@gmail.com> * add new mVAD model to doc Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * update sd tutorial with mVAD Signed-off-by: fayejf <fayejf07@gmail.com> * typo fix Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: fayejf <fayejf07@gmail.com> * publish pretrained itn t5 model for English (#4748) Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> * Updated docs and doc paths (#4754) * Updated docs and doc paths Signed-off-by: Virginia Adams <vadams@nvidia.com> * Update Multitask_Prompt_and_PTuning.ipynb * Update README.rst * Changed branch name to use single quotes Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> * fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762) * [TTS] updated config with a German IPA phoneme tokenizer (#4756) * [TTS] added a German IPA phoneme tokenizer * [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence. * [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer. * [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Update r1.11 to new heteronyms list (#4745) * Update configs to new heteronyms list * Remove old heteronyms list, add alt 'merchandise' pron to CMUdict * Update remaining references to old heteronyms list Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Update CMUdict with more recent 0.7b entries (#4768) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Install pynini in docker container (#4733) Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Fix tutorial formatting (#4778) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * [TTS] deprecated old scripts for ljspeech. (#4780) * deprecated old scripts for ljspeech. * removed relevent function calls in TTS docs. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * update branch and typos (#4788) Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> * Adding support for models trained with full context for cache-aware streaming. (#4687) * added support for models trained with full context. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * dropped seq_range Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed indexing in caching methods. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addressed comments. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> * Update megatron encoder decoder model to support py37 for colab (#4791) * [ASR] Add pretrained ASR models for Croatian (#4682) * [ASR] Add pretrained ASR models for Croatian Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Fix style for import Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * added/fixed export for Megatron models (#4712) * added/fixed export for Megatron models Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * fixed style Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * fixed FusedScaleMaskSoftmax in BioMegatron Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * included comments Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> * update branch for qa notebook Signed-off-by: ericharper <complex451@gmail.com> * Fix initializing weights from ptl ckpt with exclude (#4807) Signed-off-by: sam1373 <samuelkriman@gmail.com> Signed-off-by: sam1373 <samuelkriman@gmail.com> * Fix index error from addition of voiced_mask and p_voiced (#4811) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * T5 prompt learning fixes (#4771) * RPE, hidden size and config fixes Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update to reflect new config names Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Sentencepiece fixes Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix finetuning Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add encoder seq len to gpt Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add finetune eval script Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix name Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update Jenkinsfile Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update config Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix CI test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update check Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Backward compat Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update CI test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Split rank for Enc-Dec models Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Address comments Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com> * G2P docs (#4841) * g2p docs added Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix references Signed-off-by: ekmb <ebakhturina@nvidia.com> * address review feedback Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: ekmb <ebakhturina@nvidia.com> * Fix providing glue in seq2seq eval (#4843) * Fix providing glue in seq2seq eval Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Updated inference code and squad scripts (#4835) * Updated inference code and squad scripts Signed-off-by: Virginia Adams <vadams@nvidia.com> * Reverted GPT & T5 inference files back to use NLPDDPlugin Signed-off-by: Virginia Adams <vadams@nvidia.com> * Overwrite frozen LM to use fused adam Signed-off-by: Virginia Adams <vadams@nvidia.com> * Added padded vocab size Signed-off-by: Virginia Adams <vadams@nvidia.com> * Fixed val check interval value Signed-off-by: Virginia Adams <vadams@nvidia.com> * Python format fix Signed-off-by: Virginia Adams <vadams@nvidia.com> * Make t5 prompt learning preds write to file Signed-off-by: Virginia Adams <vadams@nvidia.com> * Added back dp=1 check Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> * Update README.rst * Fix uppercasing mismatch for IPA heteronyms (#4860) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790) * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Revert pin memory Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Sean Naren <sean.narenthiran@gmail.com> * Fix mha (#4866) * fix bug in mha forward function related to cache update return type Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * fix lgtm Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Co-authored-by: Sean Naren <snarenthiran@nvidia.com> * ipa bug fix (#4871) Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: ekmb <ebakhturina@nvidia.com> * Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884) * Fix nmt and ckpt_to_nemo Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * added utf8 encoding (#4892) Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> * 1. Applying the same patch to r1.11.0 (#4894) Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * Update tutorials.rst (#4897) * update readme with apex commit Signed-off-by: ericharper <complex451@gmail.com> * Add support for Apex distributed Adam optimizer with GPT-3 (#4487) * Add support for Apex distributed Adam optimizer with GPT-3 Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix bug in grad clipping with dist Adam Grad norm was computed over all params, not respecting model parallelism. Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix bug with DDP initialization Signed-off-by: Tim Moon <tmoon@nvidia.com> * Make distopt dependent on megatron_amp_o2 Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix code formatting Signed-off-by: Tim Moon <tmoon@nvidia.com> * Handle dist Adam in optimizer unit tests Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> * update readme Signed-off-by: ericharper <complex451@gmail.com> * update readme Signed-off-by: ericharper <complex451@gmail.com> * latent model support * 1. Debugging. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. * update branch Signed-off-by: ericharper <complex451@gmail.com> * fix replace_bos_with_pad not found (#6443) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Support Swiglu in TP PP Conversion (#6437) * Support Swiglu in TP PP Conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Guard activation Signed-off-by: smajumdar <titu1994@gmail.com> * Guard activation Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> * BERT pre-training mp fork to spawn (#6442) * change bert fork to spawn Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * num_workers=0 fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Meagtron encoder decoder fix for empty validation outputs (#6459) * 1. Meagtron encoder decoder fix for empty validation outputs. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. --------- Signed-off-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> * Added/updated new Conformer configs (#6426) * updated conf files. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * moved longconformer confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated readme. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated readme. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated batch sizes and added fastconformer ctc streaming configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated batch sizes. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added hybrid support. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added hybrid support. Signed-off-by: Vahid <vnoroozi@nvidia.com> --------- Signed-off-by: Vahid <vnoroozi@nvidia.com> * reduce workers on NMT CI (#6472) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * move to nvidia megatron repo (#6465) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Megatron KERPLE positional embeddings (#6478) * [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] FastPitch speaker encoder (#6417) * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove wemb Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix import Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add ConditionalInput Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix import error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix dataset error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove reference speaker embedding Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove SV encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix length type Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix append Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Move error msg Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add look-up into speaker encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add valueerror msg Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Move lookup Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove unused Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Rebase and Fix error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix spk encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Rename n_speakers Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Sharded manifests for tarred datasets (#6395) * testing sharded manifests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * compatibility Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * proper fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * adding flag tot convert_to_tarred_audio_dataset Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * shard_manifests conf param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * distributed checks Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * typo Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * typo Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * reversing manifest shards flag Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * excluding manifests from webdataset url expansion Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * explicit use of UTF-8 for manifest i/o Signed-off-by: Dima Rekesh <bmwshop@gmail.com> --------- Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> --------- Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * added tests for rampup batch size Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * fixed the typos Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * added assertions Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * changed assertion rules Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * deleted unused imports Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * changed tests for rampup batch size Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * updated rampup batch size tests Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * rampup batch size tests changes Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> --------- Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- Signed-off-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * updated README with the new agg_tokenizer_manifest flag Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * fixed typo in scripts/speech_recognition/code_switching/README.md Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * changed agg_tokenizer_manifest to is_lid_manifest Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> --------- Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add return encoding embedding option Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * update script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add rnnt and hat ngram decoding script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add some parameters Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * replace return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * generalization of scipt behavior Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add manual encoder_embeddings calculation Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> * fix rescoring description Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> --------- Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * fix replace_bos_with_pad not found (#6443) (#6450) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * reduce workers on NMT CI (#6472) (#6474) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * 1. Added KERPLE positional embeddings to encoder-decoder. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added a missing file. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: Micha Livne <mlivne@nvidia.com> Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: Jim O’Regan <jaoregan@tcd.ie> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * 1. Added external index sample. (#6462) Signed-off-by: Micha Livne <mlivne@nvidia.com> * Fix cache aware hybrid bugs (#6466) * Update README to add core installation (#6488) * update README for megatron-core Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Fix typos (#6494) Signed-off-by: smajumdar <titu1994@gmail.com> * fix broken links r1.18.0 (#6501) * fix broken links Signed-off-by: Evelina <ebakhturina@nvidia.com> * fix broken links Signed-off-by: Evelina <ebakhturina@nvidia.com> --------- Signed-off-by: Evelina <ebakhturina@nvidia.com> * 1. Fixed gaussian hidden transform. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Finished updating hidden loss for MIM. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix custom forward_torch_softmax (#6512) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500) * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0 Signed-off-by: Taejin Park <tango4j@gmail.com> * ignore keep_logits in FrameBatchASRLogits Signed-off-by: Taejin Park <tango4j@gmail.com> --------- Signed-off-by: Taejin Park <tango4j@gmail.com> * [TTS] fixed broken path. (#6514) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * 1. Added a hiddens module. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix typos (#6523) (#6539) * Fix typos Signed-off-by: smajumdar <titu1994@gmail.com> * Fix typos Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> (cherry picked from commit 5468077f5127be1a4c88065de2544f4268b9a6e4) * added back the fast emit section to the configs. (#6540) * added back the fast emit section to the configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added back the fast emit section to the configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> --------- Signed-off-by: Vahid <vnoroozi@nvidia.com> * Fix fp16 (#6543) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * fix (#6529) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * pass .scale instead of scaler object to core (#6545) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Change Megatron Enc Dec model to use persistent_workers (#6548) * persistent workers Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) * Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * updated ASR docs with the fastconformer hybrid checkpoints Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * added the fastconformer RNNT and CTC models Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> --------- Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * Add scores for FastConformer models (#6557) Signed-off-by: smajumdar <titu1994@gmail.com> * Patch transcribe and support offline transcribe for hybrid model (#6550) Signed-off-by: fayejf <fayejf07@gmail.com> * Not doing CastToFloat by default (#6524) * Not doing CastToFloat by default Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Added docustring Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Dummy commit Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * temp rtd fix (#6568) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py for speedup (#6565) * Update manifest.py Re-order the checks for faster processing audio filepaths that are already absolute paths Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update manifest.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> --------- Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * Turn autocast off when precision is fp32 (#6554) * Turn autocast off when precision is fp32 Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * address review Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * merge Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * More streaming conformer export fixes (#6567) Signed-off-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * Fix batch size reconf for T5 FT for multi-validation (#6582) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Updated Megatron LM encoder/decoder to use cfg for hiddens. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added support to register externalhidden loss / transforms. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make tensor split contiguous (#6580) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Patches from main to r1.18.0 for Virtual Parallel (#6592) * Add interleaved pp support (#6498) * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Switch to megatron core Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 892987169ef277f328e15b71a5a0c9bd961c8ee7) * Add patches for Virtual Parallel conversion (#6589) * Add patches for Virtual Parllel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 1d813a372ab51688e3af6395d905a4c0366ffd23) * Documentation for ASR-TTS models (#6594) * Add docs about hybrid ASR-TTS models Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about text-only datasets Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about ASR-TTS checkpoints Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about ASR-TTS configs and training Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Clean up Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * ASR-TTS docs: add to api, fix imports Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Clean up Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Wrap optional import Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Revert general ASR import Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> --------- Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Update SDP docs (#6485) * add info about SDP e.g. processor classes in docs Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * add link to SDP docs in README Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * address code review comments and add SDP overview diagram Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * Fix spelling typo Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> --------- Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * Create dummy iters to satisy len checks (#6600) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * Restore GPT support for interleaved pipeline parallelism (#6528) * Restore logic for data-parallel communication with pipeline parallelism in GPT Signed-off-by: Tim Moon <tmoon@nvidia.com> * Support dynamic attention masks in GPT Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug typos Signed-off-by: Tim Moon <tmoon@nvidia.com> * Debug data iterator caching with interleaved pipeline parallelism Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples. Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Megatron-LM commit Signed-off-by: Tim Moon <tmoon@nvidia.com> * Distinguish between list of data iterators and data iterator that is a list Signed-off-by: Tim Moon <tmoon@nvidia.com> * Create dummy iters to satisy len checks Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Kludge while waiting for Megatron-LM update Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set transformers offline to avoid rate limiting Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601) * fix write Signed-off-by: fayejf <fayejf07@gmail.com> * decoding ctc Signed-off-by: fayejf <fayejf07@gmail.com> * temp set rnnt decoding return_best_hypothesis to true Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to transcribe_speech as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_rnnt as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_ctc as requested Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * reflect change in asr_evaluator Signed-off-by: fayejf <fayejf07@gmail.com> * reflect som and vahid comment Signed-off-by: fayejf <fayejf07@gmail.com> * remove return_best_hy=true in transcribe_speech Signed-off-by: fayejf <fayejf07@gmail.com> * no text skip Signed-off-by: fayejf <fayejf07@gmail.com> --------- Signed-off-by: fayejf <fayejf07@gmail.com> * 1. Added example conf YAML. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support in tensor_parallel. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add hat image to docs (#6619) Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * update core commit hash in readme (#6622) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Patch decoding for PC models (#6630) * Patch decoding logic for PC models Signed-off-by: smajumdar <titu1994@gmail.com> * Patch decoding logic for PC models Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> * Fix wer.py where 'errors' variable was not set (#6633) Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * fix att_context_size bug for older models. (#6635) Signed-off-by: Vahid <vnoroozi@nvidia.com> * Add megatron_core to requirements (#6639) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * remove from jenkins Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: ericharper <complex451@gmail.com> * remove dup (#6643) Signed-off-by: ericharper <complex451@gmail.com> * 1. Fixed config to use names, and added better error messages. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support to pass extra data to hiddens for loss computation. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Working on passing extra data to hiddnes. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed support in loading .nemo without hiddnes module. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Improved and fixed logging of validation and testing. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed training logging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed logging of hidden loss. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed logging names. 2. Added logging to hiddens and tokens loss. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed conflicts. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> --------- Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Signed-off-by: sam1373 <samuelkriman@gmail.com> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Micha Livne <mlivne@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Signed-off-by: Evelina <ebakhturina@nvidia.com> Signed-off-by: Taejin Park <tango4j@gmail.com> Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Signed-off-by: Greg Clark <grclark@nvidia.com> Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> Co-authored-by: Jocelyn <jocelynh@nvidia.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: bene-ges <61418381+bene-ges@users.noreply.github.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com> Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com> Co-authored-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> Co-authored-by: anteju <108555623+anteju@users.noreply.github.com> Co-authored-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: David <amosalla@asu.edu> Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: Samuel Kriman <samuelkriman@gmail.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com> Co-authored-by: Sean Naren <sean.narenthiran@gmail.com> Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com> Co-authored-by: Sean Naren <snarenthiran@nvidia.com> Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: Neha Tadimeti <ntadimeti@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: Jim O’Regan <jaoregan@tcd.ie> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: Taejin Park <tango4j@gmail.com> Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com> Signed-off-by: dorotat <dorotat@nvidia.com>
yaoyu-33
pushed a commit
that referenced
this pull request
Oct 16, 2023
rohitrango
pushed a commit
to rohitrango/NeMo
that referenced
this pull request
Jun 25, 2024
* [TTS] bugfix for missing configs. (#4725) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * docs typo fix Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> * Fix pynini install in TTS tutorials (#4729) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Fix ASR notebooks (#4738) Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: smajumdar <smajumdar@nvidia.com> * Multilingual VAD model (#4734) * add ngc link Signed-off-by: fayejf <fayejf07@gmail.com> * add tuned VAD config on ASR data Signed-off-by: fayejf <fayejf07@gmail.com> * yaml note Signed-off-by: fayejf <fayejf07@gmail.com> * update vad asr notebook with mVAD Signed-off-by: fayejf <fayejf07@gmail.com> * update vad infer config comment Signed-off-by: fayejf <fayejf07@gmail.com> * fix Signed-off-by: fayejf <fayejf07@gmail.com> * mvad sd config for ch109 Signed-off-by: fayejf <fayejf07@gmail.com> * update sd readme Signed-off-by: fayejf <fayejf07@gmail.com> * add new mVAD model to doc Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * update sd tutorial with mVAD Signed-off-by: fayejf <fayejf07@gmail.com> * typo fix Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: fayejf <fayejf07@gmail.com> * publish pretrained itn t5 model for English (#4748) Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> * Updated docs and doc paths (#4754) * Updated docs and doc paths Signed-off-by: Virginia Adams <vadams@nvidia.com> * Update Multitask_Prompt_and_PTuning.ipynb * Update README.rst * Changed branch name to use single quotes Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> * fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762) * [TTS] updated config with a German IPA phoneme tokenizer (#4756) * [TTS] added a German IPA phoneme tokenizer * [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence. * [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer. * [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Update r1.11 to new heteronyms list (#4745) * Update configs to new heteronyms list * Remove old heteronyms list, add alt 'merchandise' pron to CMUdict * Update remaining references to old heteronyms list Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Update CMUdict with more recent 0.7b entries (#4768) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * Install pynini in docker container (#4733) Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Fix tutorial formatting (#4778) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * [TTS] deprecated old scripts for ljspeech. (#4780) * deprecated old scripts for ljspeech. * removed relevent function calls in TTS docs. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * update branch and typos (#4788) Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> * Adding support for models trained with full context for cache-aware streaming. (#4687) * added support for models trained with full context. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * dropped seq_range Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed indexing in caching methods. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addressed comments. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * change frame-wise to cache-aware. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed code style. Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> * Update megatron encoder decoder model to support py37 for colab (#4791) * [ASR] Add pretrained ASR models for Croatian (#4682) * [ASR] Add pretrained ASR models for Croatian Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Fix style for import Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * added/fixed export for Megatron models (#4712) * added/fixed export for Megatron models Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * fixed style Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * fixed FusedScaleMaskSoftmax in BioMegatron Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> * included comments Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> * update branch for qa notebook Signed-off-by: ericharper <complex451@gmail.com> * Fix initializing weights from ptl ckpt with exclude (#4807) Signed-off-by: sam1373 <samuelkriman@gmail.com> Signed-off-by: sam1373 <samuelkriman@gmail.com> * Fix index error from addition of voiced_mask and p_voiced (#4811) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * T5 prompt learning fixes (#4771) * RPE, hidden size and config fixes Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update to reflect new config names Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Sentencepiece fixes Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix finetuning Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add encoder seq len to gpt Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add finetune eval script Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix name Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update Jenkinsfile Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update config Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix CI test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update check Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Backward compat Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update CI test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Split rank for Enc-Dec models Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Address comments Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com> * G2P docs (#4841) * g2p docs added Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix references Signed-off-by: ekmb <ebakhturina@nvidia.com> * address review feedback Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: ekmb <ebakhturina@nvidia.com> * Fix providing glue in seq2seq eval (#4843) * Fix providing glue in seq2seq eval Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Updated inference code and squad scripts (#4835) * Updated inference code and squad scripts Signed-off-by: Virginia Adams <vadams@nvidia.com> * Reverted GPT & T5 inference files back to use NLPDDPlugin Signed-off-by: Virginia Adams <vadams@nvidia.com> * Overwrite frozen LM to use fused adam Signed-off-by: Virginia Adams <vadams@nvidia.com> * Added padded vocab size Signed-off-by: Virginia Adams <vadams@nvidia.com> * Fixed val check interval value Signed-off-by: Virginia Adams <vadams@nvidia.com> * Python format fix Signed-off-by: Virginia Adams <vadams@nvidia.com> * Make t5 prompt learning preds write to file Signed-off-by: Virginia Adams <vadams@nvidia.com> * Added back dp=1 check Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> * Update README.rst * Fix uppercasing mismatch for IPA heteronyms (#4860) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790) * Set workers to 0 for validation and test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Revert pin memory Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Sean Naren <sean.narenthiran@gmail.com> * Fix mha (#4866) * fix bug in mha forward function related to cache update return type Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * fix lgtm Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Co-authored-by: Sean Naren <snarenthiran@nvidia.com> * ipa bug fix (#4871) Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: ekmb <ebakhturina@nvidia.com> * Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884) * Fix nmt and ckpt_to_nemo Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * added utf8 encoding (#4892) Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> * 1. Applying the same patch to r1.11.0 (#4894) Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * Update tutorials.rst (#4897) * update readme with apex commit Signed-off-by: ericharper <complex451@gmail.com> * Add support for Apex distributed Adam optimizer with GPT-3 (#4487) * Add support for Apex distributed Adam optimizer with GPT-3 Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix bug in grad clipping with dist Adam Grad norm was computed over all params, not respecting model parallelism. Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix bug with DDP initialization Signed-off-by: Tim Moon <tmoon@nvidia.com> * Make distopt dependent on megatron_amp_o2 Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fix code formatting Signed-off-by: Tim Moon <tmoon@nvidia.com> * Handle dist Adam in optimizer unit tests Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> * update readme Signed-off-by: ericharper <complex451@gmail.com> * update readme Signed-off-by: ericharper <complex451@gmail.com> * latent model support * 1. Debugging. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. * update branch Signed-off-by: ericharper <complex451@gmail.com> * fix replace_bos_with_pad not found (#6443) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Support Swiglu in TP PP Conversion (#6437) * Support Swiglu in TP PP Conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Guard activation Signed-off-by: smajumdar <titu1994@gmail.com> * Guard activation Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> * BERT pre-training mp fork to spawn (#6442) * change bert fork to spawn Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * num_workers=0 fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Meagtron encoder decoder fix for empty validation outputs (#6459) * 1. Meagtron encoder decoder fix for empty validation outputs. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. --------- Signed-off-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> * Added/updated new Conformer configs (#6426) * updated conf files. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * moved longconformer confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated readme. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated readme. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated batch sizes and added fastconformer ctc streaming configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated batch sizes. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added hybrid support. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added hybrid support. Signed-off-by: Vahid <vnoroozi@nvidia.com> --------- Signed-off-by: Vahid <vnoroozi@nvidia.com> * reduce workers on NMT CI (#6472) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * move to nvidia megatron repo (#6465) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Megatron KERPLE positional embeddings (#6478) * [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416) --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] whitelist broken path fix. (#6412) * [TTS] whitelist broken path fix. Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [TTS] FastPitch speaker encoder (#6417) * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove wemb Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix import Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore aligner loss Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add ConditionalInput Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix error and support pre-trained config Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change copyright and random weight test Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix import error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add initial codes Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix dataset error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove reference speaker embedding Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove SV encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix length type Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix append Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Move error msg Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add look-up into speaker encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Add valueerror msg Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Move lookup Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Remove unused Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Rebase and Fix error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Fix spk encoder Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Rename n_speakers Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * Follow comments Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix n_speakers None error Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Sharded manifests for tarred datasets (#6395) * testing sharded manifests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * compatibility Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * proper fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * adding flag tot convert_to_tarred_audio_dataset Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * shard_manifests conf param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * propagating the shard_manifests param Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * distributed checks Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * typo Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * typo Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * fixes Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes based on PR comments and tests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes to convert_to_tarred_audio_dataset.py Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * reversing manifest shards flag Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * excluding manifests from webdataset url expansion Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * expand manifest paths before attempting to cache from datastore Signed-off-by: Dima Rekesh <bmwshop@gmail.com> * explicit use of UTF-8 for manifest i/o Signed-off-by: Dima Rekesh <bmwshop@gmail.com> --------- Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update wfst_text_normalization.rst (#6374) Add Hungarian (incoming in NeMo-text-processing) Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> * Support Swiglu in TP PP Conversion (#6437) (#6451) * Support Swiglu in TP PP Conversion * Guard activation * Guard activation --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * Update NeMo_TTS_Primer.ipynb (#6436) * Update NeMo_TTS_Primer.ipynb Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch. Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> * Update NeMo_TTS_Primer.ipynb Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point. Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> --------- Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> * add rampup batch size support for Megatron GPT (#6424) * added rampup batch size support Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * added tests for rampup batch size Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * fixed the typos Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * added assertions Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * changed assertion rules Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * deleted unused imports Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * changed tests for rampup batch size Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * updated rampup batch size tests Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed styling Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> * rampup batch size tests changes Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> --------- Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461) * 1. Meagtron encoder decoder fix for empty validation outputs. * 1. Debugging. --------- Signed-off-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> * Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448) * added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * updated README with the new agg_tokenizer_manifest flag Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * fixed typo in scripts/speech_recognition/code_switching/README.md Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> * changed agg_tokenizer_manifest to is_lid_manifest Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> --------- Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> * Added/updated new Conformer configs (#6426) (#6467) * Update script for ngram rnnt and hat beam search decoding (#6370) * add rnnt ngram beamsearch script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add return encoding embedding option Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * update script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add rnnt and hat ngram decoding script Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add some parameters Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add return_encoder_embeddings parameter to RNNTDecodingConfig Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * replace return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * generalization of scipt behavior Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * remove return_encoder_embeddings parameter Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add manual encoder_embeddings calculation Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix beam_width value to 8 Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> * fix rescoring description Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> --------- Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * BERT pre-training mp fork to spawn (#6442) (#6454) * change bert fork to spawn * num_workers=0 fix --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * fix replace_bos_with_pad not found (#6443) (#6450) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * reduce workers on NMT CI (#6472) (#6474) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * 1. Added KERPLE positional embeddings to encoder-decoder. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added a missing file. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixing commits. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. * 1. Debugging. * 1. Debugging. * 1. Debugging. --------- Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: Micha Livne <mlivne@nvidia.com> Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: Jim O’Regan <jaoregan@tcd.ie> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * 1. Added external index sample. (#6462) Signed-off-by: Micha Livne <mlivne@nvidia.com> * Fix cache aware hybrid bugs (#6466) * Update README to add core installation (#6488) * update README for megatron-core Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Fix typos (#6494) Signed-off-by: smajumdar <titu1994@gmail.com> * fix broken links r1.18.0 (#6501) * fix broken links Signed-off-by: Evelina <ebakhturina@nvidia.com> * fix broken links Signed-off-by: Evelina <ebakhturina@nvidia.com> --------- Signed-off-by: Evelina <ebakhturina@nvidia.com> * 1. Fixed gaussian hidden transform. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Finished updating hidden loss for MIM. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix custom forward_torch_softmax (#6512) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500) * [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0 Signed-off-by: Taejin Park <tango4j@gmail.com> * ignore keep_logits in FrameBatchASRLogits Signed-off-by: Taejin Park <tango4j@gmail.com> --------- Signed-off-by: Taejin Park <tango4j@gmail.com> * [TTS] fixed broken path. (#6514) Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> * 1. Added a hiddens module. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix typos (#6523) (#6539) * Fix typos Signed-off-by: smajumdar <titu1994@gmail.com> * Fix typos Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> (cherry picked from commit 7fa47bfcd359f1ea80ad056d8bb2d78f97c6206a) * added back the fast emit section to the configs. (#6540) * added back the fast emit section to the configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added back the fast emit section to the configs. Signed-off-by: Vahid <vnoroozi@nvidia.com> --------- Signed-off-by: Vahid <vnoroozi@nvidia.com> * Fix fp16 (#6543) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * fix (#6529) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * pass .scale instead of scaler object to core (#6545) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Change Megatron Enc Dec model to use persistent_workers (#6548) * persistent workers Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * fix Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> * Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) * Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * updated ASR docs with the fastconformer hybrid checkpoints Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * added the fastconformer RNNT and CTC models Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> --------- Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> * Add scores for FastConformer models (#6557) Signed-off-by: smajumdar <titu1994@gmail.com> * Patch transcribe and support offline transcribe for hybrid model (#6550) Signed-off-by: fayejf <fayejf07@gmail.com> * Not doing CastToFloat by default (#6524) * Not doing CastToFloat by default Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Added docustring Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Dummy commit Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * temp rtd fix (#6568) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update manifest.py for speedup (#6565) * Update manifest.py Re-order the checks for faster processing audio filepaths that are already absolute paths Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * Update manifest.py Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> --------- Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * Turn autocast off when precision is fp32 (#6554) * Turn autocast off when precision is fp32 Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * address review Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * merge Signed-off-by: Abhinav Khattar <aklife97@gmail.com> --------- Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> * More streaming conformer export fixes (#6567) Signed-off-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> * Fix batch size reconf for T5 FT for multi-validation (#6582) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Updated Megatron LM encoder/decoder to use cfg for hiddens. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added support to register externalhidden loss / transforms. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make tensor split contiguous (#6580) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Patches from main to r1.18.0 for Virtual Parallel (#6592) * Add interleaved pp support (#6498) * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Add support for Virtual Pipeline Parallel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * Switch to megatron core Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit fa9ab5b41301c9d92c97bf37894f9c94de6b91b5) * Add patches for Virtual Parallel conversion (#6589) * Add patches for Virtual Parllel conversion Signed-off-by: smajumdar <titu1994@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: smajumdar <titu1994@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 4d639ef36b10d8536db02bc47ea30f4acbc061bb) * Documentation for ASR-TTS models (#6594) * Add docs about hybrid ASR-TTS models Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about text-only datasets Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about ASR-TTS checkpoints Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Add docs about ASR-TTS configs and training Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Clean up Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * ASR-TTS docs: add to api, fix imports Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Clean up Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Wrap optional import Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Revert general ASR import Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> --------- Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> * Update SDP docs (#6485) * add info about SDP e.g. processor classes in docs Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * add link to SDP docs in README Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * address code review comments and add SDP overview diagram Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * Fix spelling typo Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> --------- Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> * Create dummy iters to satisy len checks (#6600) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * Restore GPT support for interleaved pipeline parallelism (#6528) * Restore logic for data-parallel communication with pipeline parallelism in GPT Signed-off-by: Tim Moon <tmoon@nvidia.com> * Support dynamic attention masks in GPT Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Debug typos Signed-off-by: Tim Moon <tmoon@nvidia.com> * Debug data iterator caching with interleaved pipeline parallelism Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples. Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Megatron-LM commit Signed-off-by: Tim Moon <tmoon@nvidia.com> * Distinguish between list of data iterators and data iterator that is a list Signed-off-by: Tim Moon <tmoon@nvidia.com> * Create dummy iters to satisy len checks Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Kludge while waiting for Megatron-LM update Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set transformers offline to avoid rate limiting Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> * Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601) * fix write Signed-off-by: fayejf <fayejf07@gmail.com> * decoding ctc Signed-off-by: fayejf <fayejf07@gmail.com> * temp set rnnt decoding return_best_hypothesis to true Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to transcribe_speech as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_rnnt as requested Signed-off-by: fayejf <fayejf07@gmail.com> * add wer cal back to speech_to_text_buffered_infer_ctc as requested Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * reflect change in asr_evaluator Signed-off-by: fayejf <fayejf07@gmail.com> * reflect som and vahid comment Signed-off-by: fayejf <fayejf07@gmail.com> * remove return_best_hy=true in transcribe_speech Signed-off-by: fayejf <fayejf07@gmail.com> * no text skip Signed-off-by: fayejf <fayejf07@gmail.com> --------- Signed-off-by: fayejf <fayejf07@gmail.com> * 1. Added example conf YAML. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support in tensor_parallel. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add hat image to docs (#6619) Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * update core commit hash in readme (#6622) Signed-off-by: Abhinav Khattar <aklife97@gmail.com> * Patch decoding for PC models (#6630) * Patch decoding logic for PC models Signed-off-by: smajumdar <titu1994@gmail.com> * Patch decoding logic for PC models Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> * Fix wer.py where 'errors' variable was not set (#6633) Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> * fix att_context_size bug for older models. (#6635) Signed-off-by: Vahid <vnoroozi@nvidia.com> * Add megatron_core to requirements (#6639) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Remove from jenkins (#6641) * add megatron_core to requirements Signed-off-by: ericharper <complex451@gmail.com> * remove from jenkins Signed-off-by: ericharper <complex451@gmail.com> --------- Signed-off-by: ericharper <complex451@gmail.com> * remove dup (#6643) Signed-off-by: ericharper <complex451@gmail.com> * 1. Fixed config to use names, and added better error messages. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Added support to pass extra data to hiddens for loss computation. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Working on passing extra data to hiddnes. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed support in loading .nemo without hiddnes module. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Improved and fixed logging of validation and testing. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed training logging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Fixed logging of hidden loss. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed logging names. 2. Added logging to hiddens and tokens loss. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed conflicts. Signed-off-by: Micha Livne <mlivne@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> --------- Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> Signed-off-by: smajumdar <smajumdar@nvidia.com> Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Signed-off-by: Virginia Adams <vadams@nvidia.com> Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: ericharper <complex451@gmail.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com> Signed-off-by: sam1373 <samuelkriman@gmail.com> Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: ekmb <ebakhturina@nvidia.com> Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Micha Livne <mlivne@nvidia.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu> Signed-off-by: Dima Rekesh <bmwshop@gmail.com> Signed-off-by: Jim O’Regan <jaoregan@tcd.ie> Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Signed-off-by: Evelina <ebakhturina@nvidia.com> Signed-off-by: Taejin Park <tango4j@gmail.com> Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com> Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Signed-off-by: Greg Clark <grclark@nvidia.com> Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> Signed-off-by: Eric Harper <complex451@gmail.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> Co-authored-by: Jocelyn <jocelynh@nvidia.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: bene-ges <61418381+bene-ges@users.noreply.github.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com> Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com> Co-authored-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com> Co-authored-by: anteju <108555623+anteju@users.noreply.github.com> Co-authored-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: David <amosalla@asu.edu> Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com> Co-authored-by: Samuel Kriman <samuelkriman@gmail.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com> Co-authored-by: Sean Naren <sean.narenthiran@gmail.com> Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com> Co-authored-by: Sean Naren <snarenthiran@nvidia.com> Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: Neha Tadimeti <ntadimeti@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: Jim O’Regan <jaoregan@tcd.ie> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com> Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com> Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com> Co-authored-by: Taejin Park <tango4j@gmail.com> Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Co-authored-by: Greg Clark <grclark@nvidia.com> Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Make tensor split contiguous for qkv and kv in attention
Collection: NLP
Changelog
Usage
# Add a code snippet demonstrating how to use thisBefore your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information