Zero-shot evaluation pipeline for mcore RETRO#8941

Merged

huvunvidia merged 146 commits intomainfrom huvu/mcore_retro_eval

Apr 23, 2024

+856-145

Commits on Jan 29, 2024

update branch
ericharper
committed

Commits on Jan 31, 2024

Add dist ckpt support for regular optimizers (#7749 )

authored

Commits on Feb 1, 2024

Commits on Feb 3, 2024

Commits on Feb 6, 2024

add code for calling mcore_retro in NeMo

huvunvidia
authored and
Huy Vu2
committed
add code for calling mcore_retro in NeMo

huvunvidia
authored and
Huy Vu2
committed
runnable, training curve match retro mcore and nemo

huvunvidia
authored and
Huy Vu2
committed
working on retro inference

huvunvidia
authored and
Huy Vu2
committed
working on megatron_retro_eval.py and megatron_retro_inference.yaml

huvunvidia
authored and
Huy Vu2
committed
refactoring text_generation_utils code and retro inference relevant files
Huy Vu2
committed
clean PR
Huy Vu2
committed
resolving quick hacks (reading number of train/valid samples from workdir, discrepancy in total samples and samples with neighbors retrieved, tokenizers)
Huy Vu2
committed
clean repository
Huy Vu2
committed
revert changes to inference/eval code to original in main
Huy Vu2
committed
clean code
Huy Vu2
committed
runable training code, with already implemented eval code
Huy Vu2
committed
[tutorial] fixed missing RIR scripts file. (#8257 )

XuesongYang
authored and
Huy Vu2
committed
add values to en tts dict (#7879 )

mgrafu
authored and
Huy Vu2
committed
Add Bert HF checkpoint converter (#8088 )

authored and
Huy Vu2
committed
revert to original eval code files
Huy Vu2
committed
revert to original eval code files 2
Huy Vu2
committed
revert to original eval code files 3
Huy Vu2
committed
revert to original eval code files 4
Huy Vu2
committed
clean code
Huy Vu2
committed
clean code
Huy Vu2
committed
update my code to support changes from lastest main
Huy Vu2
committed
commit before rebase r1.23.0
Huy Vu2
committed
Multimodal r1.23.0 bug fix (#8315 )

authored
copy paste files from r1.23.0
Huy Vu2
committed
clean PR
Huy Vu2
committed
Fixes for MoE parameter passing & use of AutoTokenizer/Model for mistral. (#8272 )
akoumpa
authored
Keep max_seqlen and cu_seqlens_argmin for later micro-batches when PP>1 (#8334 )

erhoo82
and
ericharper
authored
Remove asr webapp (#8347 )
titu1994
authored
remove _target_ at model level in aed config (#8351 )

krishnacpuvvada
and
Krishna Puvvada
authored

Commits on Feb 7, 2024

Commits on Feb 8, 2024

Commits on Feb 9, 2024

Commits on Feb 11, 2024

MCore dataset compatibility for tokenizers (#8390 )

vysarge
and
pablo-garay
authored

Commits on Feb 12, 2024

Commits on Feb 14, 2024

Fixed errors in the CTM gen functions (#8416 )
tango4j
authored

Commits on Feb 15, 2024

Commits on Feb 16, 2024

Commits on Feb 22, 2024

Add settings to suppress bf16 compile errors in CI on V100 (#8481 )

athitten
and
pre-commit-ci[bot]
authored

Commits on Feb 23, 2024

Commits on Feb 25, 2024

Add fp8 support for SD/Update notebook paths (#8489 )

authored

Commits on Feb 26, 2024

Commits on Feb 27, 2024

Commits on Mar 4, 2024

before update branch with latest r1.23.0
Huy Vu2
committed
Merge remote-tracking branch 'origin/r1.23.0' into huvu/mcore_retro
Huy Vu2
committed

Commits on Mar 5, 2024

update to run with MLM ae2817b3dde4efb1515061a5311d01d8f85bd99c (runnable training and saving checkpoint)
Huy Vu2
committed
remove compile_helpers
Huy Vu2
committed
reverse changes from main branch to r1.23.0
Huy Vu2
committed
adding *_legacy files
Huy Vu2
committed

Commits on Mar 6, 2024

update MLM commit in Jenkinsfile to latest
Huy Vu2
committed
debugging Jenkinstest: test different mcore import in retro_dataset
Huy Vu2
committed

Commits on Mar 7, 2024

Commits on Mar 20, 2024

update NeMo to work with latest mcore RETRO (solving TE problems)
Huy Vu2
committed
update TE commit Jenkinsfile to be the same with r1.23.0's Jenkinsfile
Huy Vu2
committed
update commit for MLM
Huy Vu2
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot]
committed
jenkinstest debugging
Huy Vu2
committed

Commits on Mar 21, 2024

Commits on Mar 22, 2024

add model.data.dataloader_type=cyclic to jenkinsfile
Huy Vu2
committed

Commits on Mar 31, 2024

runnable for inference
Huy Vu2
committed

Commits on Apr 4, 2024

update code to work with latest megatron-lm main 81dab6067
Huy Vu2
committed
update M-LM commit in Jenkinsfile to latest main M-LM 81dab6067
Huy Vu2
committed
cleaning inference code
Huy Vu2
committed

Commits on Apr 8, 2024

fix to by pass CI test bf16 problem (following this PR https://github.com/NVIDIA/NeMo/pull/8481/files )
Huy Vu2
committed
isort and black
Huy Vu2
committed
adjusting model.micro_batch_size to 1
Huy Vu2
committed

Commits on Apr 9, 2024

fix conflicts
Huy Vu2
committed
fix BRANCH = 'r1.23.0'
Huy Vu2
committed
replace tutorials dir from main branch to huvu/mcore_retro
Huy Vu2
committed
fix minor merges conflict
Huy Vu2
committed
update Jenkinsfile
Huy Vu2
committed

Commits on Apr 10, 2024

runnable with a temporary fix from Jacek (unfound -unfinished problem)
Huy Vu2
committed
runnable with a temporary fix from Jacek (unfound -unfinished problem)
Huy Vu2
committed
merged from main on 10apr
Huy Vu2
committed
modified nlp_overrides.py back to original
Huy Vu2
committed
fix checkpoint from Jacek Bieniusiewicz
Huy Vu2
committed
config Jenkinsfile test
Huy Vu2
committed

Commits on Apr 11, 2024

set RETRO Jenkins MBS to 1
Huy Vu2
committed
black fix
Huy Vu2
committed
isort fix
Huy Vu2
committed
update TE commit
Huy Vu2
committed
update to latest Jenkinsfile with latest container and commits
Huy Vu2
committed
remove new RETRO jenkinstest
Huy Vu2
committed
Merge remote-tracking branch 'origin/main' into huvu/mcore_retro
Huy Vu2
committed
merge latest main
Huy Vu2
committed
put RETRO Jenkinstest to the right place
Huy Vu2
committed
merge from latest origin/huvu/nemo_retro
Huy Vu2
committed
update code for megatron_retro_pretraining_legacy.py
Huy Vu2
committed

Commits on Apr 12, 2024

update Jenkins and _legacy.py
Huy Vu2
committed
update new RETRO jenkinstest to run faster
Huy Vu2
committed
fixing errors from GitHub Advanced Security / CodeQL
Huy Vu2
committed
fixing errors from GitHub Advanced Security / CodeQL
Huy Vu2
committed
update manually branch to huvu/mcore_retro
Huy Vu2
committed
remove DEBUGGING markers
Huy Vu2
committed

Commits on Apr 16, 2024

Commits on Apr 19, 2024

cleaning code, addressing Shanmugam's comments
root
committed
saving before pulling from main
Huy Vu2
committed
pulled from main
Huy Vu2
committed
cleaning code
Huy Vu2
committed
adding deprecations note
Huy Vu2
committed

Commits on Apr 22, 2024

Merge remote-tracking branch 'origin/main' into huvu/mcore_retro_eval
Huy Vu2
committed

Commits on Apr 23, 2024