Implement DeepSpeed Main autotuning for NeoX#739

Merged

Quentin-Anthony merged 101 commits intomainfrom autotune

Mar 9, 2023

+509-38

Commits on Sep 21, 2022

Merge branch 'srun' into autotune
Dashiell Stander
committed

Commits on Sep 26, 2022

Add autotuning
dashstander
committed
Add autotuning config
Dashiell Stander
committed

Commits on Sep 27, 2022

Need to add it to deepspeed args
dashstander
committed
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox into autotune
dashstander
committed

Commits on Sep 29, 2022

Commits on Oct 14, 2022

Move configs out of \/tmp
dashstander
committed

Commits on Oct 17, 2022

Commits on Oct 18, 2022

Merge main
dashstander
committed

Commits on Oct 19, 2022

Commits on Oct 20, 2022

json configs for DeepSpeed
Dashiell Stander
committed
only two nodes
Dashiell Stander
committed

Commits on Oct 21, 2022

Needed to change up the configs
Dashiell Stander
committed

Commits on Oct 25, 2022

Do not actually need to do that
dashstander
committed
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox into autotune
dashstander
committed
Tune 6.7B model
Dashiell Stander
committed
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox into autotune
Dashiell Stander
committed
New types for zero stage
dashstander
committed
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox into autotune
Dashiell Stander
committed
New types for zero stage
dashstander
committed
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox into autotune
Dashiell Stander
committed
Tuning a larger model
Dashiell Stander
committed
Always copy autotuning args from ds_config
dashstander
committed
Always copy autotuning args from ds_config
dashstander
committed
Cleaner this way, I think...
dashstander
committed
New debug print statement
dashstander
committed
New debug print statement
dashstander
committed
Need to copy this over as well
dashstander
committed
Need to copy over train_batch_size as well
dashstander
committed

Commits on Nov 8, 2022

debug print
dashstander
committed

Commits on Nov 10, 2022

new configs
Dashiell Stander
committed
Tests
Dashiell Stander
committed

Commits on Dec 7, 2022

Merge branch 'main' into autotune
dashstander
committed

Commits on Dec 12, 2022

Sync with new method of passing in autotuning configs
dashstander
committed

Commits on Jan 6, 2023

Merge main
Dashiell Stander
committed
Replicate on different cluster
Dashiell Stander
committed
Update NeoXArgs docs automatically
github-actions
committed
Use typing List and fix bug in decoding
dashstander
committed

Commits on Jan 7, 2023

Commits on Jan 9, 2023

no_ssh_check should be a configured value
dashstander
committed

Commits on Jan 12, 2023

Only write to file when doing autotuning
dashstander
committed

Commits on Jan 15, 2023

Merge branch 'main' into autotune
Quentin-Anthony
authored
Update NeoXArgs docs automatically
github-actions
committed

Commits on Jan 16, 2023

Remove debugging configs
dashstander
committed
Remove test scripts
dashstander
committed
Update NeoXArgs docs automatically
github-actions
committed
Remove test script
dashstander
committed
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox into autotune
dashstander
committed
Update NeoXArgs docs automatically
github-actions
committed
Clean up
dashstander
committed
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox into autotune
dashstander
committed
Update NeoXArgs docs automatically
github-actions
committed
Run pre-commit hooks
dashstander
committed
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox into autotune
dashstander
committed
Update NeoXArgs docs automatically
github-actions
committed
base64 error
dashstander
committed
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox into autotune
dashstander
committed
Update NeoXArgs docs automatically
github-actions
committed

Commits on Feb 14, 2023