Add mcore full TE transformer layer spec#8328

Merged

jbaczek merged 26 commits intomainfrom jbaczek/add_full_te_mcore_spec

Feb 23, 2024

+369-118

Commits on Feb 22, 2024

Add spec and implement autocast layer

jbaczek
authored and
ShriyaPalsamudram
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
ShriyaPalsamudram
committed
remove try-catchs, these dependecies are mandatory for this file

jbaczek
authored and
ShriyaPalsamudram
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
ShriyaPalsamudram
committed
Check out this cool try/except clause

jbaczek
authored and
ShriyaPalsamudram
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
ShriyaPalsamudram
committed
Remove unused import

jbaczek
authored and
ShriyaPalsamudram
committed
Add import tests to Jenkinsfile

jbaczek
authored and
ShriyaPalsamudram
committed
Move import tests to Jenkins and remove code that is developed only for passing tests

jbaczek
authored and
ShriyaPalsamudram
committed
Make test robust to faulty base configs

jbaczek
authored and
ShriyaPalsamudram
committed
Use proper GPT implementation in the test

jbaczek
authored and
ShriyaPalsamudram
committed
Update nemo/collections/nlp/models/language_modeling/megatron/gpt_full_te_layer_autocast_spec.py

authored and
ShriyaPalsamudram
committed
Update nemo/collections/nlp/models/language_modeling/megatron/gpt_full_te_layer_autocast_spec.py

authored and
ShriyaPalsamudram
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
ShriyaPalsamudram
committed
Update nemo/collections/nlp/models/language_modeling/megatron/gpt_full_te_layer_autocast_spec.py

authored and
ShriyaPalsamudram
committed
Update nemo/collections/nlp/models/language_modeling/megatron/gpt_full_te_layer_autocast_spec.py

authored and
ShriyaPalsamudram
committed
Add TE knobs to the copy of AutocastTransformerLayer

jbaczek
authored and
ShriyaPalsamudram
committed
Add TE knobs to the copy of AutocastTransformerLayer

jbaczek
authored and
ShriyaPalsamudram
committed
Add dummy parameter to accomodated for the changes in mcore

jbaczek
authored and
ShriyaPalsamudram
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
ShriyaPalsamudram
committed
Update mcore to 0.5.0 in Jenkins pipeline

jbaczek
authored and
ShriyaPalsamudram
committed
Bump mcore commit. This is commit from tot, not any release.

jbaczek
authored and
ShriyaPalsamudram
committed
Remove from the test config option that is incompatible with bias_activation_fusion

jbaczek
authored and
ShriyaPalsamudram
committed
Bump TE version in CI to 1.4

jbaczek
authored and
ShriyaPalsamudram
committed
Update test

jbaczek
authored and
ShriyaPalsamudram
committed
Change precision for the test - current runnens don't support bf16

jbaczek
authored and
ShriyaPalsamudram
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add mcore full TE transformer layer spec#8328

Commits on Feb 22, 2024

Add spec and implement autocast layer

[pre-commit.ci] auto fixes from pre-commit.com hooks

remove try-catchs, these dependecies are mandatory for this file

[pre-commit.ci] auto fixes from pre-commit.com hooks

Check out this cool try/except clause

[pre-commit.ci] auto fixes from pre-commit.com hooks

Remove unused import

Add import tests to Jenkinsfile

Move import tests to Jenkins and remove code that is developed only for passing tests

Make test robust to faulty base configs

Use proper GPT implementation in the test

Update nemo/collections/nlp/models/language_modeling/megatron/gpt_full_te_layer_autocast_spec.py

Update nemo/collections/nlp/models/language_modeling/megatron/gpt_full_te_layer_autocast_spec.py

[pre-commit.ci] auto fixes from pre-commit.com hooks

Update nemo/collections/nlp/models/language_modeling/megatron/gpt_full_te_layer_autocast_spec.py

Update nemo/collections/nlp/models/language_modeling/megatron/gpt_full_te_layer_autocast_spec.py

Add TE knobs to the copy of AutocastTransformerLayer

Add TE knobs to the copy of AutocastTransformerLayer

Add dummy parameter to accomodated for the changes in mcore

[pre-commit.ci] auto fixes from pre-commit.com hooks

Update mcore to 0.5.0 in Jenkins pipeline

Bump mcore commit. This is commit from tot, not any release.

Remove from the test config option that is incompatible with bias_activation_fusion

Bump TE version in CI to 1.4

Update test

Change precision for the test - current runnens don't support bf16