Upgrade NeMo to latest mcore and TE #7862

dimapihtar · 2023-11-06T14:47:27Z

What does this PR do ?

Upgrades NeMo to latest mcore and TE versions.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: dimapihtar <dpihtar@gmail.com>

for more information, see https://pre-commit.ci

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

Signed-off-by: dimapihtar <dpihtar@gmail.com>

examples/nlp/language_modeling/conf/megatron_gpt_config.yaml

examples/nlp/language_modeling/conf/megatron_gpt_inference.yaml

Signed-off-by: dimapihtar <dpihtar@gmail.com>

Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>

ericharper · 2023-11-15T16:58:51Z

jenkins

Signed-off-by: dimapihtar <dpihtar@gmail.com>

dimapihtar · 2023-11-16T20:24:37Z

jenkins

Signed-off-by: dimapihtar <dpihtar@gmail.com>

dimapihtar · 2023-11-16T21:39:27Z

jenkins

dimapihtar · 2023-11-16T21:45:43Z

jenkins

dimapihtar · 2023-11-16T22:32:19Z

jenkins

Signed-off-by: dimapihtar <dpihtar@gmail.com>

dimapihtar · 2023-11-16T22:53:05Z

jenkins

Jenkinsfile

ericharper · 2023-11-16T22:57:25Z

Jenkinsfile

@@ -3150,51 +3160,51 @@ assert_frame_equal(training_curve, gt_curve, rtol=1e-3, atol=1e-3)"'''
        sh "rm -rf examples/nlp/language_modeling/gpt_index_mappings"
      }
    }
-    stage('L2: Megatron GPT with Rope Pretraining and Resume Training TP=2') {


Do we need to remove this test?

@ericharper this one uses position_embedding_type=rope which causes an error (TE bug).

This test is not even using mcore_gpt or transformer_engine flags though....

Signed-off-by: dimapihtar <dpihtar@gmail.com>

dimapihtar · 2023-11-16T23:02:02Z

jenkins

Signed-off-by: dimapihtar <dpihtar@gmail.com>

dimapihtar · 2023-11-16T23:20:08Z

jenkins

Signed-off-by: dimapihtar <dpihtar@gmail.com>

dimapihtar · 2023-11-16T23:53:21Z

jenkins

Signed-off-by: dimapihtar <dpihtar@gmail.com>

dimapihtar · 2023-11-17T02:05:54Z

jenkins

Signed-off-by: eharper <eharper@nvidia.com>

ericharper · 2023-11-17T03:36:26Z

jenkins

Signed-off-by: eharper <eharper@nvidia.com>

ericharper · 2023-11-17T06:45:52Z

jenkins

ericharper

LGTM. Thanks!

* mcore upgrade Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove GPTEmbedding import (deprecated) Signed-off-by: dimapihtar <dpihtar@gmail.com> * switch to LanguageModelEmbedding Signed-off-by: dimapihtar <dpihtar@gmail.com> * reset config Signed-off-by: dimapihtar <dpihtar@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pass attn_mask-type through the forward method Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * reset conf Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * add more attn_mask_type params Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * attn_,ask_type fixes Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove attn_mask_type param Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * t5/mt5 fix Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert configs Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert configs Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove attn_mask_type param Signed-off-by: dimapihtar <dpihtar@gmail.com> * update mcore commit Signed-off-by: dimapihtar <dpihtar@gmail.com> * change mcore commit Signed-off-by: dimapihtar <dpihtar@gmail.com> * add TE installation Signed-off-by: dimapihtar <dpihtar@gmail.com> * add env var Signed-off-by: dimapihtar <dpihtar@gmail.com> * comment out rope test Signed-off-by: dimapihtar <dpihtar@gmail.com> * change mcore installation Signed-off-by: dimapihtar <dpihtar@gmail.com> * update mcore commit Signed-off-by: dimapihtar <dpihtar@gmail.com> * change mcore installation Signed-off-by: dimapihtar <dpihtar@gmail.com> * change mcore commit Signed-off-by: dimapihtar <dpihtar@gmail.com> * add -e Signed-off-by: eharper <eharper@nvidia.com> * revert Signed-off-by: eharper <eharper@nvidia.com> * revert jenkins test comment Signed-off-by: eharper <eharper@nvidia.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: eharper <eharper@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: eharper <eharper@nvidia.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* mcore upgrade Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove GPTEmbedding import (deprecated) Signed-off-by: dimapihtar <dpihtar@gmail.com> * switch to LanguageModelEmbedding Signed-off-by: dimapihtar <dpihtar@gmail.com> * reset config Signed-off-by: dimapihtar <dpihtar@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pass attn_mask-type through the forward method Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * reset conf Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * add more attn_mask_type params Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * attn_,ask_type fixes Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove attn_mask_type param Signed-off-by: dimapihtar <dpykhtar@nvidia.com> * t5/mt5 fix Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert configs Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert configs Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove attn_mask_type param Signed-off-by: dimapihtar <dpihtar@gmail.com> * update mcore commit Signed-off-by: dimapihtar <dpihtar@gmail.com> * change mcore commit Signed-off-by: dimapihtar <dpihtar@gmail.com> * add TE installation Signed-off-by: dimapihtar <dpihtar@gmail.com> * add env var Signed-off-by: dimapihtar <dpihtar@gmail.com> * comment out rope test Signed-off-by: dimapihtar <dpihtar@gmail.com> * change mcore installation Signed-off-by: dimapihtar <dpihtar@gmail.com> * update mcore commit Signed-off-by: dimapihtar <dpihtar@gmail.com> * change mcore installation Signed-off-by: dimapihtar <dpihtar@gmail.com> * change mcore commit Signed-off-by: dimapihtar <dpihtar@gmail.com> * add -e Signed-off-by: eharper <eharper@nvidia.com> * revert Signed-off-by: eharper <eharper@nvidia.com> * revert jenkins test comment Signed-off-by: eharper <eharper@nvidia.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dpykhtar@nvidia.com> Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Signed-off-by: eharper <eharper@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: eharper <eharper@nvidia.com>

dimapihtar and others added 6 commits October 13, 2023 07:14

mcore upgrade

3e235df

Signed-off-by: dimapihtar <dpihtar@gmail.com>

remove GPTEmbedding import (deprecated)

3f10321

Signed-off-by: dimapihtar <dpihtar@gmail.com>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

f1329f8

switch to LanguageModelEmbedding

6a0b36d

Signed-off-by: dimapihtar <dpihtar@gmail.com>

reset config

a1c8e41

Signed-off-by: dimapihtar <dpihtar@gmail.com>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

148e331

github-actions bot added the NLP label Nov 6, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

5dccef6

for more information, see https://pre-commit.ci

dimapihtar requested a review from ericharper November 6, 2023 15:59

ericharper and others added 9 commits November 6, 2023 19:00

Merge branch 'main' into dpykhtar/te_mcore_upgrade

177cec7

pass attn_mask-type through the forward method

22ee169

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

reset conf

5ab2655

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

add more attn_mask_type params

79f95b2

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

d02ec45

for more information, see https://pre-commit.ci

attn_,ask_type fixes

a7368fc

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

8ed2cf1

for more information, see https://pre-commit.ci

remove attn_mask_type param

989b9a5

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

t5/mt5 fix

d1b7189

Signed-off-by: dimapihtar <dpihtar@gmail.com>

ericharper reviewed Nov 10, 2023

View reviewed changes

examples/nlp/language_modeling/conf/megatron_gpt_config.yaml Outdated Show resolved Hide resolved

ericharper reviewed Nov 10, 2023

View reviewed changes

examples/nlp/language_modeling/conf/megatron_gpt_inference.yaml Outdated Show resolved Hide resolved

ericharper reviewed Nov 10, 2023

View reviewed changes

examples/nlp/language_modeling/conf/megatron_gpt_inference.yaml Outdated Show resolved Hide resolved

dimapihtar and others added 4 commits November 10, 2023 06:37

revert configs

683b277

Signed-off-by: dimapihtar <dpihtar@gmail.com>

revert configs

6518f88

Signed-off-by: dimapihtar <dpihtar@gmail.com>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

d11cadd

Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

3acaf9c

ericharper requested review from aklife97 and gshennvm November 15, 2023 17:16

dimapihtar and others added 2 commits November 16, 2023 06:52

remove attn_mask_type param

6963155

Signed-off-by: dimapihtar <dpihtar@gmail.com>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

3693888

add TE installation

57c2487

Signed-off-by: dimapihtar <dpihtar@gmail.com>

dimapihtar and others added 2 commits November 16, 2023 13:37

add env var

68895b9

Signed-off-by: dimapihtar <dpihtar@gmail.com>

Merge branch 'main' into dpykhtar/te_mcore_upgrade

83c708b

comment out rope test

4aff223

Signed-off-by: dimapihtar <dpihtar@gmail.com>

ericharper reviewed Nov 16, 2023

View reviewed changes

Jenkinsfile Show resolved Hide resolved

ericharper reviewed Nov 16, 2023

View reviewed changes

change mcore installation

197f26a

Signed-off-by: dimapihtar <dpihtar@gmail.com>

update mcore commit

a5390c9

Signed-off-by: dimapihtar <dpihtar@gmail.com>

change mcore installation

fbfbe3a

Signed-off-by: dimapihtar <dpihtar@gmail.com>

change mcore commit

d5d0a6f

Signed-off-by: dimapihtar <dpihtar@gmail.com>

ericharper and others added 3 commits November 16, 2023 20:04

Merge branch 'main' into dpykhtar/te_mcore_upgrade

447507a

add -e

0ad5d7f

Signed-off-by: eharper <eharper@nvidia.com>

revert

fda6c93

Signed-off-by: eharper <eharper@nvidia.com>

revert jenkins test comment

d102831

Signed-off-by: eharper <eharper@nvidia.com>

ericharper approved these changes Nov 17, 2023

View reviewed changes

dimapihtar merged commit ce6afd7 into main Nov 17, 2023
12 checks passed

dimapihtar deleted the dpykhtar/te_mcore_upgrade branch November 17, 2023 13:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade NeMo to latest mcore and TE #7862

Upgrade NeMo to latest mcore and TE #7862

dimapihtar commented Nov 6, 2023

ericharper commented Nov 15, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

ericharper Nov 16, 2023

dimapihtar Nov 16, 2023 •

edited

Loading

ericharper Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 17, 2023

ericharper commented Nov 17, 2023

ericharper commented Nov 17, 2023

ericharper left a comment

Upgrade NeMo to latest mcore and TE #7862

Upgrade NeMo to latest mcore and TE #7862

Conversation

dimapihtar commented Nov 6, 2023

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

ericharper commented Nov 15, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

ericharper Nov 16, 2023

Choose a reason for hiding this comment

dimapihtar Nov 16, 2023 • edited Loading

Choose a reason for hiding this comment

ericharper Nov 16, 2023

Choose a reason for hiding this comment

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 16, 2023

dimapihtar commented Nov 17, 2023

ericharper commented Nov 17, 2023

ericharper commented Nov 17, 2023

ericharper left a comment

Choose a reason for hiding this comment

dimapihtar Nov 16, 2023 •

edited

Loading