Enable Sequence Packing and Pipeline Parallel in NeVA #8957

yaoyu-33 · 2024-04-17T20:30:46Z

What does this PR do ?

Enable Sequence Packing and Pipeline Parallel in NeVA

Collection: [Multimodal]

Changelog

Added Sequence Packing dataset preparation script
Update Neva Model and dataset to support packed dataset, also add args to turn on packed dataset
Enable Pipeline parallelism

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

# Conflicts: # examples/multimodal/vision_language_foundation/clip/megatron_clip_pretrain.py

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

# Conflicts: # nemo/collections/multimodal/models/multimodal_llm/neva/neva_model.py

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

for more information, see https://pre-commit.ci

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

…neva_pp_support

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

nemo/collections/multimodal/parts/utils.py

examples/multimodal/multimodal_llm/neva/eval/vqa_science.py

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

examples/multimodal/multimodal_llm/neva/sequence_packing/preprocess_dataset.py

nemo/collections/multimodal/models/multimodal_llm/neva/neva_model.py

examples/multimodal/multimodal_llm/neva/sequence_packing/preprocess_dataset.py

nemo/collections/multimodal/parts/utils.py

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

for more information, see https://pre-commit.ci

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

cuichenx

LGTM!

yaoyu-33 · 2024-04-26T04:35:51Z

jenkins

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2024-04-26T16:42:35Z

jenkins

yaoyu-33 · 2024-04-26T16:42:58Z

jenkins

yaoyu-33 · 2024-04-29T16:54:34Z

jenkins

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

for more information, see https://pre-commit.ci

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* temp save Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * temp save 2 Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update code Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * enable seq packing Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix neva and clip Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Enable parallel seq packing algo and few other fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Pipeline parallel support Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update data preprocess Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix few pp issues Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * enable sequence packing w/ PP Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix cu_seqlens in inputs Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * add assert Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Depend on PP to decide whether do padding Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add docstring Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix few evaluation issues Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix few PP evaluation issues Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix license Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Few neva bugs Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Few neva bugs Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Ao Tang <aot@nvidia.com>

* temp save Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * temp save 2 Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update code Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * enable seq packing Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix neva and clip Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Enable parallel seq packing algo and few other fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Pipeline parallel support Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update data preprocess Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix few pp issues Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * enable sequence packing w/ PP Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix cu_seqlens in inputs Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * add assert Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Depend on PP to decide whether do padding Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add docstring Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix few evaluation issues Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix few PP evaluation issues Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix license Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Few neva bugs Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Few neva bugs Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

yaoyu-33 added 19 commits March 15, 2024 18:27

temp save

f2f267d

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Merge branch 'main' into yuya/neva_seq_pack

b532d1b

temp save 2

b0ace6b

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

update code

0020fe3

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Merge branch 'main' into yuya/neva_seq_pack

76fb748

enable seq packing

a8f2248

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

fix neva and clip

9fab5a5

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Enable parallel seq packing algo and few other fixes

d8474fb

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Merge branch 'main' into yuya/neva_seq_pack

c56ec9b

# Conflicts: # examples/multimodal/vision_language_foundation/clip/megatron_clip_pretrain.py

Pipeline parallel support

e8a9a6d

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Update data preprocess

c5ffa83

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Merge branch 'main' into yuya/neva_pp_support

e11e260

fix few pp issues

2bc5d66

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Merge branch 'yuya/neva_seq_pack' into yuya/neva_pp_support

4843e54

enable sequence packing w/ PP

78034ce

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Fix cu_seqlens in inputs

8561e60

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

add assert

2ac6b27

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Merge branch 'main' into yuya/neva_pp_support

d138b1e

# Conflicts: # nemo/collections/multimodal/models/multimodal_llm/neva/neva_model.py

Depend on PP to decide whether do padding

5e6994d

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

github-actions bot added the Multi Modal label Apr 17, 2024

pre-commit-ci bot and others added 3 commits April 17, 2024 20:32

[pre-commit.ci] auto fixes from pre-commit.com hooks

0544758

for more information, see https://pre-commit.ci

Add docstring

6af32af

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Merge remote-tracking branch 'origin/yuya/neva_pp_support' into yuya/…

cd513b3

…neva_pp_support

yaoyu-33 requested review from cuichenx and Victor49152 April 19, 2024 16:29

Fix few evaluation issues

3655f7d

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

github-actions bot added the NLP label Apr 23, 2024

github-advanced-security bot found potential problems Apr 23, 2024

View reviewed changes

nemo/collections/multimodal/parts/utils.py Show resolved Hide resolved

examples/multimodal/multimodal_llm/neva/eval/vqa_science.py Dismissed Show dismissed Hide dismissed

yaoyu-33 added 2 commits April 24, 2024 09:52

Merge branch 'main' into yuya/neva_pp_support

f54e565

Fix few PP evaluation issues

4bb0313

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

cuichenx reviewed Apr 24, 2024

View reviewed changes

yaoyu-33 and others added 3 commits April 24, 2024 13:35

Address comments

6efa4fa

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

9c44e30

for more information, see https://pre-commit.ci

address comments

56700e7

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

cuichenx previously approved these changes Apr 25, 2024

View reviewed changes

Fix license

b5a4c27

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 dismissed cuichenx’s stale review via b5a4c27 April 26, 2024 16:33

Merge branch 'main' into yuya/neva_pp_support

f2588a0

Merge branch 'main' into yuya/neva_pp_support

45a6c61

Victor49152 previously approved these changes Apr 29, 2024

View reviewed changes

Few neva bugs

8d9f8c1

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 dismissed Victor49152’s stale review via 8d9f8c1 April 30, 2024 03:57

yaoyu-33 added the Run CICD label Apr 30, 2024

[pre-commit.ci] auto fixes from pre-commit.com hooks

449a42f

for more information, see https://pre-commit.ci

yaoyu-33 added Run CICD and removed Run CICD labels Apr 30, 2024

Few neva bugs

0380c3b

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 added Run CICD and removed Run CICD labels Apr 30, 2024

Victor49152 approved these changes Apr 30, 2024

View reviewed changes

yaoyu-33 merged commit 33494f5 into main Apr 30, 2024
129 checks passed

yaoyu-33 deleted the yuya/neva_pp_support branch April 30, 2024 20:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Sequence Packing and Pipeline Parallel in NeVA #8957

Enable Sequence Packing and Pipeline Parallel in NeVA #8957

yaoyu-33 commented Apr 17, 2024

cuichenx left a comment

yaoyu-33 commented Apr 26, 2024

yaoyu-33 commented Apr 26, 2024

yaoyu-33 commented Apr 26, 2024

yaoyu-33 commented Apr 29, 2024

Enable Sequence Packing and Pipeline Parallel in NeVA #8957

Enable Sequence Packing and Pipeline Parallel in NeVA #8957

Conversation

yaoyu-33 commented Apr 17, 2024

What does this PR do ?

Changelog

Usage

Jenkins CI

Before your PR is "Ready for review"

Who can review?

Additional Information

cuichenx left a comment

Choose a reason for hiding this comment

yaoyu-33 commented Apr 26, 2024

yaoyu-33 commented Apr 26, 2024

yaoyu-33 commented Apr 26, 2024

yaoyu-33 commented Apr 29, 2024