Test different input sequence lengths for Llama #1070

pmarkovicTT · 2025-01-20T16:23:04Z

Add test to make sure Llama compiles and run fwd pass with different input sequence lengths as we will have inputs of various lengths during training.

Close #1071

forge/test/mlir/llama/test_llama_inference.py

github-actions · 2025-01-20T17:43:52Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	823 ran	490 passed	333 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-01-20T17:48:19Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	665 ran	434 passed	231 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-01-21T16:09:01Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	665 ran	437 passed	228 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-01-21T16:15:23Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	823 ran	492 passed	331 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-02-04T17:42:52Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	510 ran	451 passed	59 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-02-04T17:43:01Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	510 ran	451 passed	59 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-02-04T17:44:13Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	568 ran	489 passed	79 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-02-04T17:47:29Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	568 ran	489 passed	79 skipped	0 failed

Test	Result
No test annotations available

forge/test/mlir/llama/test_llama_inference.py

forge/test/mlir/llama/utils/utils.py

forge/test/mlir/llama/test_llama_inference.py

nvukobratTT · 2025-02-24T09:03:04Z

@pmarkovicTT is this one ready for review and potential merge? If not, can we move it to draft?

github-actions · 2025-02-25T13:42:48Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	670 ran	536 passed	134 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-02-25T13:45:20Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	611 ran	482 passed	129 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-02-25T13:45:55Z

	Tests	Passed ☑️	Skipped ⚠️	Failed ❌️
TT-Forge-FE Tests	611 ran	481 passed	129 skipped	1 failed

Test	Result
TT-Forge-FE Tests
pytest
test_dla.test_dla_pytorch[dla34]	❌ failure

github-actions · 2025-02-25T13:48:25Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	670 ran	536 passed	134 skipped	0 failed

Test	Result
No test annotations available

pmarkovicTT · 2025-02-27T12:22:22Z

@nvukobratTT PR is ready for review/merge.

Summary of previous conversations:

We selected input sequence sizes that matter to us (128, 512, 2048)
Added test for only 1 layer as we talked in one of previous syncs to run test faster
Embedding inputs have to be int and it's explicitly casted to int32 due to following issue Invalid Runtime inputs to embedding #952
When it comes to training/testing in lower precision, that's something we will do and incorporate in future training tests we add

github-actions · 2025-02-27T13:52:38Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	616 ran	484 passed	132 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-02-27T13:56:44Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	675 ran	538 passed	137 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-02-27T14:02:07Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	616 ran	484 passed	132 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-02-27T14:22:33Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	675 ran	538 passed	137 skipped	0 failed

Test	Result
No test annotations available

forge/test/mlir/llama/test_llama_inference.py

nvukobratTT

Not to block anymore, left are minor details, so I'm approving this PR :))

Regarding CI, I would suggest adding 1B variant just to make sure that it's passing. 3B we can skip.

And make sure to check do you need input IDs DF (non-blocker) :))

codecov-commenter · 2025-03-06T11:52:11Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 43.40%. Comparing base (9213144) to head (97c3e69).
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1070   +/-   ##
=======================================
  Coverage   43.40%   43.40%           
=======================================
  Files          48       48           
  Lines        7860     7860           
=======================================
  Hits         3412     3412           
  Misses       4448     4448

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2025-03-06T13:04:30Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	625 ran	489 passed	136 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-03-06T13:07:57Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	684 ran	541 passed	143 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-03-06T13:27:44Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	625 ran	489 passed	136 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-03-06T13:29:01Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	684 ran	541 passed	143 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-03-06T16:06:22Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	634 ran	492 passed	142 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-03-06T16:22:08Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	693 ran	556 passed	137 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-03-06T16:35:30Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	634 ran	492 passed	142 skipped	0 failed

Test	Result
No test annotations available

github-actions · 2025-03-06T16:51:10Z

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Forge-FE Tests	693 ran	556 passed	137 skipped	0 failed

Test	Result
No test annotations available

Add test to make sure Llama compiles and run fwd pass with different input sequence lengths as we will have inputs of various lengths during training. Close #1071

pmarkovicTT requested review from nvukobratTT, pilkicTT and dgolubovicTT as code owners January 20, 2025 16:23

pmarkovicTT requested a review from vladimirjovanovicTT January 20, 2025 16:23

pmarkovicTT self-assigned this Jan 20, 2025

nvukobratTT reviewed Jan 20, 2025

View reviewed changes

forge/test/mlir/llama/test_llama_inference.py Outdated Show resolved Hide resolved

forge/test/mlir/llama/test_llama_inference.py Outdated Show resolved Hide resolved

forge/test/mlir/llama/test_llama_inference.py Show resolved Hide resolved

pmarkovicTT mentioned this pull request Jan 21, 2025

Find a good finetuning sequence length for LoRA tenstorrent/tt-blacksmith#71

Closed

pmarkovicTT force-pushed the pmarkovic/test-input-seq-lengths-llama branch from 5b45490 to 710afb4 Compare February 4, 2025 16:47

pmarkovicTT requested a review from nvukobratTT February 4, 2025 16:52

nvukobratTT reviewed Feb 7, 2025

View reviewed changes

pmarkovicTT force-pushed the pmarkovic/test-input-seq-lengths-llama branch from 710afb4 to f081375 Compare February 25, 2025 13:01

pmarkovicTT requested a review from nvukobratTT February 27, 2025 12:22

nvukobratTT reviewed Mar 4, 2025

View reviewed changes

forge/test/mlir/llama/test_llama_inference.py Show resolved Hide resolved

forge/test/mlir/llama/test_llama_inference.py Outdated Show resolved Hide resolved

nvukobratTT approved these changes Mar 5, 2025

View reviewed changes

pmarkovicTT added 4 commits March 6, 2025 10:47

Add test for different input seq lens

011a21b

Add padding and truncation

ea9b0bf

Add verify check

cfa8191

clean code

Loading
Loading status checks…

463414e

pmarkovicTT force-pushed the pmarkovic/test-input-seq-lengths-llama branch from 2ab4f57 to 463414e Compare March 6, 2025 11:05

Remove explict ids int32 cast

Loading
Loading status checks…

c497d5a

Merge branch 'main' into pmarkovic/test-input-seq-lengths-llama

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

Loading
Loading status checks…

97c3e69

pmarkovicTT merged commit 70f40ea into main Mar 6, 2025
11 checks passed

pmarkovicTT deleted the pmarkovic/test-input-seq-lengths-llama branch March 6, 2025 16:52

Test different input sequence lengths for Llama #1070

Test different input sequence lengths for Llama #1070

Conversation

pmarkovicTT commented Jan 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 20, 2025

Uh oh!

github-actions bot commented Jan 20, 2025

Uh oh!

github-actions bot commented Jan 21, 2025

Uh oh!

github-actions bot commented Jan 21, 2025

Uh oh!

github-actions bot commented Feb 4, 2025

Uh oh!

github-actions bot commented Feb 4, 2025

Uh oh!

github-actions bot commented Feb 4, 2025

Uh oh!

github-actions bot commented Feb 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nvukobratTT commented Feb 24, 2025

Uh oh!

github-actions bot commented Feb 25, 2025

Uh oh!

github-actions bot commented Feb 25, 2025

Uh oh!

github-actions bot commented Feb 25, 2025

Uh oh!

github-actions bot commented Feb 25, 2025

Uh oh!

pmarkovicTT commented Feb 27, 2025

Uh oh!

github-actions bot commented Feb 27, 2025

Uh oh!

github-actions bot commented Feb 27, 2025

Uh oh!

github-actions bot commented Feb 27, 2025

Uh oh!

github-actions bot commented Feb 27, 2025

Uh oh!

Uh oh!

Uh oh!

nvukobratTT left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

Uh oh!

pmarkovicTT commented Jan 20, 2025 •

edited

Loading

nvukobratTT left a comment •

edited

Loading

codecov-commenter commented Mar 6, 2025 •

edited

Loading