[Model] LoRA Support for Ultravox model #11253

thedebugger · 2024-12-17T06:17:41Z

This should also work w/ mistral models which also uses LlamaForCasualLM architecture re: here

github-actions · 2024-12-17T06:18:13Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

thedebugger · 2024-12-17T06:29:06Z

Hi folks, I'm working with @petersalas on this. PR is not complete but wanted to start the discussion as I have some open questions and need some help from vLLM community

vllm/model_executor/models/ultravox.py

tests/lora/conftest.py

mergify · 2024-12-31T17:48:45Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @thedebugger.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

tests/lora/conftest.py

Signed-off-by: Sumit Vij <sumitvij11+github@gmail.com>

WIP: lora tests Minor tweaks Moar fixes Temp changes Cleanup Add more debugging logs and packed modules Signed-off-by: Sumit Vij <sumitvij11+github@gmail.com>

Remove stale comment Add llama lora modules Add llama test case Add test case and log warning on missing lora modules Rollback unwanted changes and format fixes Signed-off-by: Sumit Vij <sumitvij11+github@gmail.com>

jeejeelee · 2025-01-02T10:13:39Z

Can you refer to #10022 to minimize the changes?

thedebugger · 2025-01-02T13:37:05Z

Can you refer to #10022 to minimize the changes?

Changes are inline with 10022 except the test case and other minor logging changes. Do you have any concerns with any particular change?

jeejeelee · 2025-01-02T15:05:29Z

It looks like there are issues with both the added tests and logs. We should only modify the Ultravox scipt, following the changes made in the #10022

thedebugger · 2025-01-02T16:48:48Z

What is/are the issue(s)? Maybe I miss something but tests are passing

thedebugger · 2025-01-04T13:13:02Z

@jeejeelee lmk what are your concerns please? Happy to address it. Having a test case was super helpful to make sure LoRA works as expected with llama and ultravox

vllm/lora/models.py

tests/lora/conftest.py

…-ultravox-lora-dec-16

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

jeejeelee · 2025-01-06T10:37:44Z

@thedebugger We should only modify ultravox.py, please revert other changes. After you revert them, we can merge this PR.

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

jeejeelee

Let's keep the unit tests as they are for now. I'll look into how to train Ultravox LoRA and update in another PR

thedebugger · 2025-01-20T07:25:10Z

@jeejeelee I'll fix the test tomorrow. I pulled up the latest code and it is failing for me with Cuda OOM (different than CI failure). I also ran the test again on 208e662, it works. So likely something changed on master that is causing test to fail.

AFAICT, the failure is not related to LoRA, so training ultravox lora shouldn't have any impact. Let me troubleshoot this tomorrow and figure out what is going on before you spend time on this.

thedebugger · 2025-01-21T07:21:30Z

I spent more time looking at the failure. The failure happens during init when running ultravox with dummy data and I haven't been able to reproduce it locally (though locally test fails with cuda oom error for llama on latest version). I'll look further tomorrow why CI is seeing device mismatch when running whisper. I checked the other ultravox test, that is working fine. I'll look further into it tomorrow

jeejeelee · 2025-01-21T07:33:54Z

I spent more time looking at the failure. The failure happens during init when running ultravox with dummy data and I haven't been able to reproduce it locally (though locally test fails with cuda oom error for llama on latest version). I'll look further tomorrow why CI is seeing device mismatch when running whisper. I checked the other ultravox test, that is working fine. I'll look further into it tomorrow

We can first remove the LoRA test-related code and merge this PR. I'll spend some time later training a LoRA model. What do you think?

thedebugger · 2025-01-23T06:48:45Z

@jeejeelee can you trigger the lora test and add me in Buildkite org (if you have perms) so that i can trigger lora tests next time? I made few tweaks and want to verify if it works

jeejeelee · 2025-01-23T06:53:46Z

I've already started it. I suggest that if it still doesn't pass, we should remove the lora test. There's no need to waste time here.

jeejeelee · 2025-01-23T09:22:52Z

tests/lora/test_ultravox.py

+    """
+    #Check if set_default_device fixes the CI failure. Other lora tests set
+    # device "to cuda which might be causing device mismatch in CI
+    torch.set_default_device("cpu")


That's too hacky, don't do it. This significantly increased the CI testing time.

I checked the time before and after: delta was ~4 mins. Knowing this works, let me check if I can find a better fix for this

Source

Failing job: https://buildkite.com/vllm/ci/builds/12168#019481d9-b1d0-4e2c-866c-d6699816acac

Passing test: https://buildkite.com/vllm/fastcheck/builds/12263#019491e7-f538-4d7b-9849-ea8ba32d15f7

So it is likely a bug in transformer's whisper. I've opened a PR to fix that.

Workaround isn't ideal but impact is limited IMO given the passing test takes roughly 2-3 mins anyways. Moreover, default device is used only when device is not explicitly passed in functions param which is why we aren't seeing much impact

I also tried few more options like passing device all way through from vllm -> ultravox -> whisper but it is lot more complicated and require more changes in bunch of places. So I think it is okay to merge for now and I can clean up when it is fixed in transformer. Sounds okay?

@jeejeelee okay to merge if tests are passing?

Btw, I verified that device bug exists upstream and I'm working on getting that patched up

Hi @jeejeelee is set_default_device a blocker? Let me know how do you want to move forward

- Reduce model len and max num seq to reduce memory - Re-trigger tests Signed-off-by: Sumit Vij <sumitvij11+github@gmail.com>

…c-16

Signed-off-by: Felix Marty <felmarty@amd.com>

DarkLight1337 requested a review from jeejeelee December 17, 2024 06:19

jeejeelee reviewed Dec 17, 2024

View reviewed changes

vllm/model_executor/models/ultravox.py Show resolved Hide resolved

thedebugger commented Dec 17, 2024

View reviewed changes

vllm/model_executor/models/ultravox.py Show resolved Hide resolved

jeejeelee reviewed Dec 17, 2024

View reviewed changes

vllm/model_executor/models/ultravox.py Outdated Show resolved Hide resolved

vllm/model_executor/models/ultravox.py Outdated Show resolved Hide resolved

thedebugger commented Dec 17, 2024

View reviewed changes

tests/lora/conftest.py Outdated Show resolved Hide resolved

mergify bot added the needs-rebase label Dec 31, 2024

thedebugger commented Dec 31, 2024

View reviewed changes

tests/lora/conftest.py Outdated Show resolved Hide resolved

thedebugger force-pushed the svij-ultravox-lora-dec-16 branch 2 times, most recently from 771484d to 64a664f Compare December 31, 2024 18:49

mergify bot removed the needs-rebase label Dec 31, 2024

thedebugger changed the title ~~WIP: Ultravox Support for LoRA~~ Ultravox Support for LoRA Dec 31, 2024

thedebugger marked this pull request as ready for review December 31, 2024 18:52

thedebugger added 3 commits January 1, 2025 09:30

WIP: early draft of lora support in Ultravox

1c55938

Signed-off-by: Sumit Vij <sumitvij11+github@gmail.com>

format fixes

5a6b79f

WIP: lora tests Minor tweaks Moar fixes Temp changes Cleanup Add more debugging logs and packed modules Signed-off-by: Sumit Vij <sumitvij11+github@gmail.com>

Fix lora modules and formatting

3f5996c

Remove stale comment Add llama lora modules Add llama test case Add test case and log warning on missing lora modules Rollback unwanted changes and format fixes Signed-off-by: Sumit Vij <sumitvij11+github@gmail.com>

thedebugger force-pushed the svij-ultravox-lora-dec-16 branch from 64a664f to 3f5996c Compare January 1, 2025 17:31

jeejeelee reviewed Jan 6, 2025

View reviewed changes

vllm/lora/models.py Outdated Show resolved Hide resolved

jeejeelee reviewed Jan 6, 2025

View reviewed changes

tests/lora/conftest.py Outdated Show resolved Hide resolved

jeejeelee added 3 commits January 6, 2025 03:21

Merge branch 'main' of https://github.com/vllm-project/vllm into svij…

d1b65eb

…-ultravox-lora-dec-16

Merge branch 'main' of https://github.com/vllm-project/vllm into svij…

7367bc2

…-ultravox-lora-dec-16

Done

2abf2ab

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

Optmize unit test

f483d9a

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

jeejeelee approved these changes Jan 20, 2025

View reviewed changes

auto-merge was automatically disabled January 22, 2025 06:29
Head branch was pushed to by a user without write access

thedebugger force-pushed the svij-ultravox-lora-dec-16 branch from d247036 to 1195ad8 Compare January 22, 2025 07:48

jeejeelee removed the ready ONLY add when PR is ready to merge/full CI is needed label Jan 22, 2025

thedebugger force-pushed the svij-ultravox-lora-dec-16 branch 2 times, most recently from d5d023b to 09c8388 Compare January 23, 2025 06:44

jeejeelee reviewed Jan 23, 2025

View reviewed changes

thedebugger force-pushed the svij-ultravox-lora-dec-16 branch 2 times, most recently from 7f010af to 2b8db17 Compare January 26, 2025 05:19

Test setting cpu as a default device

1976ee0

- Reduce model len and max num seq to reduce memory - Re-trigger tests Signed-off-by: Sumit Vij <sumitvij11+github@gmail.com>

thedebugger force-pushed the svij-ultravox-lora-dec-16 branch from 2b8db17 to 1976ee0 Compare January 27, 2025 02:03

thedebugger added 3 commits January 26, 2025 23:22

Merge remote-tracking branch 'origin/main' into svij-ultravox-lora-de…

80fb1b8

…c-16

Merge remote-tracking branch 'origin/main' into svij-ultravox-lora-de…

c5cdde7

…c-16

Merge remote-tracking branch 'origin/main' into svij-ultravox-lora-de…

0b18650

…c-16

jeejeelee enabled auto-merge (squash) February 6, 2025 01:37

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Feb 6, 2025

DarkLight1337 changed the title ~~LoRA Support for Ultravox model~~ [Model] LoRA Support for Ultravox model Feb 6, 2025

simon-mo merged commit d88506d into vllm-project:main Feb 6, 2025
53 of 57 checks passed

houseroad mentioned this pull request Feb 6, 2025

[Bugfix] Fix the test_ultravox.py's license #12806

Merged

fxmarty-amd pushed a commit to fxmarty-amd/vllm that referenced this pull request Feb 7, 2025

[Model] LoRA Support for Ultravox model (vllm-project#11253)

05cc994

Signed-off-by: Felix Marty <felmarty@amd.com>

AoyuQC pushed a commit to AoyuQC/vllm that referenced this pull request Feb 8, 2025

[Model] LoRA Support for Ultravox model (vllm-project#11253)

5d4e6e3

ShangmingCai pushed a commit to ShangmingCai/vllm that referenced this pull request Feb 10, 2025

[Model] LoRA Support for Ultravox model (vllm-project#11253)

42a9fc7

panf2333 pushed a commit to yottalabsai/vllm that referenced this pull request Feb 18, 2025

[Model] LoRA Support for Ultravox model (vllm-project#11253)

87ce9a6

kerthcet pushed a commit to kerthcet/vllm that referenced this pull request Feb 21, 2025

[Model] LoRA Support for Ultravox model (vllm-project#11253)

a228018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Model] LoRA Support for Ultravox model #11253

[Model] LoRA Support for Ultravox model #11253

thedebugger commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024

thedebugger commented Dec 17, 2024 •

edited

Loading

mergify bot commented Dec 31, 2024

jeejeelee commented Jan 2, 2025

thedebugger commented Jan 2, 2025

jeejeelee commented Jan 2, 2025

thedebugger commented Jan 2, 2025

thedebugger commented Jan 4, 2025

jeejeelee commented Jan 6, 2025

jeejeelee left a comment •

edited

Loading

thedebugger commented Jan 20, 2025 •

edited

Loading

thedebugger commented Jan 21, 2025 •

edited

Loading

jeejeelee commented Jan 21, 2025

thedebugger commented Jan 23, 2025 •

edited

Loading

jeejeelee commented Jan 23, 2025

jeejeelee Jan 23, 2025

thedebugger Jan 24, 2025 •

edited

Loading

thedebugger Jan 24, 2025

thedebugger Jan 27, 2025 •

edited

Loading

thedebugger Jan 30, 2025

[Model] LoRA Support for Ultravox model #11253

[Model] LoRA Support for Ultravox model #11253

Conversation

thedebugger commented Dec 17, 2024 • edited Loading

github-actions bot commented Dec 17, 2024

thedebugger commented Dec 17, 2024 • edited Loading

mergify bot commented Dec 31, 2024

jeejeelee commented Jan 2, 2025

thedebugger commented Jan 2, 2025

jeejeelee commented Jan 2, 2025

thedebugger commented Jan 2, 2025

thedebugger commented Jan 4, 2025

jeejeelee commented Jan 6, 2025

jeejeelee left a comment • edited Loading

Choose a reason for hiding this comment

thedebugger commented Jan 20, 2025 • edited Loading

thedebugger commented Jan 21, 2025 • edited Loading

jeejeelee commented Jan 21, 2025

thedebugger commented Jan 23, 2025 • edited Loading

jeejeelee commented Jan 23, 2025

jeejeelee Jan 23, 2025

Choose a reason for hiding this comment

thedebugger Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

thedebugger Jan 24, 2025

Choose a reason for hiding this comment

thedebugger Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

thedebugger Jan 30, 2025

Choose a reason for hiding this comment

thedebugger commented Dec 17, 2024 •

edited

Loading

thedebugger commented Dec 17, 2024 •

edited

Loading

jeejeelee left a comment •

edited

Loading

thedebugger commented Jan 20, 2025 •

edited

Loading

thedebugger commented Jan 21, 2025 •

edited

Loading

thedebugger commented Jan 23, 2025 •

edited

Loading

thedebugger Jan 24, 2025 •

edited

Loading

thedebugger Jan 27, 2025 •

edited

Loading