Test Prompt Embeds/LoRA compatibility and Enable LoRA Support for OPT Models #25717
Conversation
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
qthequartermasterman
requested review from
DarkLight1337,
NickLucche,
aarnphm,
robertgshaw2-redhat and
simon-mo
as code owners
|
@DarkLight1337. I'm not sure who else would need to look at this. I also wonder that using this model could speed up some of the other entrypoints LoRA tests that are currently using zephyr-7b, just like you sped up these tests in #25663. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds LoRA support for OPT models and includes corresponding tests. The changes to enable LoRA in the OPT model implementation are mostly correct, following patterns from other models in the repository. However, I found a critical issue in the initialization of the LogitsProcessor which would lead to incorrect behavior when using LoRA adapters with extra vocabulary tokens. My review provides a code suggestion to fix this.
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
…list Signed-off-by: Andrew Sansom <andrew@protopia.ai>
There was a problem hiding this comment.
LGTM if the tests pass, cc @jeejeelee if you want to double check the model
github-actions
bot
added
the
ready
cc @jeejeelee |
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Head branch was pushed to by a user without write access
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
|
@DarkLight1337 This looks like it's ready for re-review. Thanks! Thanks @jeejeelee for your help. |
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <andrew@protopia.ai>
… Models (#25717) Signed-off-by: Andrew Sansom <andrew@protopia.ai> Signed-off-by: yewentao256 <zhyanwentao@126.com>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <andrew@protopia.ai> Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <andrew@protopia.ai> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <andrew@protopia.ai>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <andrew@protopia.ai>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <andrew@protopia.ai> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
3 participants
Purpose
It was previously unknown if #24278 was compatible with LoRA adapters or not. This PR adds tests explicitly for that combination. Since #25663 swapped out Zephyr for OPT125-m for testing prompt embeds, this PR also adds LoRA support for opt125-m.
Test Plan
Updated tests cases. I've also tested it locally with a meta-llama/Llama-3.1-8B-Instruct LoRA and everything seems to work as expected there.
Test Result
New tests are working locally. Pending CI.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.