[Hardware][Intel-Gaudi] Enable long-contexts + LoRA support for Intel Gaudi #12812
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
vllm/lora/punica_wrapper/utils.py
Outdated
| long_lora_offsets = torch.zeros(len(index_mapping_indices), | ||
| device=device, | ||
| dtype=torch.long) | ||
| if device == torch.device("hpu"): |
There was a problem hiding this comment.
Can you limit the modifications to PunicaWrapperHPU, such as overriding the _update_base_metadata function?
Signed-off-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>
SanjuCSudhakaran
force-pushed
the
enable-long-contexts-lora
branch
from
9dce58a to
0d3e30d
Compare
Signed-off-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>
SanjuCSudhakaran
force-pushed
the
enable-long-contexts-lora
branch
from
0d3e30d to
25dd63d
Compare
… Gaudi (vllm-project#12812) Signed-off-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>
… Gaudi (vllm-project#12812) Signed-off-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>
This PR refactors long-context + LoRA flow to align with the upstream main branch vllm-project#12812. HPU requires special handling while creating `long_lora_offsets_tensor` in `convert_mapping`. [(refer)](https://github.com/HabanaAI/vllm-fork/blob/b0a4e825370434ccaa30574ba9d20311fccdab36/vllm/lora/punica_wrapper/punica_hpu.py#L46) As suggested by the vllm team this PR sets `long_lora_context` to None while calling `convert_mapping`. This avoids HPU specific conditions inside `convert_mapping` and explicitly handles HPU long-lora logic inside overrided `_update_base_metadata`. Co-authored-by: Vivek Goel <vgoel@habana.ai>
… Gaudi (vllm-project#12812) Signed-off-by: Sanju C Sudhakaran <scsudhakaran@habana.ai> Signed-off-by: SzymonOzog <szymon.ozog@aleph-alpha.com>
… Gaudi (vllm-project#12812) Signed-off-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>
… Gaudi (vllm-project#12812) Signed-off-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>
This PR includes changes required to enable long-contexts + LoRA support in Intel Gaudi accelerators.