Support for long-contexts with LoRA #385
Conversation
SanjuCSudhakaran
force-pushed
the
lora-long-context
branch
from
752fff5 to
e19f189
Compare
Signed-off-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>
| self.model.embedding_modules, | ||
| self.model.embedding_padding_modules) | ||
| self.model.embedding_padding_modules, | ||
| max_position_embeddings=self.model.config. |
There was a problem hiding this comment.
Why we are not initializing it exactly as it is being done in upstream code? Refer to
https://github.com/vllm-project/vllm/blob/dbfa8d31d5e7627a84671c6068ecc8fa58acd1d1/vllm/worker/model_runner.py#L1083
| long_lora_offsets = torch.zeros(len(index_mapping_indices), | ||
| device=get_device(), | ||
| dtype=torch.long) | ||
| long_lora_offsets_list = [] |
|
|
||
| long_lora_offsets = torch.tensor(long_lora_offsets_list, |
There was a problem hiding this comment.
W.r.t. functionality this change is ok for other backends (e.g. CUDA) also. @kzawora-intel is it ok to keep it? or do we need to make it HPU specific to be more upstream(ing) friendly?
| key, | ||
| offsets=self.punica_wrapper.long_lora_indices, | ||
| ) | ||
| offsets = self.punica_wrapper.long_lora_indices.reshape_as(positions) |
There was a problem hiding this comment.
Is this change ok for other backends (e.g. CUDA)?
| return (type(source_layer) is LinearScalingRotaryEmbedding | ||
| or type(source_layer) is RotaryEmbedding) | ||
| if current_platform.is_hpu(): | ||
| return (type(source_layer) is HpuLinearScalingRotaryEmbedding |
There was a problem hiding this comment.
Is there any functional impact of using default code?
|
These changes are no longer required since the latest vllm-fork already refactored RoPE to handle this. PR #404 |
This PR does the following