Sliding Window Support? #390
Comments
|
Yes, I'm working on that, will be available in v0.1.2 |
|
Awesome! Happy to hear! |
yzh119
added a commit
that referenced
this issue
Jul 29, 2024
As requested in #390 , this PR implements sliding window attention. This PR also result in slight performance degration because we didn't specialize kernels that use/not use sliding window. I believe we can address them by landing the JIT compilation feature. I'll merge this feature first and improve performance in later PRs.
|
Done in #406 , available in v0.1.2 release. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi!
I'm using FlashInfer under vLLM as it is the only implementation there that supports logit capping, which is required for the gemma2 model. However, vLLM disables sliding window attention because FlashInfer doesn't support it, limiting the context length of gemma2 from 8k to 4k.
Is there a plan to add support for this?
The text was updated successfully, but these errors were encountered: